diff --git a/.claude/skills/porting-5-quants/SKILL.md b/.claude/skills/porting-5-quants/SKILL.md index 560e90e6..b39511ba 100644 --- a/.claude/skills/porting-5-quants/SKILL.md +++ b/.claude/skills/porting-5-quants/SKILL.md @@ -1,14 +1,14 @@ --- name: porting-5-quants -description: Produces the shipped quant matrix from the reference-dtype GGUF, smoke-tests each, publishes the matrix to a private HF repo, and takes a tentative WER read (Modal or local) for human review. Use after porting-4-cpp has finalized tolerances and passed validate.py + the full ref-dtype WER gate. Input: models//-.gguf. Output: F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M alongside the reference-dtype GGUF; a CLI smoke pass per file; quants pushed to a private HF repo; tentative per-quant WER. Authoritative quant WER is Stage 7. No tensor-level numerical comparison is required for quant acceptance — that is intentional. +description: Produces the shipped quant matrix from the reference-dtype GGUF, smoke-tests each, publishes the matrix to a private HF repo, and takes a preliminary 512-utterance WER read (Modal or local) for human review. Use after porting-4-cpp has finalized tolerances and passed validate.py + the full ref-dtype WER gate. Input: models//-.gguf. Output: F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M alongside the reference-dtype GGUF; a CLI smoke pass per file; quants pushed to a private HF repo; preliminary per-quant WER. Authoritative full-split quant WER is Stage 7. No tensor-level numerical comparison is required for quant acceptance — that is intentional. --- # porting-5-quants Stage 5 of the porting pipeline. Builds the quantizer, runs `scripts/quantize-all.py`, smoke-tests each GGUF, publishes the matrix to a -private HF repo, and takes a tentative WER read for human review. -Authoritative quant WER is Stage 7. +private HF repo, and takes a preliminary 512-utterance WER read for human +review. Authoritative full-split quant WER is Stage 7. ## Preconditions @@ -21,7 +21,7 @@ Authoritative quant WER is Stage 7. - `build/bin/transcribe-cli` and `build/bin/transcribe-quantize` are buildable. - `hf` authenticated for the target org (private upload). Modal optional - for the tentative WER sweep. + for the preliminary 512-utterance WER sweep. ## Workflow @@ -31,7 +31,7 @@ Quants progress: - [ ] Step 2: Run quantize-all - [ ] Step 3: CLI output-validity smoke per produced GGUF - [ ] Step 4: Publish quants to a private HF repo -- [ ] Step 5: Tentative WER sweep (Modal or local) +- [ ] Step 5: Preliminary 512-utterance WER sweep (Modal or local) - [ ] Step 6: Sign-off review ``` @@ -81,27 +81,32 @@ hf repo create /-gguf --repo-type model --private # if absent hf upload /-gguf models/ . --repo-type model ``` -### Step 5: Tentative WER sweep (execute) +### Step 5: Preliminary 512-utterance WER sweep (execute) -Per-quant WER for human review on the **full acceptance manifest**, not -a subset. "Tentative" here means "not the published number" (Stage 7 -re-runs and confirms), NOT "small N". Use Modal if credentials are -available; otherwise run locally. Do not pass `--n-utts` unless you have -a specific debugging reason and call it out in the sign-off. +Run each quant on the first 512 utterances of the acceptance manifest. This is +a bring-up signal only: it catches a clearly bad quant before benchmarking, +while Stage 7 remains the authoritative full-split publication run. Subset +artifacts carry `.n512` in their names and are never ingested into the catalog. +Use Modal if credentials are available; otherwise run locally. ```bash -# Modal: sweeps the private repo from Step 4 on GPU (full dataset) +# Modal: the output paths include .n512. modal run scripts/wer/remote/modal_sweep.py::sweep \ - --models /-gguf --quants "" -# local + --models /-gguf --quants "" --n-utts 512 + +# Local: materialize a separately named subset manifest and output. +mkdir -p build/wer +head -n 512 "$MANIFEST" > build/wer/..n512.manifest.jsonl for q in F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do uv run scripts/wer/run.py --model models//-$q.gguf \ - --manifest "$MANIFEST" --out reports/wer/-$q..jsonl - uv run scripts/wer/score.py reports/wer/-$q..jsonl + --manifest build/wer/..n512.manifest.jsonl \ + --out reports/wer/-$q..n512.jsonl + uv run scripts/wer/score.py reports/wer/-$q..n512.jsonl done ``` -Report the per-quant WER table for user review before Stage 6. +Report the preliminary per-quant table for user review before Stage 6. Do not +copy these rows into `catalog/.json`; Stage 7 owns published accuracy. ### Step 6: Sign-off @@ -109,7 +114,7 @@ Report: - Every produced GGUF with file size. - Any GGUF that failed the CLI smoke (with the failing output). - The private HF repo the matrix was pushed to. -- Tentative per-quant WER table (preliminary; Stage 7 authoritative). +- Preliminary 512-utterance per-quant WER table (Stage 7 authoritative). **Do not commit.** @@ -123,8 +128,11 @@ Report: - No tensor-level numerical comparison is required (or expected) for quant acceptance. - Quant matrix pushed to a private HF repo (`/-gguf`). -- Tentative per-quant WER produced and reviewed; authoritative WER is - Stage 7. +- Preliminary 512-utterance per-quant WER produced and reviewed; authoritative + full-split WER is Stage 7, and only Stage 7 results enter the catalog. +- The GGUFs are now the input Stage 6 seeds `catalog/.json` from + (`scripts/catalog/new_record.py`), so a wrong capability KV or licence + read propagates into the catalog. Fix it here, not in the record. ## Pointers (read, not execute) diff --git a/.claude/skills/porting-6-bench/SKILL.md b/.claude/skills/porting-6-bench/SKILL.md index 663cee7b..4092f216 100644 --- a/.claude/skills/porting-6-bench/SKILL.md +++ b/.claude/skills/porting-6-bench/SKILL.md @@ -17,6 +17,8 @@ requirement. - `models//-.gguf` exists for every shipped preset (F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M) — i.e. Stage 5 complete. +- `catalog/.json` exists, or Step 0 creates it. Nothing this stage + measures can be published without a record to hold it. - `build/bin/transcribe-bench` and `build/bin/transcribe-cli` are built under `build/`. - `scripts/bench/run.py` is runnable. @@ -47,9 +49,9 @@ Rules: rig's own `reports/perf//`. Vulkan cells only exist on the Ryzen box; Metal cells only exist on the Mac. - `reports/` is gitignored (`.gitignore:66`), so the per-rig JSON never - travels with the repo. The durable artifact is the **rendered table in - `docs/models/.md`** plus the `perf:` block in - `scripts/hf_cards/.yaml`. Transcribe the numbers into both. + travels with the repo. The durable artifact is the speed rows in + **`catalog/.json`**, ingested from those reports; the doc table + and the HF card are rendered from them. Never transcribe a number by hand. - A dev box that is neither rig (for example a base `apple-m4`) is iteration data. It may be added as an extra card section, but it does **not** substitute for either required rig. @@ -60,25 +62,50 @@ Rules: ## Standardized bench schema -Every per-cell report under -`reports/perf//__.json` is checked by the -`required_*` sets in Step 5. Missing required fields block Stage 6; -optional gaps are surfaced but do not gate. +`scripts/bench/run.py` refuses to write a report under +`reports/perf//__.json` that lacks any field +the catalog ingester or `compare.py` reads (`report_gaps()` in the driver). A +report on disk is complete by construction; a refusal is a bench-harness +regression and halts Stage 6. ## Workflow ``` Bench progress: +- [ ] Step 0: Catalog record exists (create it on a first port) - [ ] Step 1: Confirm full quant matrix present - [ ] Step 2: Rebuild transcribe-bench - [ ] Step 3: Confirm bench scope (publication default, optional widening) - [ ] Step 4: Capture publication baseline - [ ] Step 4b: Batch throughput sweep (good-to-know, non-gating) -- [ ] Step 5: Validate schema completeness -- [ ] Step 6: Iteration loop (human-driven, with validate gate per accept) -- [ ] Step 7: Sign-off review +- [ ] Step 5: Iteration loop (human-driven, with validate gate per accept) +- [ ] Step 6: Sign-off review ``` +### Step 0: Catalog record (execute, first port only) + +Stage 6 is the first stage that writes to the catalog, so the record has to +exist before a report can be ingested. A model that was ported before the +catalog existed already has one; a new port does not. + +```bash +ls catalog/.json || uv run scripts/catalog/new_record.py \ + --long-form --docs-page .md +``` + +Everything mechanical is read from artifacts Stage 5 already produced: the +intake supplies family, upstream repo and revision, and languages; the GGUFs +under `models//` supply the download table, byte sizes, parameter +count, capability KVs, licence and display name. The two required flags are +the facts no artifact carries: which `docs/input-limits.md` bucket the family +falls into, and which page under `docs/models/` documents it. Benchmark rows +are left empty for this stage and Stage 7 to fill. + +Check the seeded record before benching. `--published-repo`, +`--display-name`, `--license`, `--license-display` and `--language-tag-form` +override a wrong guess; a capability KV that the GGUF states wrongly is a +Stage 5 export bug, so fix it there and re-run rather than editing the record. + ### Step 1: Matrix presence (execute) ```bash @@ -102,19 +129,18 @@ capturing baseline. **Publication scope (default, required for sign-off).** This is the matrix that ends up rendered in `docs/models/.md`: -- Quants: `q8_0,q4_k_m` (the two columns the per-model perf table ships) -- Samples: `jfk,dots` (short + medium-length sample) -- Backends: `metal,cpu,vulkan` — `run.py` filters to whatever this machine - actually supports (Metal on macOS, Vulkan on Linux with the Vulkan - build, CPU everywhere) -- Iters: `3`, Warmup: `1` -- `--name -publication` -- Rigs: **both** of Apple M4 Max and AMD Ryzen 7 PRO 4750U (see - Reference machine matrix above). Run the identical command on each. +The checked-in profile `asr-publication-v2` is the source of truth. It measures +the publication quants Q8_0 and Q4_K_M when downloaded, on both `jfk` and +`dots`, with three iterations after one warmup, using the backends assigned to +the detected publication machine (M4 Max CPU/Metal or Ryzen 4750U CPU/Vulkan). +Every selected quant/sample/target cell is required. The Russian-only GigaAM +variants currently override the defaults with their published short `ru` +fixture; add a long Russian fixture to those overrides when one is available. +Add every future mandatory sample to the profile rather than restating the +matrix as command-line flags. -Confirm publication scope with the user, including which rigs are -reachable this session. Narrowed or widened sweeps are -allowed for iteration, but sign-off is decided on publication scope. +Narrowed or widened sweeps are allowed for iteration, but sign-off is decided +by the profile. ### Step 4: Baseline capture (execute) @@ -125,13 +151,7 @@ pollute timings. Publication-scope baseline (default): ```bash -uv run scripts/bench/run.py \ - --models \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name -publication-baseline-$(date -u +%Y%m%dT%H%M%SZ) +uv run scripts/bench/run.py --profile --models ``` Writes one report per (variant, backend) pair to `reports/perf//`. @@ -156,36 +176,7 @@ Emits per-batch `{batch_size, per_utt_ms, wall_ms}` to `reports/perf//__batch_.json`. This is exploratory and does not gate sign-off. Batch runs stay strictly serial. -### Step 5: Schema validation (execute) - -For each report file, confirm every required field is present and -surface any optional gaps: - -```python -# uv run python -c '...' -import json, pathlib, sys -required_top = {"git_sha", "machine"} -required_run = {"schema", "model_path", "backend", "per_iter", - "sample_duration_s", "rtf_wall_mean", "transcript_sha256"} -required_iter = {"mel_ms", "encode_ms", "decode_ms", "total_ms", "wall_ms"} -optional_top = {"git_dirty"} -optional_run = {"rtf_compute_mean", "token_ids_sha256"} -for p in sys.argv[1:]: - d = json.loads(pathlib.Path(p).read_text()) - missing_top = required_top - d.keys() - missing_run = any(required_run - r.keys() for r in d["runs"]) - missing_iter = any(required_iter - it.keys() for r in d["runs"] for it in r["per_iter"]) - absent_opt_top = optional_top - d.keys() - absent_opt_run = any(optional_run - r.keys() for r in d["runs"]) - print(f"{p}: required_top_missing={missing_top} required_run_missing={missing_run} " - f"required_iter_missing={missing_iter} optional_absent_top={absent_opt_top} " - f"optional_absent_run={absent_opt_run}") -``` - -Any missing **required** field is a bench-harness regression — halt Stage -6 sign-off. Absent **optional** fields are surfaced but do not block. - -### Step 6: Iteration loop (human-driven) +### Step 5: Iteration loop (human-driven) For each optimization hypothesis: @@ -225,32 +216,57 @@ For each optimization hypothesis: Repeat until the user is satisfied. -### Step 7: Sign-off +### Step 6: Sign-off Report: - Baseline reports and machine matrix covered. Name **each of the two required rigs** and its state: covered, or `INCOMPLETE — pending`. Do not report Stage 6 as complete while either rig is missing. -- Any schema gaps observed. - Total iterations run, net timing improvement, and that every accepted iteration passed `validate.py all`. **Do not commit.** Bench reports under `reports/perf/` may or may not be committed at the user's discretion. +## Catalog (mandatory exit step) + +`reports/` is gitignored, so a bench report exists only on the machine that +produced it. The stage is not finished until the numbers are in the catalog. + +1. Run the profile-selected publishable measurement, not a hand-written matrix: + + ```bash + uv run scripts/bench/run.py --profile --models + ``` + + `--profile` stamps both `publication: true` and the profile id in the + report. Hypothesis-loop runs (the optimization iterations above) MUST NOT + carry it: on CPU they differ from the shipped figure by tens of percent, + and the whole point is that they never reach a doc. + +2. Fold them in and regenerate the tables: + + ```bash + uv run scripts/catalog/ingest_perf.py + uv run scripts/catalog/render.py + uv run scripts/catalog/check.py --publication-profile --models + ``` + + `ingest_perf.py` ingests publication runs only. It refuses a row whose xRT + would move more than 5% against what the doc already published, because + that is a different build rather than a better reading of the same one; + re-bench at the current sha instead of passing `--force`. + +3. `check.py` must report every speed row for this variant as measured and + sourced. A row carrying a bare xRT with no `engine_sha` is not publishable. + ## Postconditions -- **Both required rigs benched** (Apple M4 Max and AMD Ryzen 7 PRO - 4750U), each with a bench report covering every **publication-scope** - cell (`q8_0`/`q4_k_m` × `jfk`/`dots` × that rig's backends, iters 3, - warmup 1) under that rig's `reports/perf//`, and each rig's - numbers transcribed into `docs/models/.md` and the `perf:` - block of `scripts/hf_cards/.yaml`. A missing rig is an - explicit, user-signed exception, never a silent default. The final on-doc run - uses `--name -publication` so the reproduction command in - `docs/models/.md` matches a real artifact. -- Schema completeness reported to the user; any gap is a known bench- - harness task, not a porting task. +- `catalog/.json` exists and is schema-valid. +- A sourced speed measurement for Q8_0 and Q4_K_M when downloaded, on both + `jfk` and `dots`, for every profile machine/backend target. Legacy xRT-only + rows may satisfy a cell but are explicitly marked and should be replaced + during the long-form/memory sweep. - Optimization iteration loop scripted end-to-end (user drives hypotheses; skill runs the loop). - Every accepted performance iteration was followed by a passing @@ -266,6 +282,8 @@ committed at the user's discretion. ## Pointers (read, not execute) - `docs/porting/5-benchmarks.md` — bench procedure context +- `scripts/catalog/new_record.py` — seeds the record from intake + GGUFs +- `catalog/_schema.json` — what a record is allowed to hold - `scripts/bench/run.py` — driver, already discovers `build/bin/` first - `scripts/bench/compare.py` — baseline-vs-candidate delta table - `tools/transcribe-bench/main.cpp` — bench binary source if the schema diff --git a/.claude/skills/porting-7-wer/SKILL.md b/.claude/skills/porting-7-wer/SKILL.md index 0ab45c4c..3b27c311 100644 --- a/.claude/skills/porting-7-wer/SKILL.md +++ b/.claude/skills/porting-7-wer/SKILL.md @@ -11,8 +11,9 @@ the measured Oracle reference baseline from Stage 2. Quant WER is **reviewed and signed off** by the user, not auto-gated. -Stage 4 already gated ref dtype, and Stage 5 took a tentative quant read. -Stage 7 re-confirms after bench and records human review for every quant. +Stage 4 already gated ref dtype, and Stage 5 took a preliminary +512-utterance quant read. Stage 7 runs the authoritative full split after +bench and records human review for every quant. ## Preconditions @@ -31,11 +32,12 @@ Stage 7 re-confirms after bench and records human review for every quant. ``` WER progress: - [ ] Step 1: Ensure acceptance manifest -- [ ] Step 2: Score the reference-dtype model +- [ ] Step 2: Run the publication profile (every profile cell, every quant) - [ ] Step 3: Check the ref-dtype WER limit -- [ ] Step 4: Score each shipped quant +- [ ] Step 4: Score acceptance cells the profile does not cover - [ ] Step 5: Write the summary table -- [ ] Step 6: Sign-off review +- [ ] Step 6: Ingest into the catalog and render +- [ ] Step 7: Sign-off review ``` ### Step 1: Acceptance manifest (execute or ask-point) @@ -68,26 +70,28 @@ MANIFEST=samples/wer/fleurs-${LANG}.manifest.jsonl CER auto-routes for zh / yue / ja / ko / th via the manifest's `language` field. The score JSON's `error_rate_pct` is the canonical report metric. -### Step 2: Score the reference-dtype model (execute) +### Step 2: Run the publication profile (execute) -Read intake for the reference dtype and acceptance dataset: +One sweep produces both the release numbers and the gate inputs. The +checked-in profile (`catalog/_benchmark_profiles.json`) selects the cells: +LibriSpeech test-clean at every downloaded quant for English-capable models, +and FLEURS test at Q8_0 for every supported FLEURS language. Run whatever is +missing: ```bash -REFDTYPE=$(uv run python -c "import json; d=json.load(open('reports/porting///intake.json')); \ - m={'float32':'F32','float16':'F16','bfloat16':'BF16'}; print(m[d['dtype']['expected']])") -DATASET=$(uv run python -c "import json; d=json.load(open('reports/porting///intake.json'))['upstream_benchmarks'][0]['dataset']; \ - print(d.replace(' ', '-').lower())") - -# $MANIFEST was resolved in Step 1 — use it directly. -uv run scripts/wer/run.py \ - --model models//-${REFDTYPE}.gguf \ - --manifest "$MANIFEST" \ - --out reports/wer/-${REFDTYPE}.${DATASET}.jsonl - -uv run scripts/wer/score.py reports/wer/-${REFDTYPE}.${DATASET}.jsonl +modal run scripts/wer/remote/modal_sweep.py::publication_sweep \ + --models # --plan-only to inspect the expansion +for f in reports/wer/-*.jsonl; do uv run scripts/wer/score.py "$f"; done ``` -`score.py` writes the `.score.json` consumed by the gate and summary. +The sweep writes `reports/wer/-.[.bN].jsonl` and +`score.py` writes the matching `.score.json`. Every JSONL carries its decode +recipe, engine sha, and profile id in the batch header, and the score carries +them forward, which is what makes it ingestible in Step 6. + +Do not also run the same cells locally with `run.py`: a second measurement of +one cell under a different backend is a second number to reconcile, not a +check. ### Step 3: Ref-dtype WER limit (execute) @@ -146,14 +150,21 @@ Proceed only when the higher WER is explained, reviewed, and written in the WER summary or family doc. Higher WER without evidence is a release blocker. -### Step 4: Score each shipped quant (execute) +### Step 4: Acceptance cells outside the profile (execute) + +The profile covers the acceptance dataset for most ports. Two cases fall +outside it and are scored locally, exactly as the profile would, so the files +land under the same names: -Loop over `F16, Q8_0, Q6_K, Q5_K_M, Q4_K_M`, skipping whichever equals -`REFDTYPE`: +- The acceptance dataset is FLEURS (a single-language port such as + `parakeet-primeline` or `gigaam`): the profile ran Q8_0 only. Score the + reference dtype and the remaining quants on `$MANIFEST`. +- The acceptance dataset is not a profile dataset at all (AMI for a + diarizer): score every preset on `$MANIFEST`. ```bash -for PRESET in F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do - [ "$PRESET" = "$REFDTYPE" ] && continue +for PRESET in F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do + [ -f reports/wer/-${PRESET}.${DATASET}.score.json ] && continue uv run scripts/wer/run.py \ --model models//-${PRESET}.gguf \ --manifest "$MANIFEST" \ @@ -162,10 +173,9 @@ for PRESET in F16 Q8_0 Q6_K Q5_K_M Q4_K_M; do done ``` -Quant WER is reviewed and signed off by the user, not auto-gated. - -Batch mode should be WER-neutral. If a `--batch-size > 1` sweep differs -from serial beyond dataset noise (~0.01), stop and report the numbers. +Quant WER is reviewed and signed off by the user, not auto-gated. Batch mode +should be WER-neutral; if a `--batch-size > 1` sweep differs from serial +beyond dataset noise (~0.01), stop and report the numbers. ### Step 5: Summary table (execute) @@ -176,30 +186,60 @@ gate result (`PASS` or `BLOCKED`). Quant rows record human disposition (`ACCEPTED`, `REJECTED`, or `PENDING REVIEW`) plus any short note the user gives. Stage 8 (`porting-8-ship`) consumes this into the model card. -### Step 6: Sign-off +### Step 6: Ingest into the catalog and render (execute) + +Only profile-stamped scores are ingested; a score with no engine sha or a +different recipe is rejected by name, and the rejection is the finding. + +```bash +uv run scripts/catalog/ingest_accuracy.py --models +``` + +Set `headline_benchmark` in `catalog/.json` to the cell the +download table features. A variant can carry several runs of one dataset +differing only in batch size or timestamp mode, so the pointer names the +whole identity: + +```json +"headline_benchmark": {"dataset": "librispeech", "split": "test-clean", + "language": "en", "metric": "wer", + "batch_size": 1, "timestamps": "none"} +``` + +```bash +uv run scripts/catalog/check.py --publication-profile --models +uv run scripts/catalog/render.py +``` + +Never hand-edit a WER into a doc or an HF card spec: both are rendered from +the catalog, and CI fails when they drift. + +### Step 7: Sign-off Report: - Manifest path and utterance count. - Ref-dtype status: measured Oracle reference WER, C++ WER, max allowed WER, pass/blocked, and any required justification. -- Path to every produced `.score.json`. +- Path to every produced `.score.json`, and which came from the profile + sweep versus Step 4. - Path to the summary markdown. +- The catalog check result for the variant. - Human disposition for every shipped quant. Quant WER has no automatic numeric gate; unresolved quant review means Stage 7 sign-off is pending. **Do not commit.** WER outputs under `reports/wer/` are local generated -artifacts, ignored by `.gitignore`. The summary tables and per-quant -WER cells are what ships in-repo via Stage 8. +artifacts, ignored by `.gitignore`. The catalog record and the rendered +tables are what ships in-repo. ## Postconditions - `reports/wer/-..score.json` for every - shipped preset. + shipped preset, profile-stamped where the profile covers the cell. - `reports/wer/..summary.md` table. - Ref-dtype status is known and reported as plain WER numbers against the measured Oracle reference baseline. -- Sign-off names the manifest path and utterance count so consumers can - verify which dataset was scored. +- `catalog/.json` holds every profile accuracy cell and a + `headline_benchmark`; the per-variant profile check passes. - Quant WER is reviewed and signed off by the user, not auto-gated. ## Pointers (read, not execute) diff --git a/.claude/skills/porting-8-ship/SKILL.md b/.claude/skills/porting-8-ship/SKILL.md index c4c25663..16e39028 100644 --- a/.claude/skills/porting-8-ship/SKILL.md +++ b/.claude/skills/porting-8-ship/SKILL.md @@ -20,28 +20,33 @@ is out of scope. - `reports/convert/-.json` (SHA of the reference GGUF). - `reports/wer/-..score.json` for every shipped preset. -- `reports/perf//*__.json` for **both** - publication rigs (Apple M4 Max and AMD Ryzen 7 PRO 4750U), each run - on that rig. See the Reference machine matrix in `porting-6-bench`. +- The catalog satisfies `asr-publication-v2` for this variant: complete + accuracy and every speed quant/sample/machine/backend cell selected by the + profile (currently Q8_0 and Q4_K_M on both `jfk` and `dots`, except an + explicit supported-language sample override such as GigaAM's `ru`). ## Workflow +One ordered finalization. Each step gates the next; nothing is uploaded +until the human review in Step 7 has passed on files that already survived +every check. + ``` Ship progress: -- [ ] Step 1: Pre-flight checklist (artifacts present) -- [ ] Step 2: Fill the family doc -- [ ] Step 3: Author the user-facing model card -- [ ] Step 4: Write the HF card YAML spec -- [ ] Step 5: Render the HF README -- [ ] Step 6: Pre-upload review -- [ ] Step 7: Sign-off review +- [ ] Step 1: Verify inputs and GGUF metadata +- [ ] Step 2: Ingest measurements +- [ ] Step 3: Validate the publication profile +- [ ] Step 4: Author prose (family doc, model page, HF card spec) +- [ ] Step 5: Render docs and card +- [ ] Step 6: Verify no drift +- [ ] Step 7: Human review (ask-point) +- [ ] Step 8: Upload ``` -### Step 1: Pre-flight checklist (execute) +### Step 1: Verify inputs and GGUF metadata (execute) Confirm every artifact exists. If any row is missing, halt and send the -user to the stage that owns the missing artifact — Stage 8 does not -fabricate inputs. +user to the stage that owns it. Stage 8 does not fabricate inputs. | Artifact | Expected path | Owning stage | |---|---|---| @@ -55,9 +60,9 @@ fabricate inputs. | Bench reports, rig 2 | `### AMD Ryzen 7 PRO 4750U` section in `docs/models/.md` | Stage 6 | | WER score JSONs | `reports/wer/-*..score.json` | Stage 7 | | WER summary | `reports/wer/..summary.md` | Stage 7 | +| Catalog publication profile | `catalog/_benchmark_profiles.json` + `catalog/.json` | Stages 6–7 | ```bash -# Mechanical checklist runner for path in \ reports/porting///intake.json \ tests/golden//.manifest.json \ @@ -70,32 +75,75 @@ do done ls models//-*.gguf >/dev/null 2>&1 \ && echo "OK quants" || echo "MISSING quants" -# Bench: reports/ is gitignored and each rig keeps its own JSON locally, so -# the checkable artifact is the rendered rig section in the model card. -# BOTH rigs are required. A base apple-m4 (or any other dev box) does not -# substitute for either. -for rig in "Apple M4 Max" "AMD Ryzen 7 PRO 4750U"; do - grep -q "^### $rig" docs/models/.md \ - && echo "OK bench-rig: $rig" || echo "MISSING bench-rig: $rig" -done -grep -q "^ m4-max:" scripts/hf_cards/.yaml \ - && echo "OK perf yaml m4-max" || echo "MISSING perf yaml m4-max" -grep -q "^ ryzen-4750u:" scripts/hf_cards/.yaml \ - && echo "OK perf yaml ryzen-4750u" || echo "MISSING perf yaml ryzen-4750u" -ls reports/wer/-*..score.json >/dev/null 2>&1 \ - && echo "OK wer-scores" || echo "MISSING wer-scores" ``` -Any `MISSING` halts Stage 8. +Then audit the files that will be uploaded: + +```bash +uv run --project scripts/envs/moonshine scripts/audit_gguf_metadata.py models/ +uv run scripts/catalog/sync_capabilities.py --check --models +``` + +`audit_gguf_metadata.py` exits non-zero on any metadata issue. +`sync_capabilities.py --check --models ` reads every published quant +and exits non-zero if any file is unreadable, lacks a capability KV, disagrees +with another quant, or disagrees with the record. A file and its own model +card must not contradict each other on the Hub. + +**Never hand-write the `capabilities` block.** Hand-writing it is how +moss-transcribe-diarize shipped as `diarize:false`, how the granite GGUFs came +to carry `stt.capability.translation` where the loader reads +`stt.capability.translate`, and how nemotron-3.5 shipped with no streaming KV +at all. `sync_capabilities.py` (without `--check`) reads the block back out of +the file. If it disagrees with what the model actually does, the GGUF is +wrong and the fix is a converter change plus a re-export, never an edit to the +record or the card spec. + +**Absence is not falsity.** `read_capability_bool()` leaves a field untouched +when its KV is missing, so a missing KV silently inherits the family default. +`granite/capabilities.cpp` sets `supports_translate = true` on purpose so +each variant's GGUF can lower it; `granite-speech-4.1-2b-plus` spelled that +key `stt.capability.translation`, the lowering never happened, and a model +that does not translate advertised that it does. The shared writer factory in +`scripts/lib/gguf_common.py` writes `false` for any capability KV a converter +leaves unset, so every fresh export states all four. + +**Know where the local files came from.** `models//` is for most +variants a symlink into external storage holding whatever was built there +last. That mirror can be *older* than the Hub. Before re-uploading a variant +you did not just convert, either re-download it from its published repo or +confirm the divergence is intended. + +### Step 2: Ingest measurements (execute) + +Idempotent; Stages 6 and 7 normally did this already. + +```bash +uv run scripts/catalog/ingest_perf.py +uv run scripts/catalog/ingest_accuracy.py --models +``` -A `MISSING bench-rig` is the one row that is easy to wave through, because -a model card with one rig table *looks* finished. It is not: publication -scope is two rigs (`porting-6-bench`, Reference machine matrix). Send the -user back to Stage 6 on the missing rig. The only way past it is the user -explicitly signing off on shipping with one rig, in which case the card -must name the rig the numbers were measured on. +`ingest_perf.py` takes publication runs only and refuses a row whose xRT +would move more than 5% against the published value; re-bench at the current +sha rather than passing `--force`. `ingest_accuracy.py` takes profile-stamped +scores only. Confirm `headline_benchmark` is set in `catalog/.json`. -### Step 2: Family doc (execute + ask-point) +### Step 3: Validate the publication profile (execute) + +```bash +uv run scripts/catalog/check.py --publication-profile --models +``` + +A failure halts Stage 8. A `legacy-published` provenance marker is honest +migration provenance and may satisfy the gate; it is not permission to assign +a guessed engine SHA. + +### Step 4: Author prose (execute + ask-point) + +Three files, prose only. Every number, repo, licence, language, capability, +and table is rendered from the catalog in Step 5. + +### Step 4a: Family doc Open `docs/porting/families/.md`. If still the `_template.md` shape, fill it section by section by pulling facts from the artifacts: @@ -118,7 +166,7 @@ For a new family, draft the Known Limitations section from intake capabilities (streaming flag, translation flag, language coverage, timestamp granularity) plus any sharp edges the port surfaced. Do not invent limitations the port didn't discover; do not omit limitations the -capabilities flags imply. Present the draft for human review in Step 6. +capabilities flags imply. Present the draft for human review in Step 7. State the **batch and streaming** posture from the Capability Validation rows: @@ -132,55 +180,50 @@ rows: accepted gap for a natively-streaming model. If the model does not stream, omit the row. -### Step 3: User-facing model card (execute) - -Author `docs/models/.md`. The repo ships a Jinja template at -`docs/_templates/model-card.md.j2` and existing rendered cards (e.g. -`docs/models/parakeet-tdt-0.6b-v2.md`) are the shape reference. - -Two acceptable approaches: - -1. **Copy from the closest existing model card** and edit by hand. Pull - facts directly from artifacts — quants from - `models//`, WER and the measured reference baseline from - `reports/wer/-*.score.json`, bench from - `reports/perf//` on **both** rigs (one `###` section each, - `Apple M4 Max` with metal/cpu and `AMD Ryzen 7 PRO 4750U` with - vulkan/cpu, each footnoted with its OS and the transcribe.cpp sha), - and the acceptance dataset from - `intake.upstream_benchmarks[0]`. Ask for `target_hf_repo` since it - cannot be inferred. -2. **Render from the existing template** if the template already covers - everything the variant needs and the variant has no rendered card - yet. Build the context dict in a short ad-hoc `uv run python -c` - and write the output. Do **not** extend the template with new - context fields just for one family — handcraft those sections in - the rendered markdown instead. - -Subsequent regenerations must respect human edits. - -### Step 4: HF card YAML spec (execute) - -Write `scripts/hf_cards/.yaml`, mirroring -`scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml`: +### Step 4b: Model page + +Author `docs/models/.md` by copying the closest existing model +page (e.g. `docs/models/parakeet-tdt-0.6b-v2.md`) and editing the prose. +There is one way to do this and this is it: a page is hand-written prose +around `` regions, and the copy already carries the right +set of regions in the right order. + +Replace the prose only. Every table, number, repo, licence, language and +capability comes from the `catalog:` regions, which Step 5 fills from +`catalog/.json`. Leave the region bodies as they are; the renderer +overwrites them. Reference context for the prose comes from +`reports/wer/-*.score.json` and the acceptance dataset from +`intake.upstream_benchmarks[0]`. `published_repo` is catalog data, not an +editorial value to ask for again. + +Do not restate a catalogued number in the prose. The parameter count, the +download sizes, the error rates and the validation commit are all rendered a +few lines away, and a hand-typed copy of one is a second source of truth that +nothing checks. Say what the model is for; let the regions say how big and +how accurate it is. + +Everything outside a `catalog:` region is yours, and re-running the renderer +never touches it. + +### Step 4c: HF card spec + +Write `scripts/hf_cards/.yaml`, mirroring a current nearby spec. +It contains editorial and release state only; identity, repositories, upstream +commit, license, language support, downloads, benchmark values, capability +flags, the metric column label and the link back to the model page are all +derived from `catalog/.json`. `generate.py` accepts only the keys +below and refuses anything else, so a field that belongs to the catalog +cannot creep back in and a misspelled key fails loudly: ```yaml -hf_repo: -target_repo: -gguf> -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/.md - -upstream_commit: -pin_date: +pin_date: validation: reference: - commit: - date: + commit: + date: -license: -license_display: pipeline_tag: automatic-speech-recognition -languages: [] tags: - gguf - transcribe.cpp @@ -188,61 +231,90 @@ tags: - speech-to-text - - +summary: | + + +wer: + notes: | + ``` -### Step 5: Render the HF README (execute) +`validation.commit` is a real commit and is rendered as a link into the +tree, so it must be a SHA. Record the commit the validation ran at; the +reference framework's version belongs in `validation.reference`. Never +copy a plausible-looking SHA from a sibling spec: a pin that names no +real run is worse than no pin, because it cannot be told from one that does. + +`wer.notes` holds editorial caveats only. The mechanical sentence +(dataset, size, batch, timestamps, build) is generated from the headline +rows, and the metric column label is the catalog's own; state a number here +only to compare against something the catalog does not hold, such as an +upstream self-reported figure. + +### Step 5: Render docs and card (execute) ```bash +uv run scripts/hf_cards/check_release.py # pin_date and validation pin +uv run scripts/catalog/render.py # docs/models/*.md and the root README uv run scripts/hf_cards/generate.py scripts/hf_cards/.yaml ``` -Writes `models//README.md` by default. +`render.py` fills every marked region: download and accuracy tables, perf +tables with their provenance, the intro and WER note from the spec, family +roll-ups, and the supported-models table in the root README. `generate.py` +writes `models//README.md` from the spec and the catalog record, and +refuses a spec that states a catalog-owned field. -### Step 6: Pre-upload review (ask-point) +### Step 6: Verify no drift (execute) -Drafts from Steps 2, 3, and 5 are now on disk. Present three paths for -human review: +```bash +uv run scripts/catalog/format.py --check +uv run scripts/catalog/check.py --publication-profile --models +uv run scripts/catalog/render.py --check +``` + +All three must be clean. This is what CI runs; a failure here is a failure +there. + +### Step 7: Human review (ask-point) + +Present for review: - `docs/porting/families/.md` - `docs/models/.md` - `models//README.md` -Flag likely over-promising sections (`one_liner`, `capabilities_prose`, -Known Limitations) and wait for explicit sign-off before Step 7. +Flag likely over-promising sections (the spec's `summary` and `wer.notes`, +which render into both the HF README and the model page; Known Limitations) +and wait for explicit sign-off before Step 8. + +### Step 8: Upload (execute) -### Step 7: Sign-off +```bash +hf upload models/ . --repo-type model +``` -Report: -- All four output paths (family doc, model card, HF YAML, HF README). -- Target private HF repo. -- Pre-flight checklist outcome from Step 1. -- Push the rendered docs/README to the private repo: - ```bash - hf upload models/ . --repo-type model - ``` -- Remind the user to commit the docs/families/models/hf_cards changes. -- If this port adds a new family (or new variants under an existing - family), remind the user to update the supported-models table in the - root `README.md` so the family/variants are listed. +Report the output paths, the target private repo, and the Step 6 results. +Remind the user to commit the catalog, docs, and spec changes. **Do not commit.** Keep the repo private; flipping it public is a future action, not part of this stage. ## Postconditions -- Pre-flight checklist (Step 1) was green before any drafting. +- Steps 1 to 3 were green before any prose was drafted. +- HF validation commit exists and `validation.date` equals the UTC ship date. - `docs/porting/families/.md` filled and reviewed. -- `docs/models/.md` authored with a populated download / WER / - bench table, the bench carrying **both** rig sections (Apple M4 Max, - AMD Ryzen 7 PRO 4750U) unless the user signed off on fewer. +- `docs/models/.md` authored; its marked regions render from the + catalog. - `scripts/hf_cards/.yaml` committed-ready. -- `models//README.md` rendered. -- Docs/README pushed to the private HF repo; public flip deferred. +- `models//README.md` rendered and uploaded to the private repo; + public flip deferred. +- The root README's supported-models table renders the new variant. ## Pointers (read, not execute) - `docs/porting/families/_template.md` — family doc shape -- `docs/models/parakeet-tdt-0.6b-v2.md` — model card shape reference -- `scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml` — HF card YAML reference -- `scripts/hf_cards/template.md.j2` — Jinja template that generate.py - renders -- `scripts/hf_cards/generate.py` — renderer (execute only via Step 5) +- `docs/models/parakeet-tdt-0.6b-v2.md` — model page shape reference +- `scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml` — HF card spec reference +- `scripts/hf_cards/README.md` — the render, review, upload loop +- `scripts/catalog/render.py` — marker blocks and what each renders diff --git a/.github/workflows/catalog.yml b/.github/workflows/catalog.yml new file mode 100644 index 00000000..9ba9a25a --- /dev/null +++ b/.github/workflows/catalog.yml @@ -0,0 +1,60 @@ +name: model-catalog + +on: + pull_request: + branches: [main] + paths: + - "catalog/**" + - "scripts/catalog/**" + - "scripts/hf_cards/**" + - "docs/models/**" + - "README.md" + - "scripts/bench/**" + - "scripts/wer/languages.py" + - "scripts/wer/der.py" + - "scripts/wer/test_der.py" + - ".github/workflows/catalog.yml" + push: + branches: [main] + paths: + - "catalog/**" + - "scripts/catalog/**" + - "scripts/hf_cards/**" + - "docs/models/**" + - "README.md" + - "scripts/bench/**" + - "scripts/wer/languages.py" + - "scripts/wer/der.py" + - "scripts/wer/test_der.py" + - ".github/workflows/catalog.yml" + +concurrency: + group: model-catalog-${{ github.ref }} + cancel-in-progress: true + +jobs: + catalog: + runs-on: blacksmith-2vcpu-ubuntu-2404 + timeout-minutes: 10 + steps: + - uses: actions/checkout@v6 + - uses: astral-sh/setup-uv@v8.2.0 + - name: Canonical formatting + run: uv run scripts/catalog/format.py --check + - name: Schema, integrity and pairing + run: uv run scripts/catalog/check.py + - name: Publication profile + run: uv run scripts/catalog/check.py --publication-profile + - name: HF cards render from the catalog + run: | + for spec in scripts/hf_cards/*.yaml; do + uv run scripts/hf_cards/generate.py "$spec" --skip-upstream --stdout > /dev/null + done + - name: Published tables match the catalog + run: uv run scripts/catalog/render.py --check + - name: Rebuild portable database + run: | + uv run --with pytest pytest -q scripts/catalog/test_db_mapping.py + uv run scripts/wer/test_der.py + uv run scripts/catalog/db.py --out "$RUNNER_TEMP/catalog.db" + sqlite3 "$RUNNER_TEMP/catalog.db" "PRAGMA integrity_check" diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index 81d9f99d..5be7ca28 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -11,9 +11,10 @@ name: publish # the run before any expensive matrix work or immutable upload. # # VALIDATE (Actions artifacts only — no registry touched): wheels -# (python-wheels.yml), cuda-windows (cuda-windows.yml), swift-build -# (xcframework -> artifact, NOT the Release), rust-verify (cargo publish -# --dry-run + packed-crate smoke). These run on BOTH a release tag and a +# (python-wheels.yml), cuda-windows (cuda-windows.yml), catalog-build +# (catalog.db + checksum), swift-build (xcframework -> artifact, NOT the +# Release), rust-verify (cargo publish --dry-run + packed-crate smoke). +# These run on BOTH a release tag and a # workflow_dispatch rehearsal; Windows CUDA is release-required, not special. # # PUBLISH (tags only — immutable uploads): create-release, publish-pypi, @@ -109,9 +110,34 @@ jobs: # =========================================================================== # VALIDATE — build + verify only; produce Actions artifacts, touch no # registry. Runs on a tag AND a rehearsal. The publish phase below `needs:` - # the full set [release-preflight, wheels, cuda-windows, swift-build, - # rust-verify]. + # the full set [release-preflight, catalog-build, wheels, cuda-windows, + # swift-build, rust-verify]. # =========================================================================== + catalog-build: + # Build the portable catalog from its JSON source and stash the exact + # validated database for release-assets. Rehearsals retain it as an Actions + # artifact; tags also attach it to the GitHub Release. + needs: [release-preflight] + runs-on: blacksmith-2vcpu-ubuntu-2404 + timeout-minutes: 10 + steps: + - uses: actions/checkout@v6 + - uses: astral-sh/setup-uv@v8.2.0 + - name: Validate and build catalog.db + run: | + set -euo pipefail + uv run scripts/catalog/format.py --check + uv run scripts/catalog/check.py --publication-profile + mkdir -p release-catalog + uv run scripts/catalog/db.py --out release-catalog/catalog.db + test "$(sqlite3 release-catalog/catalog.db 'PRAGMA integrity_check')" = "ok" + (cd release-catalog && sha256sum catalog.db > catalog.db.sha256) + - uses: actions/upload-artifact@v7 + with: + name: catalog-db + path: release-catalog/ + if-no-files-found: error + wheels: # Full build + validation matrix; the artifacts published below are the # exact files these workflows tested post-repair. @@ -196,7 +222,7 @@ jobs: # release-required lane is unproven" rule the tag path enforces. A later # CUDA/Swift/TS failure must not force a version bump just to re-rehearse. if: github.event_name == 'workflow_dispatch' - needs: [release-preflight, wheels, cuda-windows, rust-rehearsal, swift-rehearsal, ts-rehearsal] + needs: [release-preflight, catalog-build, wheels, cuda-windows, rust-rehearsal, swift-rehearsal, ts-rehearsal] runs-on: blacksmith-2vcpu-ubuntu-2404 timeout-minutes: 30 environment: testpypi @@ -380,7 +406,7 @@ jobs: # at workflow start. Keep it draft until the mandatory publishers have # succeeded and assets are verified (finalize-release). if: startsWith(github.ref, 'refs/tags/v') - needs: [release-preflight, wheels, cuda-windows, swift-build, rust-verify, ts-pack] + needs: [release-preflight, catalog-build, wheels, cuda-windows, swift-build, rust-verify, ts-pack] runs-on: blacksmith-2vcpu-ubuntu-2404 permissions: contents: write @@ -403,7 +429,7 @@ jobs: # Also waits on create-release: no immutable registry upload starts unless the # GitHub Release object itself was created (else PyPI could land with no Release). if: startsWith(github.ref, 'refs/tags/v') - needs: [create-release, release-preflight, wheels, cuda-windows, swift-build, rust-verify, ts-pack] + needs: [create-release, release-preflight, catalog-build, wheels, cuda-windows, swift-build, rust-verify, ts-pack] # Blacksmith for the same docker-container-action reason as testpypi. runs-on: blacksmith-2vcpu-ubuntu-2404 timeout-minutes: 30 @@ -436,7 +462,7 @@ jobs: # NO needs: and could burn the immutable crates.io version before anything # else built, or before the Release object even existed). if: startsWith(github.ref, 'refs/tags/v') - needs: [create-release, release-preflight, wheels, cuda-windows, swift-build, rust-verify, ts-pack] + needs: [create-release, release-preflight, catalog-build, wheels, cuda-windows, swift-build, rust-verify, ts-pack] runs-on: blacksmith-2vcpu-ubuntu-2404 timeout-minutes: 40 environment: crates-io @@ -528,7 +554,7 @@ jobs: # tarball does NOT re-run prepublishOnly, so no build happens here — only the # upload. The `npm` environment carries the approval gate. if: startsWith(github.ref, 'refs/tags/v') - needs: [create-release, release-preflight, wheels, cuda-windows, swift-build, rust-verify, ts-pack] + needs: [create-release, release-preflight, catalog-build, wheels, cuda-windows, swift-build, rust-verify, ts-pack] runs-on: blacksmith-2vcpu-ubuntu-2404 environment: npm steps: @@ -554,9 +580,10 @@ jobs: release-assets: # cu12's primary distribution home (wheels as GitHub release assets, served # to pip through the PEP 503 index on Pages), the canonical native bundles, - # AND the Swift xcframework (built+validated in swift-build, uploaded here). + # the portable catalog database, AND the Swift xcframework (built+validated + # in swift-build, uploaded here). if: startsWith(github.ref, 'refs/tags/v') - needs: [create-release, release-preflight, wheels, cuda-windows, swift-build, rust-verify, ts-pack] + needs: [create-release, release-preflight, catalog-build, wheels, cuda-windows, swift-build, rust-verify, ts-pack] # Blacksmith, not the hetzner self-hosted box: this job only downloads # artifacts and `gh release upload`s them, and the hetzner runner has no `gh` # on PATH (release v0.0.3 failed here with "gh: command not found"). The @@ -586,6 +613,19 @@ jobs: with: name: swift-xcframework path: swift + - uses: actions/download-artifact@v8 + with: + name: catalog-db + path: catalog + - name: Attach the catalog database + env: + GH_TOKEN: ${{ github.token }} + run: | + set -euo pipefail + tag="${GITHUB_REF#refs/tags/}" + (cd catalog && sha256sum --check catalog.db.sha256) + gh release upload "$tag" catalog/catalog.db catalog/catalog.db.sha256 \ + --repo "$GITHUB_REPOSITORY" --clobber - name: Attach the cu12 wheels env: GH_TOKEN: ${{ github.token }} @@ -653,6 +693,8 @@ jobs: } require_asset "TranscribeCpp.xcframework.zip" + require_asset "catalog.db" + require_asset "catalog.db.sha256" require_asset "transcribe-native-${ver}-linux-x86_64-cpu-vulkan.tar.gz" require_asset "transcribe-native-${ver}-linux-aarch64-cpu-vulkan.tar.gz" require_asset "transcribe-native-${ver}-macos-arm64-metal.tar.gz" diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2cd33edc..2eb6e1cc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -233,7 +233,7 @@ Required before canonical publication: | Gate | Owner | Requirement | | --- | --- | --- | | WER / benchmarks | maintainer or contributor | Per-quant numbers recorded in `docs/models/.md` | -| HF card | maintainer | `scripts/hf_cards/.yaml` updated and README rendered | +| HF card | maintainer | Catalog record current; README rendered per `scripts/hf_cards/README.md` | | Canonical upload | maintainer | GGUFs uploaded to the `handy-computer` Hugging Face organization | | Download roundtrip | maintainer | Downloaded canonical GGUF reloads and validates cleanly; validation commit recorded in docs/HF card | | Preflight D | maintainer | Planned post-quantization gate once implemented | @@ -256,7 +256,7 @@ project release artifact only after a maintainer publishes it there and records the validation commit in both: - `docs/models/.md` -- `scripts/hf_cards/.yaml`, rendered into the HF README +- `scripts/hf_cards/.yaml` (`validation` block), rendered into the HF README Golden manifests are not mutated for uploads. They pin validation provenance for the port. Release state lives in the model card and rendered HF README; HF diff --git a/README.md b/README.md index 85319a7e..19c924c4 100644 --- a/README.md +++ b/README.md @@ -6,32 +6,78 @@ C/C++ speech-to-text inference library. Runs diverse STT model families via [GGU **Supported models:** -| Family | Variants | Docs | -| --- | --- | --- | -| Parakeet | 11 variants: TDT, RNN-T, CTC, TDT+CTC (110M–1.1B), incl. German `parakeet-primeline` | [docs/models/parakeet.md](docs/models/parakeet.md) | -| Canary | `canary-1b`, `canary-1b-v2`, `canary-1b-flash`, `canary-180m-flash` | [docs/models/canary.md](docs/models/canary.md) | -| Canary-Qwen | `canary-qwen-2.5b` (FastConformer + Qwen3-1.7B SALM) | [docs/models/canary-qwen-2.5b.md](docs/models/canary-qwen-2.5b.md) | -| Whisper | 12 variants (`tiny` through `large-v3-turbo`, plus `.en` siblings) | [docs/models/whisper.md](docs/models/whisper.md) | -| GigaAM | `gigaam-v3-{e2e-rnnt,e2e-ctc,rnnt,ctc}` | [docs/models/gigaam.md](docs/models/gigaam.md) | -| Moonshine | `moonshine-tiny`, `moonshine-base` | [docs/models/moonshine.md](docs/models/moonshine.md) | -| Moonshine Streaming | `moonshine-streaming-{tiny,small,medium}` | [docs/models/moonshine-streaming.md](docs/models/moonshine-streaming.md) | -| Qwen3-ASR | `qwen3-asr-0.6b`, `qwen3-asr-1.7b` | [docs/models/qwen3-asr.md](docs/models/qwen3-asr.md) | -| Cohere Transcribe | `cohere-transcribe-03-2026` | [docs/models/cohere-transcribe-03-2026.md](docs/models/cohere-transcribe-03-2026.md) | -| SenseVoice | `sensevoice-small` | [docs/models/sensevoice-small.md](docs/models/sensevoice-small.md) | -| FunASR Nano | `fun-asr-nano-2512`, `fun-asr-mlt-nano-2512` | [docs/models/fun-asr-nano.md](docs/models/fun-asr-nano.md) | -| Nemotron Speech Streaming | `nemotron-speech-streaming-en-0.6b` | [docs/models/nemotron-speech-streaming-en-0.6b.md](docs/models/nemotron-speech-streaming-en-0.6b.md) | -| Nemotron 3.5 ASR Streaming | `nemotron-3.5-asr-streaming-0.6b` (multilingual, 40 locales) | [docs/models/nemotron-3.5-asr-streaming-0.6b.md](docs/models/nemotron-3.5-asr-streaming-0.6b.md) | -| Multitalker Parakeet Streaming | `multitalker-parakeet-streaming-0.6b-v1` (single-speaker ASR path only) | [docs/models/multitalker-parakeet-streaming-0.6b-v1.md](docs/models/multitalker-parakeet-streaming-0.6b-v1.md) | -| Granite Speech 4 / 4.1 | `granite-4.0-1b-speech`, `granite-speech-4.1-2b{,-plus,-nar}` | [docs/models/granite-speech.md](docs/models/granite-speech.md) | -| Granite Speech 5.0 TurboCTC | `granite-speech-5.0-470m-turboctc` (Apache-2.0), `granite-speech-5.0-470m-turboctc-nc` (non-commercial) | [docs/models/granite-speech-5.0-turboctc.md](docs/models/granite-speech-5.0-turboctc.md) | -| Voxtral | `voxtral-mini-3b-2507`, `voxtral-small-24b-2507` (audio-LLM; transcription + translation) | [docs/models/voxtral.md](docs/models/voxtral.md) | -| Voxtral Realtime | `voxtral-mini-4b-realtime-2602` (streaming audio-LLM) | [docs/models/voxtral-realtime.md](docs/models/voxtral-realtime.md) | -| MedASR | `medasr` (Conformer + CTC, English medical-dictation, gated) | [docs/models/medasr.md](docs/models/medasr.md) | -| MOSS Transcribe-Diarize | `moss-transcribe-diarize` (audio-LLM; English + Chinese ASR with inline speaker diarization) | [docs/models/moss-transcribe-diarize.md](docs/models/moss-transcribe-diarize.md) | -| Sortformer | `diar_streaming_sortformer_4spk-v2.1` (streaming speaker diarizer, up to 4 speakers; no transcription) | [docs/models/diar_streaming_sortformer_4spk-v2.1.md](docs/models/diar_streaming_sortformer_4spk-v2.1.md) | + +| Family | Variants | Available capabilities | Docs | +| --- | --- | --- | --- | +| Canary | `canary-180m-flash`, `canary-1b`, `canary-1b-flash`, `canary-1b-v2` | translate | [docs/models/canary.md](docs/models/canary.md) | +| Canary-Qwen 2.5B | `canary-qwen-2.5b` | - | [docs/models/canary-qwen-2.5b.md](docs/models/canary-qwen-2.5b.md) | +| Cohere Transcribe | `cohere-transcribe-03-2026`, `cohere-transcribe-arabic-07-2026` | - | [docs/models/cohere.md](docs/models/cohere.md) | +| Fun-ASR-Nano | `fun-asr-mlt-nano-2512`, `fun-asr-nano-2512` | - | [docs/models/fun-asr-nano.md](docs/models/fun-asr-nano.md) | +| GigaAM-v3 | `gigaam-v3-ctc`, `gigaam-v3-e2e-ctc`, `gigaam-v3-e2e-rnnt`, `gigaam-v3-rnnt` | token timestamps | [docs/models/gigaam.md](docs/models/gigaam.md) | +| Granite Speech 4 / 4.1 | `granite-4.0-1b-speech`, `granite-speech-4.1-2b`, `granite-speech-4.1-2b-nar`, `granite-speech-4.1-2b-plus` | diarize, translate, word timestamps | [docs/models/granite-speech.md](docs/models/granite-speech.md) | +| Granite Speech 5.0 TurboCTC | `granite-speech-5.0-470m-turboctc`, `granite-speech-5.0-470m-turboctc-nc` | - | [docs/models/granite-speech-5.0-turboctc.md](docs/models/granite-speech-5.0-turboctc.md) | +| MedASR | `medasr` | token timestamps | [docs/models/medasr.md](docs/models/medasr.md) | +| Moonshine | `moonshine-base`, `moonshine-base-ar`, `moonshine-base-ja`, `moonshine-base-ko`, `moonshine-base-uk`, `moonshine-base-vi`, `moonshine-base-zh`, `moonshine-tiny`, `moonshine-tiny-ar`, `moonshine-tiny-ja`, `moonshine-tiny-ko`, `moonshine-tiny-uk`, `moonshine-tiny-vi`, `moonshine-tiny-zh` | - | [docs/models/moonshine.md](docs/models/moonshine.md) | +| Moonshine Streaming | `moonshine-streaming-medium`, `moonshine-streaming-small`, `moonshine-streaming-tiny` | streaming | [docs/models/moonshine-streaming.md](docs/models/moonshine-streaming.md) | +| MOSS-Transcribe-Diarize | `moss-transcribe-diarize` | diarize, segment timestamps | [docs/models/moss-transcribe-diarize.md](docs/models/moss-transcribe-diarize.md) | +| Multitalker Parakeet Streaming 0.6B v1 | `multitalker-parakeet-streaming-0.6b-v1` | diarize, streaming, token timestamps | [docs/models/multitalker-parakeet-streaming-0.6b-v1.md](docs/models/multitalker-parakeet-streaming-0.6b-v1.md) | +| Nemotron 3.5 ASR Streaming 0.6B | `nemotron-3.5-asr-streaming-0.6b` | streaming, token timestamps | [docs/models/nemotron-3.5-asr-streaming-0.6b.md](docs/models/nemotron-3.5-asr-streaming-0.6b.md) | +| Nemotron Speech Streaming EN 0.6B | `nemotron-speech-streaming-en-0.6b` | streaming, token timestamps | [docs/models/nemotron-speech-streaming-en-0.6b.md](docs/models/nemotron-speech-streaming-en-0.6b.md) | +| Parakeet | `parakeet-ctc-0.6b`, `parakeet-ctc-1.1b`, `parakeet-primeline`, `parakeet-rnnt-0.6b`, `parakeet-rnnt-1.1b`, `parakeet-tdt-0.6b-v2`, `parakeet-tdt-0.6b-v3`, `parakeet-tdt-1.1b`, `parakeet-tdt_ctc-1.1b`, `parakeet-tdt_ctc-110m`, `parakeet-unified-en-0.6b` | streaming, token timestamps | [docs/models/parakeet.md](docs/models/parakeet.md) | +| Qwen3-ASR | `qwen3-asr-0.6b`, `qwen3-asr-1.7b` | - | [docs/models/qwen3-asr.md](docs/models/qwen3-asr.md) | +| SenseVoice Small | `sensevoice-small` | - | [docs/models/sensevoice-small.md](docs/models/sensevoice-small.md) | +| Voxtral (2507) | `voxtral-mini-3b-2507`, `voxtral-small-24b-2507` | translate | [docs/models/voxtral.md](docs/models/voxtral.md) | +| Voxtral Realtime (2602) | `voxtral-mini-4b-realtime-2602` | streaming | [docs/models/voxtral-realtime.md](docs/models/voxtral-realtime.md) | +| Whisper | `breeze-asr-25`, `whisper-base`, `whisper-base.en`, `whisper-large`, `whisper-large-v2`, `whisper-large-v3`, `whisper-large-v3-turbo`, `whisper-medium`, `whisper-medium.en`, `whisper-small`, `whisper-small.en`, `whisper-tiny`, `whisper-tiny.en` | segment timestamps, translate | [docs/models/whisper.md](docs/models/whisper.md) | + + +**Speaker diarization models** (no transcription; verified by DER/JER rather than WER): + + +| Family | Variants | Available capabilities | Docs | +| --- | --- | --- | --- | +| Streaming Sortformer Diarizer 4spk v2.1 | `diar_streaming_sortformer_4spk-v2.1` | diarize, streaming | [docs/models/diar_streaming_sortformer_4spk-v2.1.md](docs/models/diar_streaming_sortformer_4spk-v2.1.md) | + Per-variant model cards live under [`docs/models/`](docs/models/). +## Model catalog + +[`catalog/`](catalog/) is the source of truth for model metadata, accuracy, and +performance. Each release includes a queryable +[`catalog.db`](https://github.com/handy-computer/transcribe.cpp/releases/latest/download/catalog.db) +and [`SHA-256 checksum`](https://github.com/handy-computer/transcribe.cpp/releases/latest/download/catalog.db.sha256). +Rebuild it locally with `uv run scripts/catalog/db.py --out catalog.db`. +The exact accuracy and published speed matrices and standard benchmark recipes +required for publication live in +[`catalog/_benchmark_profiles.json`](catalog/_benchmark_profiles.json). Run +`uv run scripts/catalog/check.py --publication-profile` to enforce it; the +ordinary catalog check reports the migration backlog without failing. + +Published tables are generated from it rather than hand-written. A model doc +delegates a region with a marker, and `scripts/catalog/render.py` rewrites +only what sits between the pair: + +```markdown + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | +... + +``` + +The Hugging Face card specs under [`scripts/hf_cards/`](scripts/hf_cards/) +hold editorial copy only (summary, tags, validation pin, prose notes). +`scripts/hf_cards/generate.py` reads the spec and the catalog record together, +so repos, licence, languages, capabilities, the quant table, and per-rig +speedups are never written into a YAML by hand. + +```bash +uv run scripts/catalog/format.py --check # canonical record layout +uv run scripts/catalog/check.py # schema, integrity, pairing +uv run scripts/catalog/render.py # rewrite the marked doc regions +uv run scripts/catalog/render.py --check +``` + ## Build ```bash diff --git a/catalog/_benchmark_profiles.json b/catalog/_benchmark_profiles.json new file mode 100644 index 00000000..ae2fd44b --- /dev/null +++ b/catalog/_benchmark_profiles.json @@ -0,0 +1,72 @@ +{ + "default": "asr-publication-v2", + "profiles": { + "asr-publication-v2": { + "description": "The complete benchmark set published for transcription models. v2: accuracy on L40S; batch_size is the recommended setting for new runs, and a cell is satisfied at any batch size (the row records the one used).", + "accuracy": [ + { + "dataset": "librispeech", + "split": "test-clean", + "languages": "english-if-supported", + "quants": "all-downloads", + "batch_size": 8, + "sort_by_length": true, + "timestamps": "none", + "gpu": "L40S", + "backend": "cuda" + }, + { + "dataset": "fleurs", + "split": "test", + "languages": "supported-intersect-fleurs", + "quants": [ + "Q8_0" + ], + "batch_size": 8, + "sort_by_length": true, + "timestamps": "none", + "gpu": "L40S", + "backend": "cuda" + } + ], + "speed": { + "quants": [ + "Q8_0", + "Q4_K_M" + ], + "samples": [ + "jfk", + "dots" + ], + "iterations": 3, + "warmup": 1, + "targets": [ + { + "machine": "m4-max", + "display": "Apple M4 Max", + "backends": [ + "cpu", + "metal" + ] + }, + { + "machine": "ryzen-4750u", + "display": "AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR)", + "backends": [ + "cpu", + "vulkan" + ], + "cooldown_tctl_c": 55.0 + } + ], + "single_language_samples": { + "pattern": [ + "{lang}-short", + "{lang}-long" + ], + "reason": "A variant that supports exactly one non-English language is benched on that language at the same two clip lengths as jfk/dots. English jfk/dots decode out of distribution on a single-language fine-tune and can loop until the position cap, and whether that happens differs between CPU and GPU, so the figure would not be comparable." + } + } + } + } +} diff --git a/catalog/_schema.json b/catalog/_schema.json new file mode 100644 index 00000000..69728765 --- /dev/null +++ b/catalog/_schema.json @@ -0,0 +1,545 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "transcribe-catalog-v1", + "title": "transcribe.cpp model catalog record", + "description": "", + "type": "object", + "additionalProperties": false, + "required": [ + "schema", "variant", "family", "display_name", "params", "license", + "upstream_repo", "upstream_commit", "published_repo", "languages", + "long_form_strategy", "capabilities", "downloads", "accuracy_benchmarks", + "speed_benchmarks" + ], + "properties": { + "schema": { + "const": "transcribe-catalog-v1" + }, + "variant": { + "type": "string", + "description": "The model slug. Must equal the stem of scripts/hf_cards/.yaml and docs/models/.md.", + "pattern": "^[A-Za-z0-9][A-Za-z0-9._-]*$" + }, + "family": { + "type": "string", + "description": "Loader family (the GGUF's general.architecture)" + }, + "display_name": { + "type": "string" + }, + "params": { + "type": "integer", + "description": "Exact parameter count, summed from GGUF tensor shapes." + }, + "license": { + "type": "object", + "additionalProperties": false, + "required": ["spdx","display"], + "properties": { + "spdx": { + "type": "string" + }, + "display": { + "type": "string" + }, + "name": { + "type": "string", + "description": "HF card license_name, for licences with no SPDX id (spdx \"other\")." + }, + "link": { + "type": "string", + "description": "HF card license_link -- URL of the full licence text." + } + } + }, + "upstream_repo": { + "type": "string", + "description": "The upstream model this port derives from." + }, + "upstream_commit": { + "type": "string", + "pattern": "^[0-9a-f]{7}$", + "description": "The upstream revision used to produce the published GGUFs, as the 7-character short SHA." + }, + "docs_page": { + "type": "string", + "description": "Filename under docs/models/ of the page that documents this variant: its own page, or the family page whose roll-up lists it. Drives the family roll-ups and the root README index." + }, + "published_repo": { + "type": ["string","null"], + "description": "Our GGUF repo. null until published." + }, + "languages": { + "type": "array", + "items": { + "type": "string" + }, + "minItems": 1, + "description": "Languages this port transcribes, in the spellings it advertises." + }, + "language_tag_form": { + "enum": ["bare-bcp47","locale","mixed"], + "description": "What the --language argument accepts. Needed because a model can advertise bare codes while validating hints against locales (nemotron rejects the advertised spelling with UNSUPPORTED_LANGUAGE)." + }, + "language_aliases": { + "type": "object", + "description": "{advertised spelling: canonical code}, for languages this model names differently from everyone else (whisper says tl and no where others say fil and nb).", + "additionalProperties": { + "type": "string" + } + }, + "benchmark_exceptions": { + "type": "array", + "description": "Narrow, reviewed exclusions from the repository publication profile. An exception removes only cells matched explicitly here; a free-form statement cannot satisfy a publication gate.", + "items": { + "type": "object", + "required": ["kind","match","reason"], + "additionalProperties": false, + "properties": { + "kind": { + "enum": ["accuracy","speed"] + }, + "match": { + "type": "object", + "minProperties": 1, + "additionalProperties": { + "type": "string" + }, + "description": "Cell fields to match. '*' is an explicit wildcard. Accuracy fields are dataset/split/language/quant; speed fields are machine/backend/quant/sample." + }, + "reason": { + "type": "string", + "minLength": 10 + } + } + } + }, + "encoder_window_s": { + "type": ["number","null"], + "description": "Fixed encoder window, if the architecture has one (whisper 30 s)." + }, + "long_form_strategy": { + "enum": ["chunked-unbounded","hard-cap","soft-window"], + "description": "Which bucket in docs/input-limits.md this family falls into." + }, + "max_audio_s": { + "type": ["number","null"], + "description": "Usable ceiling, mirroring transcribe_capabilities::max_audio_ms. null = no practical limit." + }, + "max_output_tokens": { + "type": ["integer","null"] + }, + "capabilities": { + "type": "object", + "additionalProperties": false, + "description": "What this port supports; every capability is listed on every record so all models render the same table rows. `supported` is read from the GGUF capability surface by sync_capabilities.py, never hand-written; `verified` means we ran it and watched it work, set by the Stage 8 gate.", + "properties": { + "transcribe": { + "$ref": "#/$defs/capability", + "description": "Speech to text in the source language." + }, + "translate": { + "$ref": "#/$defs/capability", + "description": "Speech in one language to text in another. Mirrors what the loader reads from stt.translation.target_languages / .pairs.", + "properties": { + "targets": { + "type": ["array","null"], + "items": { + "type": "string" + }, + "description": "Targets reachable from any supported source. null when the model gates on explicit pairs." + }, + "pairs": { + "type": ["array","null"], + "items": { + "oneOf": [ + { + "type": "string", + "description": "The GGUF's own form, e.g. 'en>fr'." + }, + { + "type": "object", + "required": ["from","to"], + "properties": { + "from": { + "type": "string" + }, + "to": { + "type": "string" + } + } + } + ] + }, + "description": "Explicit source->target pairs, when the model accepts only a fixed set. Carried in whichever form stt.translation.pairs uses." + } + } + }, + "lang_detect": { + "$ref": "#/$defs/capability", + "description": "Identifies the spoken language without a hint. A model without this needs a separate LID step in front of it.", + "properties": { + "over": { + "type": ["array","null"], + "items": { + "type": "string" + }, + "description": "The set detection works over. null means the model's full language list." + } + } + }, + "timestamps": { + "$ref": "#/$defs/capability", + "description": "Timestamp granularities this port emits, mirroring transcribe_capabilities::max_timestamp_kind. Not always what upstream offers.", + "properties": { + "granularities": { + "type": "array", + "minItems": 1, + "items": { + "enum": ["segment","word","token"] + } + } + } + }, + "streaming": { + "$ref": "#/$defs/capability", + "description": "Incremental transcription over a live stream; the shipped latency presets are the capability.", + "properties": { + "mode": { + "type": ["string","null"], + "description": "buffered | cache-aware | native." + }, + "presets": {"type":["array","null"],"items":{"type":"object","required":["name"],"additionalProperties":false,"properties":{"name":{"type":"string"},"left_ms":{"type":["number","null"]},"chunk_ms":{"type":["number","null"]},"right_ms":{"type":["number","null"]},"lookahead_ms":{"type":["number","null"],"description":"Audio that must be buffered before a chunk's tokens emit. The number a user feels."},"is_default":{"type":["boolean","null"]}}}} + } + }, + "diarize": { + "$ref": "#/$defs/capability", + "description": "Speaker-attributed output. A diarizing model emits turn markup plain references do not contain, so transcription scoring must remove that markup.", + "properties": { + "max_speakers": { + "type": ["integer","null"] + }, + "granularity": { + "type": ["string","null"], + "description": "turn | frame." + }, + "markup": { + "type": ["string","null"], + "description": "Literal shape of the attribution, e.g. '[start][Sxx]text[end]'." + } + } + }, + "batching": { + "$ref": "#/$defs/capability", + "description": "Explicit parallel run_batch() fast path, byte-identical to single-stream." + }, + "punctuation": { + "$ref": "#/$defs/capability", + "description": "Emits punctuation. Reserved and deferred: absent means not yet determined, which differs from supported:false." + }, + "casing": { + "$ref": "#/$defs/capability", + "description": "Emits mixed case. Reserved and deferred: absent means not yet determined, which differs from supported:false." + }, + "itn": { + "$ref": "#/$defs/capability", + "description": "Inverse text normalisation ('25' rather than 'twenty five'). Reserved and deferred: absent means not yet determined, which differs from supported:false." + } + }, + "required": [ + "transcribe", "translate", "lang_detect", "timestamps", "streaming", + "diarize", "batching" + ] + }, + "downloads": { + "type": "array", + "minItems": 1, + "description": "The GGUF files we publish.", + "items": { + "type": "object", + "additionalProperties": false, + "required": ["quant","filename","size_bytes"], + "properties": { + "quant": { + "type": "string" + }, + "filename": { + "type": "string", + "description": "Cached from the Hub listing and verified against it, not authored." + }, + "size_bytes": { + "type": "integer", + "minimum": 1, + "description": "Exact blob size; prose units are a rendering concern." + } + } + } + }, + "accuracy_benchmarks": { + "type": "array", + "description": "Published measurements only, on full dataset splits, one row per dataset/split/language/quant/metric/timestamps/scoring/mode cell. Batch size is recipe metadata; prefer the profile recipe when selecting among runs.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "dataset", "split", "language", "quant", "metric", "err_pct", + "ci95", "n_utts", "batch_size", "timestamps", "engine_sha" + ], + "properties": { + "dataset": { + "type": "string", + "description": "Dataset id as scripts/wer/ingest.py knows it." + }, + "split": { + "type": "string", + "description": "The full split measured. Named and reproducible; never an ad-hoc selection." + }, + "language": { + "type": "string", + "description": "Canonical dataset code, not necessarily the model's own spelling." + }, + "language_hint": { + "type": ["string","null"], + "description": "Actual language spelling passed to the runtime. This differs from language for locale-conditioned models and aliases such as tl/fil." + }, + "backend": { + "enum": ["cpu","metal","vulkan","cuda",null], + "description": "Backend that generated the hypotheses." + }, + "quant": { + "type": "string", + "description": "MUST appear in downloads[]." + }, + "metric": { + "enum": ["wer","cer","der","cpwer","accuracy"], + "description": "CER for zh/yue/ja/ko/th/km/lo/my; der/cpwer for diarization; accuracy for LID." + }, + "err_pct": { + "type": "number", + "minimum": 0 + }, + "ci95": { + "type": "array", + "minItems": 2, + "maxItems": 2, + "items": { + "type": ["number","null"] + }, + "description": "Bootstrap 95% interval, percent. A null pair when none was produced -- never 0.0." + }, + "n_utts": { + "type": "integer", + "minimum": 1 + }, + "batch_size": { + "type": ["integer","null"], + "minimum": 1, + "description": "Decode batch size. null when the surviving publication does not record it." + }, + "timestamps": { + "enum": ["none","segment","word","token",null], + "description": "Timestamp mode used while decoding. null when the surviving publication does not record it." + }, + "engine_sha": { + "type": ["string","null"], + "description": "transcribe.cpp commit that produced the hypotheses." + }, + "measurement_provenance": { + "enum": ["legacy-published"], + "description": "Present only for a pre-profile result migrated from a published table whose original engine SHA is unavailable; never a guessed commit." + }, + "measured_on": { + "type": ["string","null"], + "format": "date" + }, + "errors": { + "type": "object", + "additionalProperties": false, + "properties": { + "sub": { + "type": "integer" + }, + "del": { + "type": "integer" + }, + "ins": { + "type": "integer" + } + }, + "description": "Error composition. Deletion-heavy means dropping audio; insertion-heavy means hallucinating." + }, + "empty_hyp": { + "type": ["integer","null"], + "description": "Utterances that returned nothing. A generation failure, not a transcription error." + }, + "utts_over_50pct": { + "type": ["integer","null"], + "description": "Catastrophic-tail count." + }, + "publication_profile": { + "type": ["string","null"], + "description": "Id of the catalog/_benchmark_profiles.json profile whose sweep produced this row. Names the recipe; absent on rows published before profiles existed." + }, + "scoring": { + "type": "string", + "description": "A scoring step beyond the standard normalizer, e.g. \"opencc-t2s\" for Traditional-to-Simplified folding of both sides. Rows with a scoring value are a separate result set from the profile cell." + }, + "mode": { + "type": "string", + "description": "Decoding mode for models that publish one metric under several modes (multitalker \"kernel\" / \"masked\" cpWER). Rows with a mode are a separate result set." + } + } + } + }, + "headline_benchmark": { + "type": ["object","null"], + "description": "Which accuracy row-set the variant publishes as its headline number, the metric column of its doc and HF card; a renderer selects matching accuracy_benchmarks rows, one per quant. batch_size or timestamps may be null as a wildcard for a legacy table with mixed recipes; a null object means no headline column.", + "required": ["dataset","split","language","metric","batch_size","timestamps"], + "additionalProperties": false, + "properties": { + "dataset": { + "type": "string" + }, + "split": { + "type": "string" + }, + "language": { + "type": "string" + }, + "metric": { + "enum": ["wer","cer","der","cpwer","accuracy"] + }, + "batch_size": { + "type": ["integer","null"], + "minimum": 1 + }, + "timestamps": { + "enum": ["none","segment","word","token",null] + } + } + }, + "speed_benchmarks": { + "type": "array", + "description": "Published measurements only, one row per quant/sample/machine/backend cell selected by catalog/_benchmark_profiles.json. The profile also owns iteration and warmup counts and thermal preconditions; rows carry the measured identity and result.", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "machine", "backend", "quant", "sample", "sample_duration_s", + "total_ms", "xrt_compute", "engine_sha" + ], + "properties": { + "machine": { + "type": "string", + "description": "Canonical machine slug, as scripts/bench/run.py derives it from the CPU model string. One slug per physical machine." + }, + "backend": { + "enum": ["cpu","metal","vulkan","cuda"] + }, + "quant": { + "type": "string", + "description": "MUST appear in downloads[]." + }, + "sample": { + "type": "string", + "description": "Sample id; the set is open, so a new sample means new rows, not a schema change." + }, + "sample_duration_s": { + "type": "number", + "description": "Carried per cell so nothing downstream assumes a fixed sample set or has to look the duration up." + }, + "total_ms": { + "type": ["number","null"], + "description": "mel + encode + decode, mean over iters after warmup. null for legacy rows where only rounded xRT survives." + }, + "xrt_compute": { + "type": "number", + "description": "sample_duration_s / total. Stored rather than derived because it is the published figure and must not drift with rounding." + }, + "wall_ms": { + "type": ["number","null"], + "description": "End-to-end wall time for the same run, mean over iters after warmup. Carried because the granite audio-LLMs publish wall rather than compute time." + }, + "xrt_wall": { + "type": ["number","null"], + "description": "sample_duration_s / wall. Stored, not derived, for the same reason as xrt_compute." + }, + "load_ms": { + "type": ["number","null"] + }, + "mel_ms": { + "type": ["number","null"], + "description": "Stage split (mel/encode/decode) so a regression is attributable." + }, + "encode_ms": { + "type": ["number","null"] + }, + "decode_ms": { + "type": ["number","null"] + }, + "engine_sha": { + "type": ["string","null"], + "description": "A perf number without a build is not reproducible." + }, + "measurement_provenance": { + "enum": ["legacy-published"], + "description": "Present only for a pre-profile result migrated from a published table whose original engine SHA is unavailable. The stored xRT remains publishable; detailed timing awaits re-benchmarking." + }, + "measured_on": { + "type": ["string","null"], + "format": "date" + }, + "thermal_gated": { + "type": ["boolean","null"], + "description": "Whether the machine's thermal precondition was met before this block; ungated runs can differ by tens of percent." + }, + "publication_profile": { + "type": ["string","null"], + "description": "Id of the profile whose bench run produced this row; it pins iterations, warmup, samples and thermal policy. Absent on rows measured before profile stamping." + } + } + } + } + }, + "$defs": { + "capability": { + "type": "object", + "required": ["supported"], + "description": "Shared envelope. `verified` is required when supported is true and omitted otherwise.", + "properties": { + "supported": { + "type": "boolean", + "description": "Whether this port has the capability." + }, + "verified": { + "type": "boolean", + "description": "Whether we ran it and observed it working. Set by the Stage 8 gate." + }, + "note": { + "type": "string", + "description": "Only where a consumer would otherwise be surprised -- a gap against what upstream advertises, or a caveat on how it behaves." + } + }, + "allOf": [ + { + "if": { + "properties": { + "supported": { + "const": true + } + }, + "required": ["supported"] + }, + "then": { + "required": ["verified"] + }, + "else": { + "not": { + "required": ["verified"] + } + } + } + ] + } + } +} diff --git a/catalog/breeze-asr-25.json b/catalog/breeze-asr-25.json new file mode 100644 index 00000000..01975dfa --- /dev/null +++ b/catalog/breeze-asr-25.json @@ -0,0 +1,76 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "breeze-asr-25", + "family": "whisper", + "display_name": "breeze-asr-25", + "params": 1543321440, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "MediaTek-Research/Breeze-ASR-25", + "upstream_commit": "cffe7cc", + "published_repo": "handy-computer/Breeze-ASR-25-gguf", + "docs_page": "whisper.md", + "languages": ["zh","en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Breeze-ASR-25-BF16.gguf","size_bytes":3096013408}, + {"quant":"F16","filename":"Breeze-ASR-25-F16.gguf","size_bytes":3106458208}, + {"quant":"Q8_0","filename":"Breeze-ASR-25-Q8_0.gguf","size_bytes":1667964224}, + {"quant":"Q6_K","filename":"Breeze-ASR-25-Q6_K.gguf","size_bytes":1296353280}, + {"quant":"Q5_K_M","filename":"Breeze-ASR-25-Q5_K_M.gguf","size_bytes":1160366080}, + {"quant":"Q4_K_M","filename":"Breeze-ASR-25-Q4_K_M.gguf","size_bytes":996526080} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.14,"ci95":[3.7,4.61],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":416,"del":96,"ins":90},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":8.1,"ci95":[7.28,8.94],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1239,"del":1415,"ins":233},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":2.29,"ci95":[null,null],"n_utts":2620,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.29,"ci95":[null,null],"n_utts":2620,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.27,"ci95":[null,null],"n_utts":2620,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.29,"ci95":[null,null],"n_utts":2620,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.25,"ci95":[null,null],"n_utts":2620,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.26,"ci95":[null,null],"n_utts":2620,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"BF16","metric":"cer","err_pct":8.12,"ci95":[null,null],"n_utts":945,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","scoring":"opencc-t2s"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"F16","metric":"cer","err_pct":8.11,"ci95":[null,null],"n_utts":945,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","scoring":"opencc-t2s"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":8.1,"ci95":[null,null],"n_utts":945,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","scoring":"opencc-t2s"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q6_K","metric":"cer","err_pct":8.12,"ci95":[null,null],"n_utts":945,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","scoring":"opencc-t2s"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q5_K_M","metric":"cer","err_pct":8.12,"ci95":[null,null],"n_utts":945,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","scoring":"opencc-t2s"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q4_K_M","metric":"cer","err_pct":8.08,"ci95":[null,null],"n_utts":945,"batch_size":1,"timestamps":"segment","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","scoring":"opencc-t2s"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "segment" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":8061.8,"xrt_compute":4.38,"load_ms":257.2,"mel_ms":4.8,"encode_ms":2823.8,"decode_ms":5233.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8062.3,"xrt_wall":4.38,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3894.9,"xrt_compute":2.82,"load_ms":232.8,"mel_ms":4.2,"encode_ms":2871.2,"decode_ms":1019.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3895.4,"xrt_wall":2.82,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7830.2,"xrt_compute":4.51,"load_ms":422.6,"mel_ms":4.8,"encode_ms":2684.0,"decode_ms":5141.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7830.7,"xrt_wall":4.51,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3676.6,"xrt_compute":2.99,"load_ms":361.8,"mel_ms":4.2,"encode_ms":2712.1,"decode_ms":960.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3677.1,"xrt_wall":2.99,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1412.7,"xrt_compute":25.01,"load_ms":279.7,"mel_ms":5.7,"encode_ms":271.1,"decode_ms":1135.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1412.9,"xrt_wall":25.01,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":538.5,"xrt_compute":20.43,"load_ms":248.8,"mel_ms":5.0,"encode_ms":269.9,"decode_ms":263.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":538.7,"xrt_wall":20.42,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1481.2,"xrt_compute":23.85,"load_ms":449.1,"mel_ms":5.2,"encode_ms":272.9,"decode_ms":1203.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1481.4,"xrt_wall":23.85,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":530.7,"xrt_compute":20.73,"load_ms":381.1,"mel_ms":4.1,"encode_ms":260.2,"decode_ms":266.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":530.9,"xrt_wall":20.72,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":29374.8,"xrt_compute":1.2,"load_ms":709.5,"mel_ms":36.5,"encode_ms":9564.5,"decode_ms":19773.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":29382.6,"xrt_wall":1.2,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":13603.0,"xrt_compute":0.81,"load_ms":1125.8,"mel_ms":29.4,"encode_ms":9548.1,"decode_ms":4025.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13610.8,"xrt_wall":0.81,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":30922.9,"xrt_compute":1.14,"load_ms":1167.6,"mel_ms":33.1,"encode_ms":9831.3,"decode_ms":21058.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":30931.0,"xrt_wall":1.14,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":14074.1,"xrt_compute":0.78,"load_ms":1931.4,"mel_ms":32.7,"encode_ms":9840.0,"decode_ms":4201.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14082.3,"xrt_wall":0.78,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":14873.8,"xrt_compute":2.38,"load_ms":490.9,"mel_ms":41.7,"encode_ms":4513.2,"decode_ms":10318.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14874.6,"xrt_wall":2.38,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":6560.5,"xrt_compute":1.68,"load_ms":990.1,"mel_ms":45.5,"encode_ms":4494.8,"decode_ms":2020.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6561.3,"xrt_wall":1.68,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":15017.0,"xrt_compute":2.35,"load_ms":645.4,"mel_ms":43.7,"encode_ms":4427.2,"decode_ms":10546.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15018.1,"xrt_wall":2.35,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":6504.2,"xrt_compute":1.69,"load_ms":1422.2,"mel_ms":41.8,"encode_ms":4408.3,"decode_ms":2054.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6504.9,"xrt_wall":1.69,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/canary-180m-flash.json b/catalog/canary-180m-flash.json new file mode 100644 index 00000000..a77f6bde --- /dev/null +++ b/catalog/canary-180m-flash.json @@ -0,0 +1,72 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "canary-180m-flash", + "family": "canary", + "display_name": "canary-180m-flash", + "params": 189083776, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/canary-180m-flash", + "upstream_commit": "b12ab41", + "published_repo": "handy-computer/canary-180m-flash-gguf", + "docs_page": "canary.md", + "languages": ["en","de","es","fr"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en","de","es","fr"],"pairs":["en>de","de>en","en>es","es>en","en>fr","fr>en"]}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"canary-180m-flash-F32.gguf","size_bytes":756498112}, + {"quant":"F16","filename":"canary-180m-flash-F16.gguf","size_bytes":381632192}, + {"quant":"Q8_0","filename":"canary-180m-flash-Q8_0.gguf","size_bytes":218447552}, + {"quant":"Q6_K","filename":"canary-180m-flash-Q6_K.gguf","size_bytes":176291520}, + {"quant":"Q5_K_M","filename":"canary-180m-flash-Q5_K_M.gguf","size_bytes":158704320}, + {"quant":"Q4_K_M","filename":"canary-180m-flash-Q4_K_M.gguf","size_bytes":139223744} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":7.33,"ci95":[6.67,8.0],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":977,"del":182,"ins":212},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.98,"ci95":[5.26,6.78],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":548,"del":227,"ins":95},"empty_hyp":4,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":6.54,"ci95":[5.92,7.18],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":959,"del":154,"ins":412},"empty_hyp":1,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":8.53,"ci95":[7.78,9.34],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1049,"del":205,"ins":281},"empty_hyp":1,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.94,"ci95":[1.78,2.11],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":763,"del":135,"ins":129},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.94,"ci95":[1.78,2.11],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":763,"del":135,"ins":129},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.93,"ci95":[1.77,2.1],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":761,"del":138,"ins":126},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.93,"ci95":[1.77,2.1],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":763,"del":138,"ins":123},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.9,"ci95":[1.74,2.07],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":750,"del":137,"ins":122},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.93,"ci95":[1.76,2.1],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":769,"del":139,"ins":115},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":490.1,"xrt_compute":72.09,"load_ms":73.2,"mel_ms":2.1,"encode_ms":355.6,"decode_ms":132.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":494.7,"xrt_wall":71.43,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":129.5,"xrt_compute":84.94,"load_ms":69.6,"mel_ms":0.7,"encode_ms":103.5,"decode_ms":25.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":131.6,"xrt_wall":83.6,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":492.7,"xrt_compute":71.72,"load_ms":77.2,"mel_ms":2.1,"encode_ms":331.9,"decode_ms":158.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":497.7,"xrt_wall":70.99,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":127.8,"xrt_compute":86.1,"load_ms":81.2,"mel_ms":0.8,"encode_ms":97.1,"decode_ms":29.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":130.0,"xrt_wall":84.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":240.4,"xrt_compute":146.98,"load_ms":59.7,"mel_ms":2.3,"encode_ms":67.3,"decode_ms":170.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":252.9,"xrt_wall":139.69,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":59.2,"xrt_compute":185.69,"load_ms":59.2,"mel_ms":0.9,"encode_ms":25.1,"decode_ms":33.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":64.2,"xrt_wall":171.21,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":261.8,"xrt_compute":134.95,"load_ms":73.4,"mel_ms":2.3,"encode_ms":66.9,"decode_ms":192.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":274.1,"xrt_wall":128.88,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":64.0,"xrt_compute":171.97,"load_ms":73.3,"mel_ms":0.9,"encode_ms":25.0,"decode_ms":38.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":69.1,"xrt_wall":159.11,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1794.0,"xrt_compute":19.69,"load_ms":225.5,"mel_ms":58.4,"encode_ms":1049.5,"decode_ms":686.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1854.4,"xrt_wall":19.05,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":420.9,"xrt_compute":26.13,"load_ms":176.9,"mel_ms":18.6,"encode_ms":291.8,"decode_ms":110.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":441.4,"xrt_wall":24.92,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1928.0,"xrt_compute":18.33,"load_ms":306.4,"mel_ms":57.8,"encode_ms":1003.1,"decode_ms":867.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1987.7,"xrt_wall":17.78,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":437.8,"xrt_compute":25.13,"load_ms":221.9,"mel_ms":19.3,"encode_ms":270.0,"decode_ms":148.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":456.5,"xrt_wall":24.09,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1066.1,"xrt_compute":33.14,"load_ms":153.6,"mel_ms":63.5,"encode_ms":498.8,"decode_ms":503.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1087.8,"xrt_wall":32.48,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":283.6,"xrt_compute":38.78,"load_ms":86.8,"mel_ms":20.4,"encode_ms":164.7,"decode_ms":98.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":293.4,"xrt_wall":37.49,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1164.0,"xrt_compute":30.35,"load_ms":213.1,"mel_ms":61.9,"encode_ms":490.9,"decode_ms":611.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1185.5,"xrt_wall":29.8,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":307.1,"xrt_compute":35.82,"load_ms":113.2,"mel_ms":22.1,"encode_ms":163.2,"decode_ms":121.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":316.6,"xrt_wall":34.75,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/canary-1b-flash.json b/catalog/canary-1b-flash.json new file mode 100644 index 00000000..3a83eb85 --- /dev/null +++ b/catalog/canary-1b-flash.json @@ -0,0 +1,72 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "canary-1b-flash", + "family": "canary", + "display_name": "canary-1b-flash", + "params": 890042496, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/canary-1b-flash", + "upstream_commit": "a9a55e0", + "published_repo": "handy-computer/canary-1b-flash-gguf", + "docs_page": "canary.md", + "languages": ["en","de","es","fr"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en","de","es","fr"],"pairs":["en>de","de>en","en>es","es>en","en>fr","fr>en"]}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"canary-1b-flash-F32.gguf","size_bytes":3560372000}, + {"quant":"F16","filename":"canary-1b-flash-F16.gguf","size_bytes":1785657120}, + {"quant":"Q8_0","filename":"canary-1b-flash-Q8_0.gguf","size_bytes":1048131360}, + {"quant":"Q6_K","filename":"canary-1b-flash-Q6_K.gguf","size_bytes":857603872}, + {"quant":"Q5_K_M","filename":"canary-1b-flash-Q5_K_M.gguf","size_bytes":769563424}, + {"quant":"Q4_K_M","filename":"canary-1b-flash-Q4_K_M.gguf","size_bytes":677141280} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":6.13,"ci95":[5.5,6.77],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":824,"del":162,"ins":162},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.75,"ci95":[4.14,5.51],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":408,"del":225,"ins":59},"empty_hyp":4,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":6.73,"ci95":[5.99,7.54],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":911,"del":242,"ins":417},"empty_hyp":6,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":7.22,"ci95":[6.47,8.05],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":854,"del":150,"ins":295},"empty_hyp":1,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.62,"ci95":[1.41,1.86],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":569,"del":196,"ins":96},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.62,"ci95":[1.41,1.86],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":569,"del":196,"ins":96},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.62,"ci95":[1.41,1.86],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":568,"del":196,"ins":96},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.65,"ci95":[1.42,1.9],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":562,"del":214,"ins":97},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.64,"ci95":[1.43,1.87],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":581,"del":189,"ins":99},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.59,"ci95":[1.41,1.8],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":586,"del":160,"ins":98},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1486.0,"xrt_compute":23.78,"load_ms":233.8,"mel_ms":2.1,"encode_ms":1346.7,"decode_ms":137.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1492.4,"xrt_wall":23.67,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":429.5,"xrt_compute":25.61,"load_ms":222.5,"mel_ms":0.8,"encode_ms":402.4,"decode_ms":26.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":432.5,"xrt_wall":25.43,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1443.6,"xrt_compute":24.47,"load_ms":293.1,"mel_ms":2.2,"encode_ms":1274.2,"decode_ms":167.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1450.3,"xrt_wall":24.36,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":416.7,"xrt_compute":26.4,"load_ms":290.7,"mel_ms":0.8,"encode_ms":383.8,"decode_ms":32.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":419.8,"xrt_wall":26.2,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":315.5,"xrt_compute":112.0,"load_ms":175.4,"mel_ms":2.3,"encode_ms":142.7,"decode_ms":170.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":328.8,"xrt_wall":107.47,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":91.8,"xrt_compute":119.77,"load_ms":169.8,"mel_ms":0.9,"encode_ms":57.9,"decode_ms":33.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":98.0,"xrt_wall":112.27,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":333.9,"xrt_compute":105.82,"load_ms":239.4,"mel_ms":2.3,"encode_ms":139.8,"decode_ms":191.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":347.4,"xrt_wall":101.7,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":93.5,"xrt_compute":117.69,"load_ms":239.9,"mel_ms":0.9,"encode_ms":56.2,"decode_ms":36.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":99.7,"xrt_wall":110.29,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4538.1,"xrt_compute":7.79,"load_ms":1080.3,"mel_ms":58.9,"encode_ms":3772.2,"decode_ms":707.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4607.2,"xrt_wall":7.67,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1104.2,"xrt_compute":9.96,"load_ms":804.4,"mel_ms":17.9,"encode_ms":975.2,"decode_ms":111.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1125.6,"xrt_wall":9.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4630.0,"xrt_compute":7.63,"load_ms":1508.8,"mel_ms":58.4,"encode_ms":3687.8,"decode_ms":883.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4700.3,"xrt_wall":7.52,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1092.8,"xrt_compute":10.07,"load_ms":1063.5,"mel_ms":18.2,"encode_ms":926.9,"decode_ms":147.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1113.9,"xrt_wall":9.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2239.4,"xrt_compute":15.78,"load_ms":556.5,"mel_ms":66.3,"encode_ms":1653.9,"decode_ms":519.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2262.4,"xrt_wall":15.62,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":693.5,"xrt_compute":15.86,"load_ms":241.6,"mel_ms":20.1,"encode_ms":569.2,"decode_ms":104.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":709.0,"xrt_wall":15.51,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2345.9,"xrt_compute":15.06,"load_ms":871.4,"mel_ms":64.8,"encode_ms":1648.4,"decode_ms":632.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2367.6,"xrt_wall":14.92,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":691.8,"xrt_compute":15.9,"load_ms":376.8,"mel_ms":20.8,"encode_ms":550.8,"decode_ms":120.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":707.2,"xrt_wall":15.55,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/canary-1b-v2.json b/catalog/canary-1b-v2.json new file mode 100644 index 00000000..cd46d6b7 --- /dev/null +++ b/catalog/canary-1b-v2.json @@ -0,0 +1,97 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "canary-1b-v2", + "family": "canary", + "display_name": "canary-1b-v2", + "params": 980046848, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/canary-1b-v2", + "upstream_commit": "87bc526", + "published_repo": "handy-computer/canary-1b-v2-gguf", + "docs_page": "canary.md", + "languages": [ + "bg", "hr", "cs", "da", "nl", "en", "et", "fi", "fr", "de", "el", "hu", + "it", "lv", "lt", "mt", "pl", "pt", "ro", "sk", "sl", "es", "sv", "ru", + "uk" + ], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["bg","hr","cs","da","nl","en","et","fi","fr","de","el","hu","it","lt","mt","pl","pt","ro","sk","sl","es","sv","ru","uk"],"pairs":["en>bg","bg>en","en>hr","hr>en","en>cs","cs>en","en>da","da>en","en>nl","nl>en","en>et","et>en","en>fi","fi>en","en>fr","fr>en","en>de","de>en","en>el","el>en","en>hu","hu>en","en>it","it>en","en>lt","lt>en","en>mt","mt>en","en>pl","pl>en","en>pt","pt>en","en>ro","ro>en","en>sk","sk>en","en>sl","sl>en","en>es","es>en","en>sv","sv>en","en>ru","ru>en","en>uk","uk>en"]}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"canary-1b-v2-F32.gguf","size_bytes":3920657120}, + {"quant":"F16","filename":"canary-1b-v2-F16.gguf","size_bytes":1966111456}, + {"quant":"Q8_0","filename":"canary-1b-v2-Q8_0.gguf","size_bytes":1144290016}, + {"quant":"Q6_K","filename":"canary-1b-v2-Q6_K.gguf","size_bytes":931986144}, + {"quant":"Q5_K_M","filename":"canary-1b-v2-Q5_K_M.gguf","size_bytes":836664032}, + {"quant":"Q4_K_M","filename":"canary-1b-v2-Q4_K_M.gguf","size_bytes":735476448} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":9.22,"ci95":[8.43,10.04],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1026,"del":172,"ins":108},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":8.56,"ci95":[7.91,9.23],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":960,"del":113,"ins":99},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":11.34,"ci95":[10.73,11.93],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1746,"del":268,"ins":271},"empty_hyp":1,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":4.46,"ci95":[4.05,4.89],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":646,"del":128,"ins":60},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":26.02,"ci95":[25.01,26.98],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3740,"del":120,"ins":88},"empty_hyp":0,"utts_over_50pct":38,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.47,"ci95":[4.03,4.95],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":450,"del":129,"ins":71},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.1,"ci95":[2.78,3.44],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":547,"del":100,"ins":75},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":12.72,"ci95":[12.04,13.43],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1423,"del":170,"ins":299},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":8.86,"ci95":[8.21,9.55],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1045,"del":113,"ins":153},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":5.09,"ci95":[4.61,5.59],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":681,"del":133,"ins":102},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":8.4,"ci95":[7.81,9.01],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1192,"del":162,"ins":137},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":13.06,"ci95":[12.39,13.8],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1731,"del":235,"ins":264},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":3.1,"ci95":[2.77,3.46],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":511,"del":65,"ins":83},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":13.45,"ci95":[12.74,14.16],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2046,"del":130,"ins":129},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":10.41,"ci95":[9.79,11.09],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1315,"del":120,"ins":170},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":19.75,"ci95":[17.99,22.38],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3410,"del":377,"ins":604},"empty_hyp":0,"utts_over_50pct":25,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":6.28,"ci95":[5.63,6.98],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":389,"del":77,"ins":61},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":6.88,"ci95":[6.32,7.44],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":800,"del":103,"ins":87},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":4.5,"ci95":[4.04,5.01],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":705,"del":106,"ins":156},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":6.87,"ci95":[6.35,7.38],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1076,"del":209,"ins":143},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":7.83,"ci95":[7.24,8.48],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":992,"del":89,"ins":102},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":6.84,"ci95":[6.14,7.54],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":797,"del":170,"ins":76},"empty_hyp":3,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":12.76,"ci95":[11.35,14.61],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1500,"del":191,"ins":417},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":9.74,"ci95":[9.06,10.44],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1148,"del":211,"ins":139},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":10.58,"ci95":[9.87,11.27],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1283,"del":138,"ins":99},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.92,"ci95":[1.78,2.08],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":757,"del":167,"ins":96},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.92,"ci95":[1.78,2.08],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":757,"del":167,"ins":96},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.91,"ci95":[1.76,2.07],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":751,"del":166,"ins":97},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.94,"ci95":[1.79,2.1],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":755,"del":171,"ins":101},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.93,"ci95":[1.78,2.08],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":759,"del":169,"ins":93},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.91,"ci95":[1.76,2.07],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":748,"del":163,"ins":102},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1555.9,"xrt_compute":22.71,"load_ms":263.1,"mel_ms":2.1,"encode_ms":1367.4,"decode_ms":186.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1562.0,"xrt_wall":22.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":442.4,"xrt_compute":24.86,"load_ms":232.6,"mel_ms":0.8,"encode_ms":404.2,"decode_ms":37.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":445.2,"xrt_wall":24.71,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1493.6,"xrt_compute":23.66,"load_ms":297.9,"mel_ms":2.2,"encode_ms":1278.7,"decode_ms":212.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1499.5,"xrt_wall":23.56,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":415.1,"xrt_compute":26.5,"load_ms":300.3,"mel_ms":0.8,"encode_ms":372.1,"decode_ms":42.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":417.9,"xrt_wall":26.32,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":362.1,"xrt_compute":97.57,"load_ms":185.3,"mel_ms":2.3,"encode_ms":142.5,"decode_ms":217.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":376.1,"xrt_wall":93.95,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":104.1,"xrt_compute":105.66,"load_ms":184.6,"mel_ms":0.9,"encode_ms":58.1,"decode_ms":45.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":110.1,"xrt_wall":99.92,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":384.7,"xrt_compute":91.84,"load_ms":260.1,"mel_ms":2.3,"encode_ms":139.5,"decode_ms":243.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":398.4,"xrt_wall":88.69,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":105.1,"xrt_compute":104.66,"load_ms":264.5,"mel_ms":0.8,"encode_ms":56.2,"decode_ms":48.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":111.3,"xrt_wall":98.85,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4773.3,"xrt_compute":7.4,"load_ms":1135.5,"mel_ms":60.1,"encode_ms":3784.4,"decode_ms":928.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4840.3,"xrt_wall":7.3,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1153.4,"xrt_compute":9.54,"load_ms":838.1,"mel_ms":18.7,"encode_ms":978.5,"decode_ms":156.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1175.4,"xrt_wall":9.36,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4969.0,"xrt_compute":7.11,"load_ms":1638.7,"mel_ms":58.3,"encode_ms":3731.8,"decode_ms":1178.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5036.4,"xrt_wall":7.02,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1153.6,"xrt_compute":9.54,"load_ms":1121.2,"mel_ms":18.1,"encode_ms":925.8,"decode_ms":209.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1174.4,"xrt_wall":9.37,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2413.1,"xrt_compute":14.64,"load_ms":612.2,"mel_ms":62.4,"encode_ms":1674.4,"decode_ms":676.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2439.3,"xrt_wall":14.48,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":732.3,"xrt_compute":15.02,"load_ms":261.1,"mel_ms":22.8,"encode_ms":566.8,"decode_ms":142.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":747.3,"xrt_wall":14.72,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2566.7,"xrt_compute":13.77,"load_ms":973.3,"mel_ms":64.9,"encode_ms":1656.5,"decode_ms":845.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2588.8,"xrt_wall":13.65,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":758.9,"xrt_compute":14.49,"load_ms":423.9,"mel_ms":22.5,"encode_ms":553.8,"decode_ms":182.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":774.7,"xrt_wall":14.2,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/canary-1b.json b/catalog/canary-1b.json new file mode 100644 index 00000000..816aa3dc --- /dev/null +++ b/catalog/canary-1b.json @@ -0,0 +1,72 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "canary-1b", + "family": "canary", + "display_name": "canary-1b", + "params": 1021477920, + "license": { + "spdx": "cc-by-nc-4.0", + "display": "CC-BY-NC-4.0" + }, + "upstream_repo": "nvidia/canary-1b", + "upstream_commit": "1698acf", + "published_repo": "handy-computer/canary-1b-gguf", + "docs_page": "canary.md", + "languages": ["en","de","es","fr"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"targets":["en","de","es","fr"],"pairs":["en>de","de>en","en>es","es>en","en>fr","fr>en"],"verified":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"canary-1b-F32.gguf","size_bytes":4086100544}, + {"quant":"F16","filename":"canary-1b-F16.gguf","size_bytes":2047537728}, + {"quant":"Q8_0","filename":"canary-1b-Q8_0.gguf","size_bytes":1162740288}, + {"quant":"Q6_K","filename":"canary-1b-Q6_K.gguf","size_bytes":934167616}, + {"quant":"Q5_K_M","filename":"canary-1b-Q5_K_M.gguf","size_bytes":837694272}, + {"quant":"Q4_K_M","filename":"canary-1b-Q4_K_M.gguf","size_bytes":729686848} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":6.45,"ci95":[5.83,7.08],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":860,"del":163,"ins":185},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.44,"ci95":[4.0,4.89],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":453,"del":122,"ins":71},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":6.06,"ci95":[5.52,6.66],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":890,"del":104,"ins":420},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":7.44,"ci95":[6.79,8.13],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":924,"del":114,"ins":300},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.55,"ci95":[1.41,1.69],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":619,"del":112,"ins":92},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.55,"ci95":[1.41,1.69],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":619,"del":112,"ins":92},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.55,"ci95":[1.41,1.7],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":617,"del":115,"ins":91},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.57,"ci95":[1.43,1.71],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":623,"del":113,"ins":95},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.57,"ci95":[1.44,1.72],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":627,"del":113,"ins":95},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.55,"ci95":[1.41,1.7],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":627,"del":100,"ins":97},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1731.1,"xrt_compute":20.41,"load_ms":232.4,"mel_ms":2.1,"encode_ms":1043.0,"decode_ms":686.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1737.0,"xrt_wall":20.34,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":430.2,"xrt_compute":25.57,"load_ms":217.8,"mel_ms":0.8,"encode_ms":312.2,"decode_ms":117.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":432.9,"xrt_wall":25.41,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1789.8,"xrt_compute":19.74,"load_ms":305.8,"mel_ms":2.2,"encode_ms":987.8,"decode_ms":799.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1796.6,"xrt_wall":19.67,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":425.7,"xrt_compute":25.84,"load_ms":295.9,"mel_ms":0.7,"encode_ms":290.2,"decode_ms":134.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":428.4,"xrt_wall":25.68,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":930.9,"xrt_compute":37.95,"load_ms":181.9,"mel_ms":2.3,"encode_ms":118.1,"decode_ms":810.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":944.8,"xrt_wall":37.4,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":187.5,"xrt_compute":58.67,"load_ms":184.2,"mel_ms":0.9,"encode_ms":46.7,"decode_ms":140.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":193.2,"xrt_wall":56.94,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1010.8,"xrt_compute":34.96,"load_ms":262.7,"mel_ms":2.3,"encode_ms":115.7,"decode_ms":892.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1025.0,"xrt_wall":34.47,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":207.3,"xrt_compute":53.07,"load_ms":264.9,"mel_ms":0.8,"encode_ms":45.7,"decode_ms":160.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":213.0,"xrt_wall":51.65,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":6256.2,"xrt_compute":5.65,"load_ms":1039.7,"mel_ms":58.2,"encode_ms":2999.2,"decode_ms":3198.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6322.4,"xrt_wall":5.59,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1259.9,"xrt_compute":8.73,"load_ms":736.1,"mel_ms":18.3,"encode_ms":769.9,"decode_ms":471.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1280.5,"xrt_wall":8.59,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":6957.1,"xrt_compute":5.08,"load_ms":1550.0,"mel_ms":57.1,"encode_ms":2950.6,"decode_ms":3949.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7024.8,"xrt_wall":5.03,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1395.3,"xrt_compute":7.88,"load_ms":1039.8,"mel_ms":18.2,"encode_ms":731.5,"decode_ms":645.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1417.7,"xrt_wall":7.76,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3807.2,"xrt_compute":9.28,"load_ms":585.9,"mel_ms":74.9,"encode_ms":1359.1,"decode_ms":2373.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3831.7,"xrt_wall":9.22,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":879.4,"xrt_compute":12.51,"load_ms":250.7,"mel_ms":26.9,"encode_ms":446.2,"decode_ms":406.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":891.2,"xrt_wall":12.34,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4364.3,"xrt_compute":8.1,"load_ms":958.2,"mel_ms":73.3,"encode_ms":1335.9,"decode_ms":2955.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4388.1,"xrt_wall":8.05,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":962.2,"xrt_compute":11.43,"load_ms":408.7,"mel_ms":31.1,"encode_ms":433.6,"decode_ms":497.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":974.3,"xrt_wall":11.29,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/canary-qwen-2.5b.json b/catalog/canary-qwen-2.5b.json new file mode 100644 index 00000000..a7383c76 --- /dev/null +++ b/catalog/canary-qwen-2.5b.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "canary-qwen-2.5b", + "family": "canary_qwen", + "display_name": "canary-qwen-2.5b", + "params": 2533757456, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/canary-qwen-2.5b", + "upstream_commit": "b1469e1", + "published_repo": "handy-computer/canary-qwen-2.5b-gguf", + "docs_page": "canary-qwen-2.5b.md", + "languages": ["en"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"canary-qwen-2.5b-BF16.gguf","size_bytes":5076107136}, + {"quant":"F16","filename":"canary-qwen-2.5b-F16.gguf","size_bytes":5076972928}, + {"quant":"Q8_0","filename":"canary-qwen-2.5b-Q8_0.gguf","size_bytes":2797548928}, + {"quant":"Q6_K","filename":"canary-qwen-2.5b-Q6_K.gguf","size_bytes":2208697728}, + {"quant":"Q5_K_M","filename":"canary-qwen-2.5b-Q5_K_M.gguf","size_bytes":1983729024}, + {"quant":"Q4_K_M","filename":"canary-qwen-2.5b-Q4_K_M.gguf","size_bytes":1737575808} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.58,"ci95":[3.19,4.0],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":321,"del":115,"ins":85},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.63,"ci95":[1.49,1.78],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":633,"del":123,"ins":110},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.63,"ci95":[1.48,1.78],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":631,"del":123,"ins":110},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.63,"ci95":[1.48,1.77],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":631,"del":122,"ins":110},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.63,"ci95":[1.49,1.78],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":630,"del":125,"ins":112},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.63,"ci95":[1.49,1.78],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":625,"del":127,"ins":113},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.63,"ci95":[1.49,1.78],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":643,"del":107,"ins":115},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3813.3,"xrt_compute":9.27,"load_ms":526.7,"mel_ms":2.1,"encode_ms":1403.8,"decode_ms":2407.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3820.4,"xrt_wall":9.25,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1043.1,"xrt_compute":10.55,"load_ms":478.3,"mel_ms":0.8,"encode_ms":404.3,"decode_ms":638.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1046.8,"xrt_wall":10.51,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3831.4,"xrt_compute":9.22,"load_ms":697.5,"mel_ms":2.2,"encode_ms":1336.1,"decode_ms":2493.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3838.1,"xrt_wall":9.21,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1015.4,"xrt_compute":10.83,"load_ms":700.3,"mel_ms":0.8,"encode_ms":377.2,"decode_ms":637.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1019.0,"xrt_wall":10.8,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":830.8,"xrt_compute":42.53,"load_ms":495.7,"mel_ms":2.3,"encode_ms":143.2,"decode_ms":685.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":845.2,"xrt_wall":41.8,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":204.7,"xrt_compute":53.73,"load_ms":494.0,"mel_ms":0.9,"encode_ms":58.6,"decode_ms":145.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":211.8,"xrt_wall":51.92,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":961.4,"xrt_compute":36.75,"load_ms":719.0,"mel_ms":2.2,"encode_ms":140.9,"decode_ms":818.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":976.6,"xrt_wall":36.18,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":228.6,"xrt_compute":48.11,"load_ms":728.9,"mel_ms":0.9,"encode_ms":56.8,"decode_ms":171.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":235.7,"xrt_wall":46.68,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":13865.1,"xrt_compute":2.55,"load_ms":2841.8,"mel_ms":60.9,"encode_ms":3882.7,"decode_ms":9921.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13939.2,"xrt_wall":2.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3460.7,"xrt_compute":3.18,"load_ms":2540.2,"mel_ms":20.0,"encode_ms":1045.7,"decode_ms":2394.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3494.1,"xrt_wall":3.15,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":16243.0,"xrt_compute":2.18,"load_ms":4389.6,"mel_ms":60.3,"encode_ms":3824.6,"decode_ms":12358.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":16316.6,"xrt_wall":2.17,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3871.0,"xrt_compute":2.84,"load_ms":4429.2,"mel_ms":19.2,"encode_ms":1016.5,"decode_ms":2835.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3904.5,"xrt_wall":2.82,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":8712.7,"xrt_compute":4.06,"load_ms":4646.6,"mel_ms":71.5,"encode_ms":1834.9,"decode_ms":6806.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8794.6,"xrt_wall":4.02,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2150.4,"xrt_compute":5.12,"load_ms":4759.8,"mel_ms":33.1,"encode_ms":606.8,"decode_ms":1510.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2184.3,"xrt_wall":5.04,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":10093.7,"xrt_compute":3.5,"load_ms":6344.4,"mel_ms":68.1,"encode_ms":1807.5,"decode_ms":8218.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":10179.3,"xrt_wall":3.47,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2470.9,"xrt_compute":4.45,"load_ms":6743.8,"mel_ms":38.1,"encode_ms":610.2,"decode_ms":1822.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2505.4,"xrt_wall":4.39,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/cohere-transcribe-03-2026.json b/catalog/cohere-transcribe-03-2026.json new file mode 100644 index 00000000..04270d95 --- /dev/null +++ b/catalog/cohere-transcribe-03-2026.json @@ -0,0 +1,82 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "cohere-transcribe-03-2026", + "family": "cohere_asr", + "display_name": "cohere-transcribe-03-2026", + "params": 2049026832, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "CohereLabs/cohere-transcribe-03-2026", + "upstream_commit": "76b8b23", + "published_repo": "handy-computer/cohere-transcribe-03-2026-gguf", + "docs_page": "cohere.md", + "languages": ["en","fr","de","es","it","pt","nl","pl","el","ar","ja","zh","vi","ko"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"cohere-transcribe-03-2026-BF16.gguf","size_bytes":4105263104}, + {"quant":"F16","filename":"cohere-transcribe-03-2026-F16.gguf","size_bytes":4106644992}, + {"quant":"Q8_0","filename":"cohere-transcribe-03-2026-Q8_0.gguf","size_bytes":2410655232}, + {"quant":"Q6_K","filename":"cohere-transcribe-03-2026-Q6_K.gguf","size_bytes":1972524544}, + {"quant":"Q5_K_M","filename":"cohere-transcribe-03-2026-Q5_K_M.gguf","size_bytes":1770270208}, + {"quant":"Q4_K_M","filename":"cohere-transcribe-03-2026-Q4_K_M.gguf","size_bytes":1558162944} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":13.6,"ci95":[12.12,15.16],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":727,"del":305,"ins":88},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":5.06,"ci95":[4.53,5.65],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":571,"del":292,"ins":84},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":8.96,"ci95":[8.23,9.73],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1071,"del":218,"ins":71},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.08,"ci95":[4.56,5.59],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":456,"del":181,"ins":103},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.97,"ci95":[3.56,4.39],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":589,"del":234,"ins":103},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":5.23,"ci95":[4.72,5.77],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":658,"del":175,"ins":108},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":3.24,"ci95":[2.86,3.61],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":417,"del":176,"ins":95},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":5.13,"ci95":[4.48,5.84],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":871,"del":670,"ins":133},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":6.57,"ci95":[5.5,7.65],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":485,"del":591,"ins":85},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":7.16,"ci95":[6.15,8.15],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":350,"del":195,"ins":56},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":6.15,"ci95":[5.51,6.85],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":607,"del":203,"ins":75},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":5.18,"ci95":[4.71,5.66],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":714,"del":239,"ins":159},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":7.39,"ci95":[6.8,7.98],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1324,"del":422,"ins":178},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":11.18,"ci95":[10.19,12.21],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1769,"del":2036,"ins":180},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.27,"ci95":[1.13,1.41],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":510,"del":77,"ins":86},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.25,"ci95":[1.12,1.4],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":506,"del":74,"ins":85},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.24,"ci95":[1.11,1.39],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":501,"del":75,"ins":83},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.26,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.26,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.27,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3535.3,"xrt_compute":9.99,"load_ms":506.4,"mel_ms":2.5,"encode_ms":3342.4,"decode_ms":190.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3542.8,"xrt_wall":9.97,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1004.4,"xrt_compute":10.95,"load_ms":459.3,"mel_ms":0.9,"encode_ms":966.5,"decode_ms":37.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1007.8,"xrt_wall":10.91,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3254.0,"xrt_compute":10.86,"load_ms":686.3,"mel_ms":2.4,"encode_ms":3062.8,"decode_ms":188.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3260.7,"xrt_wall":10.84,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":926.4,"xrt_compute":11.87,"load_ms":613.9,"mel_ms":0.9,"encode_ms":884.3,"decode_ms":41.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":930.1,"xrt_wall":11.83,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":492.4,"xrt_compute":71.76,"load_ms":404.5,"mel_ms":2.2,"encode_ms":304.2,"decode_ms":185.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":507.3,"xrt_wall":69.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":144.1,"xrt_compute":76.32,"load_ms":364.4,"mel_ms":0.8,"encode_ms":105.1,"decode_ms":38.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":151.5,"xrt_wall":72.61,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":470.2,"xrt_compute":75.14,"load_ms":598.1,"mel_ms":2.2,"encode_ms":272.8,"decode_ms":195.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":484.8,"xrt_wall":72.88,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":143.8,"xrt_compute":76.51,"load_ms":525.7,"mel_ms":0.8,"encode_ms":101.5,"decode_ms":41.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":151.3,"xrt_wall":72.7,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":8929.8,"xrt_compute":3.96,"load_ms":2104.3,"mel_ms":57.8,"encode_ms":8104.2,"decode_ms":767.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8979.2,"xrt_wall":3.93,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2465.2,"xrt_compute":4.46,"load_ms":2645.3,"mel_ms":18.7,"encode_ms":2299.8,"decode_ms":146.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2491.1,"xrt_wall":4.42,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":8762.2,"xrt_compute":4.03,"load_ms":2430.5,"mel_ms":58.8,"encode_ms":7737.3,"decode_ms":966.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8811.3,"xrt_wall":4.01,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2398.6,"xrt_compute":4.59,"load_ms":2439.5,"mel_ms":18.6,"encode_ms":2195.6,"decode_ms":184.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2426.2,"xrt_wall":4.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4003.0,"xrt_compute":8.83,"load_ms":541.7,"mel_ms":71.5,"encode_ms":3364.9,"decode_ms":566.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4030.8,"xrt_wall":8.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1409.0,"xrt_compute":7.81,"load_ms":1399.3,"mel_ms":31.6,"encode_ms":1246.9,"decode_ms":130.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1427.2,"xrt_wall":7.71,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4146.4,"xrt_compute":8.52,"load_ms":1087.1,"mel_ms":67.8,"encode_ms":3333.1,"decode_ms":745.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4178.5,"xrt_wall":8.46,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1433.5,"xrt_compute":7.67,"load_ms":1285.2,"mel_ms":34.4,"encode_ms":1244.2,"decode_ms":154.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1452.0,"xrt_wall":7.58,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/cohere-transcribe-arabic-07-2026.json b/catalog/cohere-transcribe-arabic-07-2026.json new file mode 100644 index 00000000..99b08fa1 --- /dev/null +++ b/catalog/cohere-transcribe-arabic-07-2026.json @@ -0,0 +1,76 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "cohere-transcribe-arabic-07-2026", + "family": "cohere_asr", + "display_name": "cohere-transcribe-arabic-07-2026", + "params": 2049026832, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "CohereLabs/cohere-transcribe-arabic-07-2026", + "upstream_commit": "0a8193c", + "published_repo": "handy-computer/cohere-transcribe-arabic-07-2026-gguf", + "docs_page": "cohere.md", + "languages": ["en","ar"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"cohere-transcribe-arabic-07-2026-BF16.gguf","size_bytes":4105263008}, + {"quant":"F16","filename":"cohere-transcribe-arabic-07-2026-F16.gguf","size_bytes":4106644896}, + {"quant":"Q8_0","filename":"cohere-transcribe-arabic-07-2026-Q8_0.gguf","size_bytes":2410655136}, + {"quant":"Q6_K","filename":"cohere-transcribe-arabic-07-2026-Q6_K.gguf","size_bytes":1972524448}, + {"quant":"Q5_K_M","filename":"cohere-transcribe-arabic-07-2026-Q5_K_M.gguf","size_bytes":1770270112}, + {"quant":"Q4_K_M","filename":"cohere-transcribe-arabic-07-2026-Q4_K_M.gguf","size_bytes":1558162848} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"BF16","metric":"wer","err_pct":11.02,"ci95":[9.64,12.53],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":527,"del":286,"ins":95},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"F16","metric":"wer","err_pct":11.0,"ci95":[9.6,12.5],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":525,"del":286,"ins":95},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":11.06,"ci95":[9.62,12.6],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":515,"del":293,"ins":103},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q6_K","metric":"wer","err_pct":11.07,"ci95":[9.63,12.59],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":517,"del":289,"ins":106},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q5_K_M","metric":"wer","err_pct":10.95,"ci95":[9.5,12.49],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":517,"del":298,"ins":87},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q4_K_M","metric":"wer","err_pct":11.18,"ci95":[9.78,12.67],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":526,"del":288,"ins":107},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.88,"ci95":[4.25,5.61],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":396,"del":222,"ins":93},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":4.88,"ci95":[4.25,5.57],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":402,"del":215,"ins":93},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":1.34,"ci95":[1.21,1.48],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":531,"del":98,"ins":80},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"BF16","metric":"wer","err_pct":1.33,"ci95":[1.2,1.47],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":527,"del":99,"ins":81},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"F16","metric":"wer","err_pct":1.33,"ci95":[1.2,1.47],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":527,"del":99,"ins":81},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q6_K","metric":"wer","err_pct":1.34,"ci95":[1.21,1.49],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":531,"del":101,"ins":81},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q5_K_M","metric":"wer","err_pct":1.34,"ci95":[1.2,1.47],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":533,"del":97,"ins":79},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q4_K_M","metric":"wer","err_pct":1.34,"ci95":[1.2,1.48],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":537,"del":94,"ins":82},"empty_hyp":0,"utts_over_50pct":2} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ar", + "metric": "wer", + "batch_size": 8, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3432.7,"xrt_compute":10.29,"load_ms":497.4,"mel_ms":2.4,"encode_ms":3253.4,"decode_ms":177.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3439.9,"xrt_wall":10.27,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1313.8,"xrt_compute":8.37,"load_ms":478.4,"mel_ms":1.0,"encode_ms":1268.7,"decode_ms":44.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1318.1,"xrt_wall":8.35,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3356.4,"xrt_compute":10.53,"load_ms":696.5,"mel_ms":2.5,"encode_ms":3146.5,"decode_ms":207.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3363.8,"xrt_wall":10.5,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":911.6,"xrt_compute":12.07,"load_ms":629.2,"mel_ms":0.8,"encode_ms":872.5,"decode_ms":38.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":915.1,"xrt_wall":12.02,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":459.5,"xrt_compute":76.89,"load_ms":405.7,"mel_ms":2.2,"encode_ms":278.8,"decode_ms":178.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":474.7,"xrt_wall":74.44,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":142.6,"xrt_compute":77.15,"load_ms":364.6,"mel_ms":0.8,"encode_ms":103.9,"decode_ms":37.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":149.9,"xrt_wall":73.38,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":468.8,"xrt_compute":75.36,"load_ms":596.9,"mel_ms":2.2,"encode_ms":271.5,"decode_ms":195.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":483.9,"xrt_wall":73.02,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":141.9,"xrt_compute":77.53,"load_ms":548.9,"mel_ms":0.8,"encode_ms":100.3,"decode_ms":40.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":149.9,"xrt_wall":73.4,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":8942.2,"xrt_compute":3.95,"load_ms":2512.3,"mel_ms":59.5,"encode_ms":8109.3,"decode_ms":773.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8990.3,"xrt_wall":3.93,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2455.3,"xrt_compute":4.48,"load_ms":2597.1,"mel_ms":18.7,"encode_ms":2292.5,"decode_ms":144.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2481.5,"xrt_wall":4.43,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":8754.0,"xrt_compute":4.04,"load_ms":3913.5,"mel_ms":59.6,"encode_ms":7725.2,"decode_ms":969.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8803.8,"xrt_wall":4.01,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2401.2,"xrt_compute":4.58,"load_ms":3556.9,"mel_ms":19.7,"encode_ms":2195.3,"decode_ms":186.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2428.5,"xrt_wall":4.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4002.7,"xrt_compute":8.83,"load_ms":572.7,"mel_ms":65.3,"encode_ms":3355.6,"decode_ms":581.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4029.7,"xrt_wall":8.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1310.3,"xrt_compute":8.4,"load_ms":1431.3,"mel_ms":20.8,"encode_ms":1163.4,"decode_ms":126.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1330.7,"xrt_wall":8.27,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4188.4,"xrt_compute":8.44,"load_ms":1238.1,"mel_ms":65.6,"encode_ms":3383.5,"decode_ms":739.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4220.8,"xrt_wall":8.37,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1346.6,"xrt_compute":8.17,"load_ms":2129.9,"mel_ms":27.9,"encode_ms":1160.9,"decode_ms":157.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1364.7,"xrt_wall":8.06,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/diar_streaming_sortformer_4spk-v2.1.json b/catalog/diar_streaming_sortformer_4spk-v2.1.json new file mode 100644 index 00000000..3e243a66 --- /dev/null +++ b/catalog/diar_streaming_sortformer_4spk-v2.1.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "diar_streaming_sortformer_4spk-v2.1", + "family": "sortformer", + "display_name": "diar_streaming_sortformer_4spk-v2.1", + "params": 117711368, + "license": { + "spdx": "other", + "display": "NVIDIA Open Model License" + }, + "upstream_repo": "nvidia/diar_streaming_sortformer_4spk-v2.1", + "upstream_commit": "fafaab5", + "published_repo": "handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf", + "docs_page": "diar_streaming_sortformer_4spk-v2.1.md", + "languages": ["en"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":true,"max_speakers":4,"verified":false}, + "batching": {"supported":false} + }, + "downloads": [ + {"quant":"F32","filename":"diar_streaming_sortformer_4spk-v2.1-F32.gguf","size_bytes":470910560}, + {"quant":"F16","filename":"diar_streaming_sortformer_4spk-v2.1-F16.gguf","size_bytes":236606560}, + {"quant":"Q8_0","filename":"diar_streaming_sortformer_4spk-v2.1-Q8_0.gguf","size_bytes":139310336} + ], + "accuracy_benchmarks": [ + {"dataset":"ami","split":"ihm-test","language":"en","quant":"F32","metric":"der","err_pct":14.59,"ci95":[null,null],"n_utts":16,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"ami","split":"ihm-test","language":"en","quant":"F16","metric":"der","err_pct":14.23,"ci95":[null,null],"n_utts":16,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"ami","split":"ihm-test","language":"en","quant":"Q8_0","metric":"der","err_pct":14.73,"ci95":[null,null],"n_utts":16,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "ami", + "split": "ihm-test", + "language": "en", + "metric": "der", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4","backend":"cpu","quant":"F16","sample":"jfk","sample_duration_s":11.0,"total_ms":136.3,"xrt_compute":80.68,"load_ms":66.4,"mel_ms":0.0,"encode_ms":136.3,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null}, + {"machine":"m4","backend":"cpu","quant":"F16","sample":"dots","sample_duration_s":35.332,"total_ms":794.2,"xrt_compute":44.49,"load_ms":68.5,"mel_ms":0.0,"encode_ms":794.2,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null}, + {"machine":"m4","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":108.8,"xrt_compute":101.09,"load_ms":49.4,"mel_ms":0.0,"encode_ms":108.8,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null}, + {"machine":"m4","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":684.9,"xrt_compute":51.59,"load_ms":49.9,"mel_ms":0.0,"encode_ms":684.9,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null}, + {"machine":"m4","backend":"metal","quant":"F16","sample":"jfk","sample_duration_s":11.0,"total_ms":68.2,"xrt_compute":161.23,"load_ms":155.2,"mel_ms":0.0,"encode_ms":68.2,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null}, + {"machine":"m4","backend":"metal","quant":"F16","sample":"dots","sample_duration_s":35.332,"total_ms":316.0,"xrt_compute":111.81,"load_ms":68.4,"mel_ms":0.0,"encode_ms":316.0,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null}, + {"machine":"m4","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":63.8,"xrt_compute":172.42,"load_ms":104.6,"mel_ms":0.0,"encode_ms":63.8,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null}, + {"machine":"m4","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":317.9,"xrt_compute":111.16,"load_ms":50.4,"mel_ms":0.0,"encode_ms":317.9,"decode_ms":0.0,"engine_sha":"d42c3bb","measured_on":"2026-07-22","thermal_gated":null} + ] +} diff --git a/catalog/fun-asr-mlt-nano-2512.json b/catalog/fun-asr-mlt-nano-2512.json new file mode 100644 index 00000000..908c4a41 --- /dev/null +++ b/catalog/fun-asr-mlt-nano-2512.json @@ -0,0 +1,105 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "fun-asr-mlt-nano-2512", + "family": "funasr_nano", + "display_name": "fun-asr-mlt-nano-2512", + "params": 829791840, + "license": { + "spdx": "other", + "display": "FunASR Model Open Source License Agreement v1.1", + "name": "funasr-model-license-v1.1", + "link": "https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE" + }, + "upstream_repo": "FunAudioLLM/Fun-ASR-MLT-Nano-2512", + "upstream_commit": "cf67a93", + "published_repo": "handy-computer/Fun-ASR-MLT-Nano-2512-gguf", + "docs_page": "fun-asr-nano.md", + "languages": [ + "zh", "en", "yue", "ja", "ko", "vi", "id", "th", "ms", "tl", "ar", "hi", + "bg", "hr", "cs", "da", "nl", "et", "fi", "el", "hu", "ga", "lv", "lt", + "mt", "pl", "pt", "ro", "sk", "sl", "sv" + ], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Fun-ASR-MLT-Nano-2512-BF16.gguf","size_bytes":1667504192}, + {"quant":"F16","filename":"Fun-ASR-MLT-Nano-2512-F16.gguf","size_bytes":1667504192}, + {"quant":"Q8_0","filename":"Fun-ASR-MLT-Nano-2512-Q8_0.gguf","size_bytes":891271232}, + {"quant":"Q6_K","filename":"Fun-ASR-MLT-Nano-2512-Q6_K.gguf","size_bytes":690744384}, + {"quant":"Q5_K_M","filename":"Fun-ASR-MLT-Nano-2512-Q5_K_M.gguf","size_bytes":631129152}, + {"quant":"Q4_K_M","filename":"Fun-ASR-MLT-Nano-2512-Q4_K_M.gguf","size_bytes":556975168} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":25.79,"ci95":[24.27,27.42],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1640,"del":354,"ins":130},"empty_hyp":0,"utts_over_50pct":34,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":84.98,"ci95":[81.46,89.45],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9508,"del":1740,"ins":794},"empty_hyp":0,"utts_over_50pct":595,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":53.56,"ci95":[51.65,55.84],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5732,"del":818,"ins":784},"empty_hyp":0,"utts_over_50pct":347,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":69.93,"ci95":[68.45,71.45],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10311,"del":3005,"ins":780},"empty_hyp":0,"utts_over_50pct":772,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":103.55,"ci95":[101.64,105.6],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11505,"del":3410,"ins":797},"empty_hyp":0,"utts_over_50pct":642,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.9,"ci95":[4.4,5.48],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":470,"del":124,"ins":120},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":64.22,"ci95":[62.44,66.21],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7260,"del":839,"ins":1457},"empty_hyp":0,"utts_over_50pct":650,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":68.16,"ci95":[66.48,69.98],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7877,"del":845,"ins":1361},"empty_hyp":0,"utts_over_50pct":723,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":15.62,"ci95":[14.78,16.63],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2605,"del":798,"ins":726},"empty_hyp":0,"utts_over_50pct":32,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ga","quant":"Q8_0","metric":"wer","err_pct":100.08,"ci95":[97.48,103.28],"n_utts":842,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":16222,"del":3744,"ins":1403},"empty_hyp":0,"utts_over_50pct":842,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":43.96,"ci95":[39.95,48.08],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3846,"del":3867,"ins":532},"empty_hyp":0,"utts_over_50pct":150,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":61.07,"ci95":[58.92,63.35],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8406,"del":1701,"ins":733},"empty_hyp":0,"utts_over_50pct":556,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":113.21,"ci95":[110.7,116.12],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15267,"del":1093,"ins":2972},"empty_hyp":0,"utts_over_50pct":904,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":7.52,"ci95":[6.85,8.23],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":744,"del":120,"ins":156},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":2.32,"ci95":[2.01,2.65],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":492,"del":146,"ins":118},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":5.2,"ci95":[4.21,6.29],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":348,"del":547,"ins":25},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":78.42,"ci95":[76.74,80.45],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11034,"del":1321,"ins":1081},"empty_hyp":0,"utts_over_50pct":919,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":56.71,"ci95":[55.13,58.29],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7277,"del":685,"ins":780},"empty_hyp":0,"utts_over_50pct":489,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":9.92,"ci95":[9.01,10.83],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1079,"del":231,"ins":154},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":91.71,"ci95":[90.91,92.56],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":16103,"del":3659,"ins":626},"empty_hyp":0,"utts_over_50pct":923,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":42.97,"ci95":[41.38,44.61],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2713,"del":483,"ins":412},"empty_hyp":0,"utts_over_50pct":112,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":59.34,"ci95":[57.63,61.19],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6800,"del":881,"ins":856},"empty_hyp":0,"utts_over_50pct":453,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":28.24,"ci95":[26.85,29.59],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4407,"del":979,"ins":678},"empty_hyp":0,"utts_over_50pct":141,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":74.39,"ci95":[72.46,76.44],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12052,"del":2025,"ins":1396},"empty_hyp":0,"utts_over_50pct":744,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":64.01,"ci95":[62.69,65.28],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8171,"del":904,"ins":683},"empty_hyp":0,"utts_over_50pct":586,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":75.22,"ci95":[73.63,76.87],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9891,"del":1916,"ins":615},"empty_hyp":0,"utts_over_50pct":726,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":75.36,"ci95":[72.22,79.42],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9020,"del":1094,"ins":1482},"empty_hyp":0,"utts_over_50pct":604,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":7.99,"ci95":[7.2,8.97],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3432,"del":1708,"ins":2651},"empty_hyp":0,"utts_over_50pct":19,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":8.32,"ci95":[7.7,8.9],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1585,"del":210,"ins":371},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yue","quant":"Q8_0","metric":"cer","err_pct":12.72,"ci95":[11.82,13.59],"n_utts":819,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3249,"del":224,"ins":216},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":8.64,"ci95":[7.7,9.55],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1274,"del":1484,"ins":322},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.74,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.74,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.74,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.69,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.77,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.89,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1932.9,"xrt_compute":18.28,"load_ms":199.2,"mel_ms":36.8,"encode_ms":691.1,"decode_ms":1205.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1964.5,"xrt_wall":17.99,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":536.6,"xrt_compute":20.5,"load_ms":183.2,"mel_ms":11.5,"encode_ms":193.5,"decode_ms":331.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":546.7,"xrt_wall":20.12,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1917.0,"xrt_compute":18.43,"load_ms":275.4,"mel_ms":36.7,"encode_ms":668.5,"decode_ms":1211.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1947.5,"xrt_wall":18.14,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":532.7,"xrt_compute":20.65,"load_ms":260.3,"mel_ms":11.5,"encode_ms":190.0,"decode_ms":331.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":542.3,"xrt_wall":20.29,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":489.2,"xrt_compute":72.22,"load_ms":206.6,"mel_ms":38.7,"encode_ms":66.5,"decode_ms":384.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":496.5,"xrt_wall":71.17,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":136.4,"xrt_compute":80.64,"load_ms":193.5,"mel_ms":11.6,"encode_ms":28.5,"decode_ms":96.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":141.2,"xrt_wall":77.9,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":546.2,"xrt_compute":64.69,"load_ms":290.9,"mel_ms":46.2,"encode_ms":65.5,"decode_ms":434.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":553.5,"xrt_wall":63.83,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":145.9,"xrt_compute":75.38,"load_ms":272.6,"mel_ms":13.1,"encode_ms":28.0,"decode_ms":104.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":151.0,"xrt_wall":72.86,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":6940.2,"xrt_compute":5.09,"load_ms":816.4,"mel_ms":80.0,"encode_ms":2093.6,"decode_ms":4766.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7040.9,"xrt_wall":5.02,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1773.5,"xrt_compute":6.2,"load_ms":576.9,"mel_ms":23.5,"encode_ms":564.9,"decode_ms":1185.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1808.3,"xrt_wall":6.08,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7395.1,"xrt_compute":4.78,"load_ms":1290.5,"mel_ms":79.1,"encode_ms":1845.5,"decode_ms":5470.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7491.3,"xrt_wall":4.72,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1786.4,"xrt_compute":6.16,"load_ms":862.2,"mel_ms":23.4,"encode_ms":478.0,"decode_ms":1285.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1818.4,"xrt_wall":6.05,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4026.4,"xrt_compute":8.78,"load_ms":1195.8,"mel_ms":87.5,"encode_ms":1261.5,"decode_ms":2677.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4102.2,"xrt_wall":8.61,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1036.8,"xrt_compute":10.61,"load_ms":757.6,"mel_ms":24.5,"encode_ms":354.4,"decode_ms":657.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1069.2,"xrt_wall":10.29,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4454.3,"xrt_compute":7.93,"load_ms":2039.4,"mel_ms":89.8,"encode_ms":1254.1,"decode_ms":3110.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4531.5,"xrt_wall":7.8,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1129.0,"xrt_compute":9.74,"load_ms":1279.3,"mel_ms":25.7,"encode_ms":350.6,"decode_ms":752.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1162.0,"xrt_wall":9.47,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/fun-asr-nano-2512.json b/catalog/fun-asr-nano-2512.json new file mode 100644 index 00000000..a2c81d68 --- /dev/null +++ b/catalog/fun-asr-nano-2512.json @@ -0,0 +1,73 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "fun-asr-nano-2512", + "family": "funasr_nano", + "display_name": "fun-asr-nano-2512", + "params": 829791840, + "license": { + "spdx": "other", + "display": "FunASR Model Open Source License Agreement v1.1", + "name": "funasr-model-license-v1.1", + "link": "https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE" + }, + "upstream_repo": "FunAudioLLM/Fun-ASR-Nano-2512", + "upstream_commit": "a7088d6", + "published_repo": "handy-computer/Fun-ASR-Nano-2512-gguf", + "docs_page": "fun-asr-nano.md", + "languages": ["zh","en","ja"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Fun-ASR-Nano-2512-BF16.gguf","size_bytes":1667503872}, + {"quant":"F16","filename":"Fun-ASR-Nano-2512-F16.gguf","size_bytes":1667503872}, + {"quant":"Q8_0","filename":"Fun-ASR-Nano-2512-Q8_0.gguf","size_bytes":891270912}, + {"quant":"Q6_K","filename":"Fun-ASR-Nano-2512-Q6_K.gguf","size_bytes":690744064}, + {"quant":"Q5_K_M","filename":"Fun-ASR-Nano-2512-Q5_K_M.gguf","size_bytes":631128832}, + {"quant":"Q4_K_M","filename":"Fun-ASR-Nano-2512-Q4_K_M.gguf","size_bytes":556974848} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.49,"ci95":[4.91,6.04],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":522,"del":126,"ins":151},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":8.5,"ci95":[7.86,9.22],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1976,"del":401,"ins":398},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":8.59,"ci95":[7.59,9.53],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1273,"del":1476,"ins":312},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.78,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.79,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.79,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.78,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.82,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.92,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1315.4,"xrt_compute":26.86,"load_ms":210.0,"mel_ms":37.2,"encode_ms":689.2,"decode_ms":589.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1347.0,"xrt_wall":26.23,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":361.8,"xrt_compute":30.4,"load_ms":192.7,"mel_ms":11.6,"encode_ms":196.6,"decode_ms":153.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":371.8,"xrt_wall":29.58,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1358.9,"xrt_compute":26.0,"load_ms":291.4,"mel_ms":37.0,"encode_ms":661.9,"decode_ms":660.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1388.9,"xrt_wall":25.44,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":365.2,"xrt_compute":30.12,"load_ms":258.3,"mel_ms":11.6,"encode_ms":187.3,"decode_ms":166.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":374.7,"xrt_wall":29.35,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":449.2,"xrt_compute":78.65,"load_ms":214.2,"mel_ms":47.2,"encode_ms":67.9,"decode_ms":334.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":456.8,"xrt_wall":77.35,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":123.8,"xrt_compute":88.83,"load_ms":192.7,"mel_ms":13.7,"encode_ms":28.1,"decode_ms":82.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":129.1,"xrt_wall":85.19,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":483.2,"xrt_compute":73.13,"load_ms":293.8,"mel_ms":43.9,"encode_ms":64.4,"decode_ms":374.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":490.4,"xrt_wall":72.05,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":132.3,"xrt_compute":83.13,"load_ms":275.6,"mel_ms":13.7,"encode_ms":27.1,"decode_ms":91.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":137.5,"xrt_wall":79.98,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4657.8,"xrt_compute":7.59,"load_ms":574.8,"mel_ms":78.6,"encode_ms":2047.4,"decode_ms":2531.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4753.1,"xrt_wall":7.43,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1145.3,"xrt_compute":9.6,"load_ms":832.1,"mel_ms":23.4,"encode_ms":557.9,"decode_ms":564.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1180.3,"xrt_wall":9.32,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":5082.5,"xrt_compute":6.95,"load_ms":945.5,"mel_ms":78.7,"encode_ms":1808.2,"decode_ms":3195.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5169.3,"xrt_wall":6.83,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1230.0,"xrt_compute":8.94,"load_ms":1176.6,"mel_ms":23.2,"encode_ms":481.5,"decode_ms":725.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1260.9,"xrt_wall":8.72,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3061.2,"xrt_compute":11.54,"load_ms":755.5,"mel_ms":85.5,"encode_ms":1247.5,"decode_ms":1728.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3137.2,"xrt_wall":11.26,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":838.3,"xrt_compute":13.12,"load_ms":1197.4,"mel_ms":26.1,"encode_ms":354.4,"decode_ms":457.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":869.9,"xrt_wall":12.64,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3800.8,"xrt_compute":9.3,"load_ms":1280.7,"mel_ms":89.4,"encode_ms":1249.0,"decode_ms":2462.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3875.4,"xrt_wall":9.12,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":904.8,"xrt_compute":12.16,"load_ms":1917.3,"mel_ms":30.4,"encode_ms":353.5,"decode_ms":520.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":937.3,"xrt_wall":11.74,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/gigaam-v3-ctc.json b/catalog/gigaam-v3-ctc.json new file mode 100644 index 00000000..1213b648 --- /dev/null +++ b/catalog/gigaam-v3-ctc.json @@ -0,0 +1,68 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "gigaam-v3-ctc", + "family": "gigaam", + "display_name": "gigaam-v3-ctc", + "params": 220718498, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "ai-sage/GigaAM-v3", + "upstream_commit": "15ef3b5", + "published_repo": "handy-computer/gigaam-v3-ctc-gguf", + "docs_page": "gigaam.md", + "languages": ["ru"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"gigaam-v3-ctc-F32.gguf","size_bytes":882913216}, + {"quant":"F16","filename":"gigaam-v3-ctc-F16.gguf","size_bytes":448750528}, + {"quant":"Q8_0","filename":"gigaam-v3-ctc-Q8_0.gguf","size_bytes":271803328}, + {"quant":"Q6_K","filename":"gigaam-v3-ctc-Q6_K.gguf","size_bytes":226091968}, + {"quant":"Q5_K_M","filename":"gigaam-v3-ctc-Q5_K_M.gguf","size_bytes":204563392}, + {"quant":"Q4_K_M","filename":"gigaam-v3-ctc-Q4_K_M.gguf","size_bytes":182150080} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":8.42,"ci95":[7.64,9.15],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":924,"del":104,"ins":243},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F32","metric":"wer","err_pct":8.42,"ci95":[7.66,9.16],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":925,"del":104,"ins":242},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F16","metric":"wer","err_pct":8.42,"ci95":[7.66,9.16],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":925,"del":104,"ins":242},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q6_K","metric":"wer","err_pct":8.38,"ci95":[7.64,9.14],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":921,"del":105,"ins":240},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q5_K_M","metric":"wer","err_pct":8.29,"ci95":[7.55,9.02],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":908,"del":103,"ins":241},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q4_K_M","metric":"wer","err_pct":8.42,"ci95":[7.67,9.15],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":926,"del":105,"ins":240},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ru", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":1258.2,"xrt_compute":26.9,"wall_ms":1259.0,"xrt_wall":26.88,"load_ms":71.1,"mel_ms":4.7,"encode_ms":1253.1,"decode_ms":0.4,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":363.5,"xrt_compute":30.21,"wall_ms":364.0,"xrt_wall":30.17,"load_ms":64.6,"mel_ms":1.6,"encode_ms":361.7,"decode_ms":0.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":1161.1,"xrt_compute":29.15,"wall_ms":1161.9,"xrt_wall":29.13,"load_ms":91.0,"mel_ms":4.6,"encode_ms":1156.1,"decode_ms":0.4,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":367.0,"xrt_compute":29.92,"wall_ms":367.5,"xrt_wall":29.88,"load_ms":83.2,"mel_ms":1.6,"encode_ms":365.2,"decode_ms":0.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":64.4,"xrt_compute":525.83,"wall_ms":66.2,"xrt_wall":510.97,"load_ms":62.0,"mel_ms":5.6,"encode_ms":58.4,"decode_ms":0.4,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":27.1,"xrt_compute":405.11,"wall_ms":28.4,"xrt_wall":386.21,"load_ms":62.0,"mel_ms":2.4,"encode_ms":24.6,"decode_ms":0.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":62.5,"xrt_compute":541.17,"wall_ms":64.3,"xrt_wall":526.31,"load_ms":79.2,"mel_ms":5.0,"encode_ms":57.2,"decode_ms":0.4,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":26.4,"xrt_compute":415.59,"wall_ms":27.7,"xrt_wall":396.53,"load_ms":79.8,"mel_ms":2.4,"encode_ms":23.9,"decode_ms":0.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":3704.1,"xrt_compute":9.14,"wall_ms":3714.9,"xrt_wall":9.11,"load_ms":243.9,"mel_ms":57.7,"encode_ms":3626.5,"decode_ms":20.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":901.5,"xrt_compute":12.18,"wall_ms":904.3,"xrt_wall":12.14,"load_ms":207.9,"mel_ms":25.9,"encode_ms":870.4,"decode_ms":5.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":3450.6,"xrt_compute":9.81,"wall_ms":3461.6,"xrt_wall":9.78,"load_ms":382.8,"mel_ms":58.1,"encode_ms":3372.5,"decode_ms":20.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":793.3,"xrt_compute":13.84,"wall_ms":796.0,"xrt_wall":13.79,"load_ms":213.2,"mel_ms":23.6,"encode_ms":764.4,"decode_ms":5.3,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":846.0,"xrt_compute":40.0,"wall_ms":866.2,"xrt_wall":39.07,"load_ms":213.0,"mel_ms":54.3,"encode_ms":776.2,"decode_ms":15.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":291.1,"xrt_compute":37.72,"wall_ms":301.6,"xrt_wall":36.41,"load_ms":101.0,"mel_ms":24.5,"encode_ms":261.4,"decode_ms":5.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":827.6,"xrt_compute":40.89,"wall_ms":850.4,"xrt_wall":39.79,"load_ms":205.9,"mel_ms":54.6,"encode_ms":757.5,"decode_ms":15.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":284.1,"xrt_compute":38.65,"wall_ms":294.8,"xrt_wall":37.25,"load_ms":127.2,"mel_ms":24.8,"encode_ms":254.3,"decode_ms":5.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/gigaam-v3-e2e-ctc.json b/catalog/gigaam-v3-e2e-ctc.json new file mode 100644 index 00000000..d5b6de6c --- /dev/null +++ b/catalog/gigaam-v3-e2e-ctc.json @@ -0,0 +1,68 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "gigaam-v3-e2e-ctc", + "family": "gigaam", + "display_name": "gigaam-v3-e2e-ctc", + "params": 220889985, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "ai-sage/GigaAM-v3", + "upstream_commit": "cec030b", + "published_repo": "handy-computer/gigaam-v3-e2e-ctc-gguf", + "docs_page": "gigaam.md", + "languages": ["ru"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"gigaam-v3-e2e-ctc-F32.gguf","size_bytes":883603552}, + {"quant":"F16","filename":"gigaam-v3-e2e-ctc-F16.gguf","size_bytes":449098336}, + {"quant":"Q8_0","filename":"gigaam-v3-e2e-ctc-Q8_0.gguf","size_bytes":272151136}, + {"quant":"Q6_K","filename":"gigaam-v3-e2e-ctc-Q6_K.gguf","size_bytes":226439776}, + {"quant":"Q5_K_M","filename":"gigaam-v3-e2e-ctc-Q5_K_M.gguf","size_bytes":204911200}, + {"quant":"Q4_K_M","filename":"gigaam-v3-e2e-ctc-Q4_K_M.gguf","size_bytes":182497888} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":5.53,"ci95":[5.01,6.07],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":702,"del":84,"ins":49},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F32","metric":"wer","err_pct":5.5,"ci95":[4.99,6.08],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":699,"del":84,"ins":48},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F16","metric":"wer","err_pct":5.5,"ci95":[4.99,6.08],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":699,"del":84,"ins":48},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q6_K","metric":"wer","err_pct":5.56,"ci95":[5.03,6.14],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":704,"del":84,"ins":52},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q5_K_M","metric":"wer","err_pct":5.58,"ci95":[5.04,6.15],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":705,"del":86,"ins":51},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q4_K_M","metric":"wer","err_pct":5.57,"ci95":[null,null],"n_utts":775,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ru", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":1243.5,"xrt_compute":27.21,"wall_ms":1244.4,"xrt_wall":27.19,"load_ms":70.2,"mel_ms":4.7,"encode_ms":1236.8,"decode_ms":2.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":361.3,"xrt_compute":30.39,"wall_ms":361.8,"xrt_wall":30.35,"load_ms":64.7,"mel_ms":1.9,"encode_ms":358.7,"decode_ms":0.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":1160.1,"xrt_compute":29.17,"wall_ms":1160.9,"xrt_wall":29.15,"load_ms":87.6,"mel_ms":4.6,"encode_ms":1153.4,"decode_ms":2.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":367.7,"xrt_compute":29.86,"wall_ms":368.2,"xrt_wall":29.82,"load_ms":80.0,"mel_ms":1.6,"encode_ms":365.5,"decode_ms":0.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":66.0,"xrt_compute":512.9,"wall_ms":67.8,"xrt_wall":498.99,"load_ms":62.4,"mel_ms":4.9,"encode_ms":59.1,"decode_ms":1.9,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":26.7,"xrt_compute":411.99,"wall_ms":28.0,"xrt_wall":392.25,"load_ms":61.2,"mel_ms":2.1,"encode_ms":23.8,"decode_ms":0.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":64.4,"xrt_compute":525.57,"wall_ms":66.2,"xrt_wall":511.44,"load_ms":80.5,"mel_ms":5.3,"encode_ms":57.1,"decode_ms":2.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":27.1,"xrt_compute":405.69,"wall_ms":28.4,"xrt_wall":386.93,"load_ms":79.7,"mel_ms":2.5,"encode_ms":23.9,"decode_ms":0.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":3735.1,"xrt_compute":9.06,"wall_ms":3745.0,"xrt_wall":9.04,"load_ms":267.5,"mel_ms":55.4,"encode_ms":3556.9,"decode_ms":122.9,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":928.3,"xrt_compute":11.83,"wall_ms":931.2,"xrt_wall":11.79,"load_ms":156.0,"mel_ms":24.8,"encode_ms":864.9,"decode_ms":38.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":3530.6,"xrt_compute":9.58,"wall_ms":3540.3,"xrt_wall":9.56,"load_ms":350.1,"mel_ms":55.4,"encode_ms":3351.3,"decode_ms":124.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":823.8,"xrt_compute":13.33,"wall_ms":826.7,"xrt_wall":13.28,"load_ms":211.9,"mel_ms":23.3,"encode_ms":761.9,"decode_ms":38.6,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":953.7,"xrt_compute":35.48,"wall_ms":969.7,"xrt_wall":34.9,"load_ms":108.8,"mel_ms":53.4,"encode_ms":782.8,"decode_ms":117.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":325.9,"xrt_compute":33.69,"wall_ms":336.3,"xrt_wall":32.65,"load_ms":105.5,"mel_ms":25.7,"encode_ms":262.1,"decode_ms":38.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":934.3,"xrt_compute":36.22,"wall_ms":950.0,"xrt_wall":35.62,"load_ms":173.1,"mel_ms":53.8,"encode_ms":763.2,"decode_ms":117.3,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":318.0,"xrt_compute":34.53,"wall_ms":328.7,"xrt_wall":33.4,"load_ms":127.0,"mel_ms":22.8,"encode_ms":257.0,"decode_ms":38.3,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/gigaam-v3-e2e-rnnt.json b/catalog/gigaam-v3-e2e-rnnt.json new file mode 100644 index 00000000..f6bb7dc2 --- /dev/null +++ b/catalog/gigaam-v3-e2e-rnnt.json @@ -0,0 +1,68 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "gigaam-v3-e2e-rnnt", + "family": "gigaam", + "display_name": "gigaam-v3-e2e-rnnt", + "params": 222518657, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "ai-sage/GigaAM-v3", + "upstream_commit": "ec1dc1f", + "published_repo": "handy-computer/gigaam-v3-e2e-rnnt-gguf", + "docs_page": "gigaam.md", + "languages": ["ru"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"gigaam-v3-e2e-rnnt-F32.gguf","size_bytes":890138592}, + {"quant":"F16","filename":"gigaam-v3-e2e-rnnt-F16.gguf","size_bytes":452381408}, + {"quant":"Q8_0","filename":"gigaam-v3-e2e-rnnt-Q8_0.gguf","size_bytes":273724832}, + {"quant":"Q6_K","filename":"gigaam-v3-e2e-rnnt-Q6_K.gguf","size_bytes":227953952}, + {"quant":"Q5_K_M","filename":"gigaam-v3-e2e-rnnt-Q5_K_M.gguf","size_bytes":206392736}, + {"quant":"Q4_K_M","filename":"gigaam-v3-e2e-rnnt-Q4_K_M.gguf","size_bytes":183948704} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":5.35,"ci95":[4.85,5.9],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":661,"del":98,"ins":49},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F32","metric":"wer","err_pct":5.35,"ci95":[4.85,5.9],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":661,"del":98,"ins":49},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F16","metric":"wer","err_pct":5.35,"ci95":[4.85,5.9],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":661,"del":98,"ins":49},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q6_K","metric":"wer","err_pct":5.37,"ci95":[4.89,5.9],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":665,"del":98,"ins":48},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q5_K_M","metric":"wer","err_pct":5.42,"ci95":[4.94,5.97],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":672,"del":95,"ins":52},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q4_K_M","metric":"wer","err_pct":5.36,"ci95":[4.84,5.94],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":671,"del":94,"ins":44},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ru", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":1257.8,"xrt_compute":26.9,"wall_ms":1258.7,"xrt_wall":26.89,"load_ms":73.5,"mel_ms":4.7,"encode_ms":1241.2,"decode_ms":11.9,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":367.5,"xrt_compute":29.88,"wall_ms":367.9,"xrt_wall":29.84,"load_ms":63.5,"mel_ms":1.6,"encode_ms":360.9,"decode_ms":5.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":1178.5,"xrt_compute":28.71,"wall_ms":1179.3,"xrt_wall":28.69,"load_ms":101.8,"mel_ms":4.7,"encode_ms":1162.0,"decode_ms":11.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":378.7,"xrt_compute":28.99,"wall_ms":379.2,"xrt_wall":28.96,"load_ms":81.7,"mel_ms":1.6,"encode_ms":371.9,"decode_ms":5.2,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":75.2,"xrt_compute":449.86,"wall_ms":77.0,"xrt_wall":439.47,"load_ms":62.9,"mel_ms":4.9,"encode_ms":58.7,"decode_ms":11.6,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":30.4,"xrt_compute":360.92,"wall_ms":31.7,"xrt_wall":345.84,"load_ms":65.0,"mel_ms":1.9,"encode_ms":23.8,"decode_ms":4.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":73.8,"xrt_compute":458.32,"wall_ms":75.7,"xrt_wall":447.18,"load_ms":83.4,"mel_ms":5.3,"encode_ms":56.7,"decode_ms":11.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":30.5,"xrt_compute":360.53,"wall_ms":31.9,"xrt_wall":344.48,"load_ms":81.7,"mel_ms":2.2,"encode_ms":23.3,"decode_ms":5.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":4848.2,"xrt_compute":6.98,"wall_ms":4858.1,"xrt_wall":6.97,"load_ms":269.2,"mel_ms":54.1,"encode_ms":3541.0,"decode_ms":1253.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":1395.4,"xrt_compute":7.87,"wall_ms":1398.4,"xrt_wall":7.85,"load_ms":161.1,"mel_ms":23.1,"encode_ms":865.9,"decode_ms":506.4,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":4647.6,"xrt_compute":7.28,"wall_ms":4658.1,"xrt_wall":7.26,"load_ms":337.5,"mel_ms":53.4,"encode_ms":3326.0,"decode_ms":1268.3,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":1296.1,"xrt_compute":8.47,"wall_ms":1299.2,"xrt_wall":8.45,"load_ms":222.8,"mel_ms":24.0,"encode_ms":761.2,"decode_ms":510.9,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":2084.2,"xrt_compute":16.24,"wall_ms":2100.2,"xrt_wall":16.11,"load_ms":141.1,"mel_ms":55.2,"encode_ms":784.4,"decode_ms":1244.6,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":799.7,"xrt_compute":13.73,"wall_ms":810.9,"xrt_wall":13.54,"load_ms":117.4,"mel_ms":25.0,"encode_ms":272.5,"decode_ms":502.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":2081.4,"xrt_compute":16.26,"wall_ms":2096.7,"xrt_wall":16.14,"load_ms":193.4,"mel_ms":54.0,"encode_ms":774.4,"decode_ms":1253.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":796.6,"xrt_compute":13.78,"wall_ms":805.5,"xrt_wall":13.63,"load_ms":144.9,"mel_ms":26.3,"encode_ms":267.5,"decode_ms":502.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/gigaam-v3-rnnt.json b/catalog/gigaam-v3-rnnt.json new file mode 100644 index 00000000..8db8af45 --- /dev/null +++ b/catalog/gigaam-v3-rnnt.json @@ -0,0 +1,68 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "gigaam-v3-rnnt", + "family": "gigaam", + "display_name": "gigaam-v3-rnnt", + "params": 221883426, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "ai-sage/GigaAM-v3", + "upstream_commit": "c7f128b", + "published_repo": "handy-computer/gigaam-v3-rnnt-gguf", + "docs_page": "gigaam.md", + "languages": ["ru"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"gigaam-v3-rnnt-F32.gguf","size_bytes":887573536}, + {"quant":"F16","filename":"gigaam-v3-rnnt-F16.gguf","size_bytes":451084832}, + {"quant":"Q8_0","filename":"gigaam-v3-rnnt-Q8_0.gguf","size_bytes":273022880}, + {"quant":"Q6_K","filename":"gigaam-v3-rnnt-Q6_K.gguf","size_bytes":227252000}, + {"quant":"Q5_K_M","filename":"gigaam-v3-rnnt-Q5_K_M.gguf","size_bytes":205690784}, + {"quant":"Q4_K_M","filename":"gigaam-v3-rnnt-Q4_K_M.gguf","size_bytes":183246752} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":8.07,"ci95":[7.34,8.79],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":897,"del":100,"ins":222},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F32","metric":"wer","err_pct":8.08,"ci95":[7.36,8.78],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":900,"del":99,"ins":221},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"F16","metric":"wer","err_pct":8.08,"ci95":[7.36,8.78],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":900,"del":99,"ins":221},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q6_K","metric":"wer","err_pct":8.07,"ci95":[7.35,8.78],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":900,"del":97,"ins":222},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q5_K_M","metric":"wer","err_pct":8.12,"ci95":[7.38,8.84],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":900,"del":98,"ins":228},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q4_K_M","metric":"wer","err_pct":8.12,"ci95":[7.4,8.85],"n_utts":775,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":901,"del":104,"ins":222},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ru", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":1278.9,"xrt_compute":26.46,"wall_ms":1279.8,"xrt_wall":26.44,"load_ms":64.9,"mel_ms":4.7,"encode_ms":1266.5,"decode_ms":7.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":371.3,"xrt_compute":29.57,"wall_ms":371.8,"xrt_wall":29.53,"load_ms":68.3,"mel_ms":1.6,"encode_ms":366.6,"decode_ms":3.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":1183.9,"xrt_compute":28.58,"wall_ms":1184.8,"xrt_wall":28.56,"load_ms":98.0,"mel_ms":4.8,"encode_ms":1171.3,"decode_ms":7.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":375.8,"xrt_compute":29.22,"wall_ms":376.3,"xrt_wall":29.18,"load_ms":89.2,"mel_ms":1.6,"encode_ms":371.0,"decode_ms":3.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":69.9,"xrt_compute":484.17,"wall_ms":71.7,"xrt_wall":471.74,"load_ms":63.0,"mel_ms":4.7,"encode_ms":57.7,"decode_ms":7.5,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":29.9,"xrt_compute":367.06,"wall_ms":31.3,"xrt_wall":350.75,"load_ms":66.3,"mel_ms":2.5,"encode_ms":24.4,"decode_ms":3.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":70.0,"xrt_compute":483.39,"wall_ms":71.9,"xrt_wall":470.91,"load_ms":81.8,"mel_ms":4.9,"encode_ms":57.6,"decode_ms":7.5,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":28.5,"xrt_compute":385.49,"wall_ms":29.9,"xrt_wall":367.62,"load_ms":83.3,"mel_ms":2.2,"encode_ms":23.3,"decode_ms":3.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":4065.1,"xrt_compute":8.32,"wall_ms":4074.6,"xrt_wall":8.31,"load_ms":227.8,"mel_ms":53.9,"encode_ms":3602.3,"decode_ms":408.9,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":1040.4,"xrt_compute":10.55,"wall_ms":1043.3,"xrt_wall":10.52,"load_ms":160.4,"mel_ms":23.8,"encode_ms":870.4,"decode_ms":146.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":3813.4,"xrt_compute":8.87,"wall_ms":3823.7,"xrt_wall":8.85,"load_ms":374.1,"mel_ms":53.3,"encode_ms":3354.3,"decode_ms":405.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":929.7,"xrt_compute":11.81,"wall_ms":932.7,"xrt_wall":11.77,"load_ms":229.2,"mel_ms":24.8,"encode_ms":760.1,"decode_ms":144.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-long","sample_duration_s":33.84,"total_ms":1237.7,"xrt_compute":27.34,"wall_ms":1257.3,"xrt_wall":26.91,"load_ms":161.8,"mel_ms":55.6,"encode_ms":786.2,"decode_ms":395.9,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"ru-short","sample_duration_s":10.98,"total_ms":437.2,"xrt_compute":25.11,"wall_ms":448.1,"xrt_wall":24.5,"load_ms":111.4,"mel_ms":24.8,"encode_ms":266.9,"decode_ms":145.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-long","sample_duration_s":33.84,"total_ms":1217.0,"xrt_compute":27.81,"wall_ms":1236.7,"xrt_wall":27.36,"load_ms":157.5,"mel_ms":54.3,"encode_ms":766.5,"decode_ms":396.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ru-short","sample_duration_s":10.98,"total_ms":427.8,"xrt_compute":25.67,"wall_ms":438.0,"xrt_wall":25.07,"load_ms":141.9,"mel_ms":23.2,"encode_ms":259.0,"decode_ms":145.6,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/granite-4.0-1b-speech.json b/catalog/granite-4.0-1b-speech.json new file mode 100644 index 00000000..32dae58b --- /dev/null +++ b/catalog/granite-4.0-1b-speech.json @@ -0,0 +1,74 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "granite-4.0-1b-speech", + "family": "granite_speech", + "display_name": "granite-4.0-1b-speech", + "params": 2313228092, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "ibm-granite/granite-4.0-1b-speech", + "upstream_commit": "bd87ab8", + "published_repo": "handy-computer/granite-4.0-1b-speech-gguf", + "docs_page": "granite-speech.md", + "languages": ["en","fr","de","es","pt","ja"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en","fr","de","es","pt","ja","it","zh"],"pairs":["en>fr","fr>en","en>de","de>en","en>es","es>en","en>pt","pt>en","en>ja","ja>en","en>it","en>zh"],"note":"translation is reached via a separate chat-template prompt; scripts/convert-granite.py is authoritative for the target set"}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"granite-4.0-1b-speech-BF16.gguf","size_bytes":4631640064}, + {"quant":"F16","filename":"granite-4.0-1b-speech-F16.gguf","size_bytes":4632623104}, + {"quant":"Q8_0","filename":"granite-4.0-1b-speech-Q8_0.gguf","size_bytes":2559878848}, + {"quant":"Q6_K","filename":"granite-4.0-1b-speech-Q6_K.gguf","size_bytes":2024967936}, + {"quant":"Q5_K_M","filename":"granite-4.0-1b-speech-Q5_K_M.gguf","size_bytes":1829704544}, + {"quant":"Q4_K_M","filename":"granite-4.0-1b-speech-Q4_K_M.gguf","size_bytes":1602904800} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":7.29,"ci95":[6.71,7.93],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1022,"del":192,"ins":151},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.66,"ci95":[4.2,5.15],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":448,"del":115,"ins":116},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":5.91,"ci95":[5.41,6.45],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":925,"del":189,"ins":264},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":8.44,"ci95":[7.72,9.24],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1030,"del":201,"ins":288},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":6.44,"ci95":[5.86,6.98],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1433,"del":302,"ins":369},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":9.6,"ci95":[8.89,10.34],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1362,"del":217,"ins":482},"empty_hyp":0,"utts_over_50pct":19,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.42,"ci95":[1.28,1.57],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":584,"del":76,"ins":93},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.42,"ci95":[1.29,1.57],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":586,"del":75,"ins":94},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.44,"ci95":[1.3,1.58],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":591,"del":76,"ins":95},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.41,"ci95":[1.28,1.55],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":582,"del":73,"ins":92},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.42,"ci95":[1.28,1.56],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":581,"del":77,"ins":93},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.48,"ci95":[1.34,1.63],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":602,"del":76,"ins":108},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4227.2,"xrt_compute":8.36,"load_ms":378.1,"mel_ms":2.0,"encode_ms":2822.6,"decode_ms":1402.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5133.6,"xrt_wall":6.88,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1386.1,"xrt_compute":7.94,"load_ms":386.2,"mel_ms":0.8,"encode_ms":889.3,"decode_ms":495.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1576.1,"xrt_wall":6.98,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4200.5,"xrt_compute":8.41,"load_ms":580.6,"mel_ms":2.1,"encode_ms":2732.6,"decode_ms":1465.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5350.0,"xrt_wall":6.6,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1323.7,"xrt_compute":8.31,"load_ms":577.9,"mel_ms":0.8,"encode_ms":823.8,"decode_ms":499.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1567.3,"xrt_wall":7.02,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":472.3,"xrt_compute":74.81,"load_ms":473.5,"mel_ms":2.1,"encode_ms":302.9,"decode_ms":167.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1186.6,"xrt_wall":29.78,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":168.9,"xrt_compute":65.11,"load_ms":416.7,"mel_ms":0.8,"encode_ms":104.0,"decode_ms":64.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":311.0,"xrt_wall":35.37,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":502.5,"xrt_compute":70.32,"load_ms":705.5,"mel_ms":2.3,"encode_ms":335.1,"decode_ms":165.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1314.1,"xrt_wall":26.89,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":136.4,"xrt_compute":80.65,"load_ms":639.5,"mel_ms":0.8,"encode_ms":83.9,"decode_ms":51.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":293.9,"xrt_wall":37.42,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":12720.7,"xrt_compute":2.78,"load_ms":2198.2,"mel_ms":42.8,"encode_ms":8738.1,"decode_ms":3939.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":17191.9,"xrt_wall":2.06,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3903.2,"xrt_compute":2.82,"load_ms":1446.8,"mel_ms":15.8,"encode_ms":2509.3,"decode_ms":1378.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4842.4,"xrt_wall":2.27,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":12928.2,"xrt_compute":2.73,"load_ms":3728.2,"mel_ms":44.1,"encode_ms":8907.2,"decode_ms":3976.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":19603.2,"xrt_wall":1.8,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3958.6,"xrt_compute":2.78,"load_ms":3735.2,"mel_ms":15.7,"encode_ms":2518.4,"decode_ms":1424.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5379.1,"xrt_wall":2.04,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":6844.4,"xrt_compute":5.16,"load_ms":3084.1,"mel_ms":47.7,"encode_ms":5082.4,"decode_ms":1714.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":10961.6,"xrt_wall":3.22,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2543.9,"xrt_compute":4.32,"load_ms":3020.8,"mel_ms":26.7,"encode_ms":1779.3,"decode_ms":737.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3422.8,"xrt_wall":3.21,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7148.2,"xrt_compute":4.94,"load_ms":3988.1,"mel_ms":57.3,"encode_ms":5409.0,"decode_ms":1681.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":12590.9,"xrt_wall":2.81,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2539.8,"xrt_compute":4.33,"load_ms":4431.0,"mel_ms":27.9,"encode_ms":1765.4,"decode_ms":746.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3759.7,"xrt_wall":2.93,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/granite-speech-4.1-2b-nar.json b/catalog/granite-speech-4.1-2b-nar.json new file mode 100644 index 00000000..15aff752 --- /dev/null +++ b/catalog/granite-speech-4.1-2b-nar.json @@ -0,0 +1,73 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "granite-speech-4.1-2b-nar", + "family": "granite_speech_nar", + "display_name": "granite-speech-4.1-2b-nar", + "params": 2254656316, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "ibm-granite/granite-speech-4.1-2b-nar", + "upstream_commit": "99a4df9", + "published_repo": "handy-computer/granite-speech-4.1-2b-nar-gguf", + "docs_page": "granite-speech.md", + "languages": ["en","fr","de","es","pt"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"granite-speech-4.1-2b-nar-BF16.gguf","size_bytes":4514736032}, + {"quant":"F16","filename":"granite-speech-4.1-2b-nar-F16.gguf","size_bytes":4515792768}, + {"quant":"Q8_0","filename":"granite-speech-4.1-2b-nar-Q8_0.gguf","size_bytes":2498105472}, + {"quant":"Q6_K","filename":"granite-speech-4.1-2b-nar-Q6_K.gguf","size_bytes":1977417568}, + {"quant":"Q5_K_M","filename":"granite-speech-4.1-2b-nar-Q5_K_M.gguf","size_bytes":1782089344}, + {"quant":"Q4_K_M","filename":"granite-speech-4.1-2b-nar-Q4_K_M.gguf","size_bytes":1560008832} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":6.07,"ci95":[5.61,6.55],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":886,"del":167,"ins":83},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.33,"ci95":[4.6,6.16],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":431,"del":270,"ins":75},"empty_hyp":1,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":4.08,"ci95":[3.71,4.48],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":718,"del":141,"ins":92},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":6.76,"ci95":[6.02,7.53],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":891,"del":241,"ins":84},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":5.57,"ci95":[5.12,6.08],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":885,"del":134,"ins":177},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.29,"ci95":[1.15,1.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":517,"del":84,"ins":82},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.29,"ci95":[1.15,1.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":518,"del":82,"ins":82},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.29,"ci95":[1.15,1.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":520,"del":81,"ins":82},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.29,"ci95":[1.15,1.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":517,"del":83,"ins":85},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.28,"ci95":[1.15,1.43],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":513,"del":87,"ins":80},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.34,"ci95":[1.19,1.48],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":538,"del":87,"ins":83},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":5748.6,"xrt_compute":6.15,"load_ms":355.1,"mel_ms":2.0,"encode_ms":3191.7,"decode_ms":2554.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6015.4,"xrt_wall":5.87,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1666.0,"xrt_compute":6.6,"load_ms":353.8,"mel_ms":0.7,"encode_ms":990.2,"decode_ms":675.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1749.3,"xrt_wall":6.29,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":5271.3,"xrt_compute":6.7,"load_ms":598.3,"mel_ms":2.0,"encode_ms":3004.2,"decode_ms":2265.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5536.5,"xrt_wall":6.38,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1589.0,"xrt_compute":6.92,"load_ms":536.8,"mel_ms":0.8,"encode_ms":933.8,"decode_ms":654.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1670.6,"xrt_wall":6.58,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":465.7,"xrt_compute":75.88,"load_ms":365.1,"mel_ms":2.0,"encode_ms":267.1,"decode_ms":196.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":670.8,"xrt_wall":52.67,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":153.0,"xrt_compute":71.92,"load_ms":363.0,"mel_ms":0.8,"encode_ms":89.1,"decode_ms":63.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":213.5,"xrt_wall":51.52,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":466.5,"xrt_compute":75.73,"load_ms":553.2,"mel_ms":2.0,"encode_ms":274.3,"decode_ms":190.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":671.7,"xrt_wall":52.6,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":149.0,"xrt_compute":73.82,"load_ms":552.3,"mel_ms":0.8,"encode_ms":87.4,"decode_ms":60.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":209.6,"xrt_wall":52.47,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":17837.8,"xrt_compute":1.98,"load_ms":1947.0,"mel_ms":49.1,"encode_ms":10554.2,"decode_ms":7234.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":18762.3,"xrt_wall":1.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":4977.4,"xrt_compute":2.21,"load_ms":1654.4,"mel_ms":14.9,"encode_ms":3100.1,"decode_ms":1862.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5264.0,"xrt_wall":2.09,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":17880.0,"xrt_compute":1.98,"load_ms":3246.3,"mel_ms":47.1,"encode_ms":10515.9,"decode_ms":7317.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":18915.9,"xrt_wall":1.87,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":4764.6,"xrt_compute":2.31,"load_ms":3264.9,"mel_ms":15.3,"encode_ms":2948.7,"decode_ms":1800.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5056.7,"xrt_wall":2.18,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":9024.1,"xrt_compute":3.92,"load_ms":1555.4,"mel_ms":47.0,"encode_ms":5688.7,"decode_ms":3288.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":10445.7,"xrt_wall":3.38,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2774.3,"xrt_compute":3.96,"load_ms":947.4,"mel_ms":15.2,"encode_ms":1998.8,"decode_ms":760.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4556.9,"xrt_wall":2.41,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":9092.7,"xrt_compute":3.89,"load_ms":2654.0,"mel_ms":44.7,"encode_ms":5669.4,"decode_ms":3378.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":10475.0,"xrt_wall":3.37,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2684.8,"xrt_compute":4.1,"load_ms":2642.8,"mel_ms":14.6,"encode_ms":1927.2,"decode_ms":743.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4439.8,"xrt_wall":2.48,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/granite-speech-4.1-2b-plus.json b/catalog/granite-speech-4.1-2b-plus.json new file mode 100644 index 00000000..b377e89c --- /dev/null +++ b/catalog/granite-speech-4.1-2b-plus.json @@ -0,0 +1,73 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "granite-speech-4.1-2b-plus", + "family": "granite_speech", + "display_name": "granite-speech-4.1-2b-plus", + "params": 2111899452, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "ibm-granite/granite-speech-4.1-2b-plus", + "upstream_commit": "edd3bf5", + "published_repo": "handy-computer/granite-speech-4.1-2b-plus-gguf", + "docs_page": "granite-speech.md", + "languages": ["en","fr","de","es","pt"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false,"note":"upstream lists ASR + speaker diarization, not translation; the family default is true so the GGUF must lower it explicitly"}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["word"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":true,"verified":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"granite-speech-4.1-2b-plus-BF16.gguf","size_bytes":4228988768}, + {"quant":"F16","filename":"granite-speech-4.1-2b-plus-F16.gguf","size_bytes":4229971808}, + {"quant":"Q8_0","filename":"granite-speech-4.1-2b-plus-Q8_0.gguf","size_bytes":2345973152}, + {"quant":"Q6_K","filename":"granite-speech-4.1-2b-plus-Q6_K.gguf","size_bytes":1859821504}, + {"quant":"Q5_K_M","filename":"granite-speech-4.1-2b-plus-Q5_K_M.gguf","size_bytes":1691297088}, + {"quant":"Q4_K_M","filename":"granite-speech-4.1-2b-plus-Q4_K_M.gguf","size_bytes":1489663424} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":8.06,"ci95":[7.46,8.69],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1129,"del":224,"ins":156},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.46,"ci95":[3.97,4.94],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":435,"del":114,"ins":100},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":6.53,"ci95":[5.96,7.12],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1030,"del":202,"ins":291},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":8.82,"ci95":[8.11,9.6],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1113,"del":168,"ins":307},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":10.61,"ci95":[9.59,11.88],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1498,"del":220,"ins":560},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.49,"ci95":[1.35,1.64],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":600,"del":81,"ins":110},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.48,"ci95":[1.34,1.63],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":596,"del":81,"ins":109},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.5,"ci95":[1.35,1.64],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":601,"del":81,"ins":111},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.46,"ci95":[1.32,1.61],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":594,"del":79,"ins":103},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.48,"ci95":[1.33,1.63],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":600,"del":80,"ins":106},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.56,"ci95":[1.42,1.72],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":627,"del":86,"ins":116},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4139.4,"xrt_compute":8.54,"load_ms":403.7,"mel_ms":2.0,"encode_ms":2743.6,"decode_ms":1393.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5227.0,"xrt_wall":6.76,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1500.9,"xrt_compute":7.33,"load_ms":364.5,"mel_ms":0.8,"encode_ms":852.4,"decode_ms":647.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1714.1,"xrt_wall":6.42,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3850.5,"xrt_compute":9.18,"load_ms":544.5,"mel_ms":2.0,"encode_ms":2592.9,"decode_ms":1255.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5153.3,"xrt_wall":6.86,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1474.0,"xrt_compute":7.46,"load_ms":548.6,"mel_ms":0.7,"encode_ms":811.3,"decode_ms":661.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1733.0,"xrt_wall":6.35,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":353.4,"xrt_compute":99.98,"load_ms":383.1,"mel_ms":2.3,"encode_ms":235.1,"decode_ms":116.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1032.9,"xrt_wall":34.21,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":136.9,"xrt_compute":80.36,"load_ms":375.6,"mel_ms":0.7,"encode_ms":79.0,"decode_ms":57.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":271.4,"xrt_wall":40.53,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":350.1,"xrt_compute":100.92,"load_ms":581.4,"mel_ms":2.1,"encode_ms":233.3,"decode_ms":114.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1179.5,"xrt_wall":29.96,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":133.6,"xrt_compute":82.33,"load_ms":557.0,"mel_ms":0.8,"encode_ms":77.6,"decode_ms":55.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":305.2,"xrt_wall":36.04,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":13100.5,"xrt_compute":2.7,"load_ms":2076.7,"mel_ms":43.6,"encode_ms":8628.5,"decode_ms":4428.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":18536.6,"xrt_wall":1.91,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":4342.5,"xrt_compute":2.53,"load_ms":1355.8,"mel_ms":15.3,"encode_ms":2508.4,"decode_ms":1818.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5405.7,"xrt_wall":2.03,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":13353.4,"xrt_compute":2.65,"load_ms":3376.6,"mel_ms":44.5,"encode_ms":8831.7,"decode_ms":4477.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":21212.1,"xrt_wall":1.67,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":4320.9,"xrt_compute":2.55,"load_ms":2918.9,"mel_ms":15.3,"encode_ms":2528.8,"decode_ms":1776.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5879.4,"xrt_wall":1.87,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":7528.1,"xrt_compute":4.69,"load_ms":3697.0,"mel_ms":50.9,"encode_ms":5543.4,"decode_ms":1933.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":12451.5,"xrt_wall":2.84,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2583.9,"xrt_compute":4.26,"load_ms":2276.6,"mel_ms":25.7,"encode_ms":1770.4,"decode_ms":787.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3575.9,"xrt_wall":3.08,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7035.7,"xrt_compute":5.02,"load_ms":3613.0,"mel_ms":51.0,"encode_ms":5014.6,"decode_ms":1970.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13363.6,"xrt_wall":2.64,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2577.1,"xrt_compute":4.27,"load_ms":2851.2,"mel_ms":31.8,"encode_ms":1755.0,"decode_ms":790.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3885.5,"xrt_wall":2.83,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/granite-speech-4.1-2b.json b/catalog/granite-speech-4.1-2b.json new file mode 100644 index 00000000..c0354ed2 --- /dev/null +++ b/catalog/granite-speech-4.1-2b.json @@ -0,0 +1,74 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "granite-speech-4.1-2b", + "family": "granite_speech", + "display_name": "granite-speech-4.1-2b", + "params": 2313228092, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "ibm-granite/granite-speech-4.1-2b", + "upstream_commit": "8f4bb5f", + "published_repo": "handy-computer/granite-speech-4.1-2b-gguf", + "docs_page": "granite-speech.md", + "languages": ["en","fr","de","es","pt","ja"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en","fr","de","es","pt","ja","it","zh"],"pairs":["en>fr","fr>en","en>de","de>en","en>es","es>en","en>pt","pt>en","en>ja","ja>en","en>it","en>zh"],"note":"translation is reached via a separate chat-template prompt; scripts/convert-granite.py is authoritative for the target set"}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"granite-speech-4.1-2b-BF16.gguf","size_bytes":4631640064}, + {"quant":"F16","filename":"granite-speech-4.1-2b-F16.gguf","size_bytes":4632623104}, + {"quant":"Q8_0","filename":"granite-speech-4.1-2b-Q8_0.gguf","size_bytes":2559878848}, + {"quant":"Q6_K","filename":"granite-speech-4.1-2b-Q6_K.gguf","size_bytes":2024967936}, + {"quant":"Q5_K_M","filename":"granite-speech-4.1-2b-Q5_K_M.gguf","size_bytes":1829704544}, + {"quant":"Q4_K_M","filename":"granite-speech-4.1-2b-Q4_K_M.gguf","size_bytes":1602904800} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":6.25,"ci95":[5.66,6.88],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":879,"del":169,"ins":122},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.14,"ci95":[3.68,4.62],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":384,"del":134,"ins":85},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":5.48,"ci95":[4.99,5.97],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":849,"del":149,"ins":279},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":7.61,"ci95":[6.9,8.36],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":922,"del":171,"ins":277},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":6.3,"ci95":[5.75,6.88],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1377,"del":331,"ins":350},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":9.8,"ci95":[9.08,10.66],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1341,"del":163,"ins":601},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.31,"ci95":[1.18,1.43],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":537,"del":63,"ins":94},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.32,"ci95":[1.19,1.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":538,"del":63,"ins":97},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.32,"ci95":[1.19,1.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":543,"del":66,"ins":90},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.29,"ci95":[1.16,1.41],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":532,"del":61,"ins":90},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.33,"ci95":[1.19,1.46],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":546,"del":68,"ins":90},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.37,"ci95":[1.24,1.5],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":559,"del":68,"ins":100},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4255.1,"xrt_compute":8.3,"load_ms":384.8,"mel_ms":2.1,"encode_ms":2860.2,"decode_ms":1392.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5283.6,"xrt_wall":6.69,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1365.8,"xrt_compute":8.05,"load_ms":385.4,"mel_ms":0.8,"encode_ms":885.5,"decode_ms":479.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1578.8,"xrt_wall":6.97,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4126.2,"xrt_compute":8.56,"load_ms":578.6,"mel_ms":2.1,"encode_ms":2675.6,"decode_ms":1448.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5421.1,"xrt_wall":6.52,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1325.1,"xrt_compute":8.3,"load_ms":578.2,"mel_ms":0.7,"encode_ms":826.1,"decode_ms":498.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1594.6,"xrt_wall":6.9,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":457.6,"xrt_compute":77.22,"load_ms":470.8,"mel_ms":2.6,"encode_ms":303.3,"decode_ms":151.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1192.8,"xrt_wall":29.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":139.0,"xrt_compute":79.11,"load_ms":402.3,"mel_ms":0.8,"encode_ms":84.9,"decode_ms":53.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":284.0,"xrt_wall":38.74,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":445.4,"xrt_compute":79.32,"load_ms":679.8,"mel_ms":2.1,"encode_ms":296.9,"decode_ms":146.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1333.2,"xrt_wall":26.5,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":144.9,"xrt_compute":75.92,"load_ms":621.3,"mel_ms":0.7,"encode_ms":89.5,"decode_ms":54.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":325.9,"xrt_wall":33.76,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":12590.3,"xrt_compute":2.81,"load_ms":2189.6,"mel_ms":41.9,"encode_ms":8622.5,"decode_ms":3926.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":17623.1,"xrt_wall":2.0,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3893.3,"xrt_compute":2.83,"load_ms":1427.4,"mel_ms":15.3,"encode_ms":2497.5,"decode_ms":1380.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4943.9,"xrt_wall":2.22,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":13063.3,"xrt_compute":2.7,"load_ms":3581.1,"mel_ms":43.0,"encode_ms":8815.4,"decode_ms":4204.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":20728.5,"xrt_wall":1.7,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3856.3,"xrt_compute":2.85,"load_ms":3586.2,"mel_ms":16.0,"encode_ms":2525.8,"decode_ms":1314.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5443.3,"xrt_wall":2.02,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":6796.7,"xrt_compute":5.2,"load_ms":3097.3,"mel_ms":50.1,"encode_ms":5030.2,"decode_ms":1716.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":11257.8,"xrt_wall":3.14,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2552.8,"xrt_compute":4.31,"load_ms":2990.9,"mel_ms":27.4,"encode_ms":1782.1,"decode_ms":743.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3523.6,"xrt_wall":3.12,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7152.9,"xrt_compute":4.94,"load_ms":3912.4,"mel_ms":53.3,"encode_ms":5406.0,"decode_ms":1693.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13443.5,"xrt_wall":2.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2561.2,"xrt_compute":4.29,"load_ms":4282.1,"mel_ms":30.1,"encode_ms":1777.5,"decode_ms":753.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3925.7,"xrt_wall":2.8,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/granite-speech-5.0-470m-turboctc-nc.json b/catalog/granite-speech-5.0-470m-turboctc-nc.json new file mode 100644 index 00000000..ce79a3c8 --- /dev/null +++ b/catalog/granite-speech-5.0-470m-turboctc-nc.json @@ -0,0 +1,70 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "granite-speech-5.0-470m-turboctc-nc", + "family": "granite5_ctc", + "display_name": "Granite Speech 5.0 470M TurboCTC NC", + "params": 473014752, + "license": { + "spdx": "cc-by-nc-sa-4.0", + "display": "CC-BY-NC-SA-4.0" + }, + "upstream_repo": "ibm-granite/granite-speech-5.0-470m-turboctc-nc", + "upstream_commit": "0eb7b4f", + "published_repo": "handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf", + "docs_page": "granite-speech-5.0-turboctc.md", + "languages": ["en"], + "language_tag_form": "bare-bcp47", + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"granite-speech-5.0-470m-turboctc-nc-BF16.gguf","size_bytes":948103840}, + {"quant":"F16","filename":"granite-speech-5.0-470m-turboctc-nc-F16.gguf","size_bytes":948562592}, + {"quant":"Q8_0","filename":"granite-speech-5.0-470m-turboctc-nc-Q8_0.gguf","size_bytes":505885856}, + {"quant":"Q6_K","filename":"granite-speech-5.0-470m-turboctc-nc-Q6_K.gguf","size_bytes":392115360}, + {"quant":"Q5_K_M","filename":"granite-speech-5.0-470m-turboctc-nc-Q5_K_M.gguf","size_bytes":336016544}, + {"quant":"Q4_K_M","filename":"granite-speech-5.0-470m-turboctc-nc-Q4_K_M.gguf","size_bytes":279393440} + ], + "accuracy_benchmarks": [ + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"BF16","metric":"wer","err_pct":1.29,"ci95":[1.17,1.44],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":520,"del":83,"ins":82},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"F16","metric":"wer","err_pct":1.28,"ci95":[1.16,1.42],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":515,"del":81,"ins":82},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":1.29,"ci95":[1.17,1.44],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":521,"del":83,"ins":82},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q6_K","metric":"wer","err_pct":1.29,"ci95":[1.17,1.43],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":523,"del":81,"ins":82},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q5_K_M","metric":"wer","err_pct":1.29,"ci95":[1.17,1.43],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":520,"del":82,"ins":83},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q4_K_M","metric":"wer","err_pct":1.34,"ci95":[1.21,1.49],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":534,"del":86,"ins":90},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"fleurs","split":"test","language":"en","language_hint":"en","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":4.3,"ci95":[3.86,4.76],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":432,"del":130,"ins":64},"empty_hyp":0,"utts_over_50pct":0} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":763.8,"xrt_compute":46.26,"wall_ms":766.0,"xrt_wall":46.12,"load_ms":96.4,"mel_ms":2.1,"encode_ms":748.6,"decode_ms":13.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":254.8,"xrt_compute":43.17,"wall_ms":256.1,"xrt_wall":42.95,"load_ms":88.3,"mel_ms":0.8,"encode_ms":250.3,"decode_ms":3.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":745.5,"xrt_compute":47.39,"wall_ms":747.7,"xrt_wall":47.26,"load_ms":144.7,"mel_ms":2.3,"encode_ms":729.5,"decode_ms":13.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":243.5,"xrt_compute":45.18,"wall_ms":244.8,"xrt_wall":44.93,"load_ms":134.2,"mel_ms":0.7,"encode_ms":239.2,"decode_ms":3.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":86.8,"xrt_compute":407.09,"wall_ms":91.7,"xrt_wall":385.4,"load_ms":100.5,"mel_ms":2.1,"encode_ms":72.3,"decode_ms":12.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":37.3,"xrt_compute":295.13,"wall_ms":39.5,"xrt_wall":278.41,"load_ms":95.7,"mel_ms":0.7,"encode_ms":33.2,"decode_ms":3.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":83.3,"xrt_compute":424.15,"wall_ms":88.0,"xrt_wall":401.64,"load_ms":154.0,"mel_ms":2.1,"encode_ms":68.8,"decode_ms":12.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":35.2,"xrt_compute":312.54,"wall_ms":37.3,"xrt_wall":294.64,"load_ms":139.8,"mel_ms":0.7,"encode_ms":31.2,"decode_ms":3.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2124.0,"xrt_compute":16.63,"wall_ms":2142.3,"xrt_wall":16.49,"load_ms":350.8,"mel_ms":44.1,"encode_ms":2061.1,"decode_ms":18.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":653.1,"xrt_compute":16.84,"wall_ms":659.1,"xrt_wall":16.69,"load_ms":208.4,"mel_ms":12.9,"encode_ms":636.6,"decode_ms":3.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2129.3,"xrt_compute":16.59,"wall_ms":2148.6,"xrt_wall":16.44,"load_ms":616.1,"mel_ms":43.7,"encode_ms":2066.5,"decode_ms":19.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":614.3,"xrt_compute":17.91,"wall_ms":620.7,"xrt_wall":17.72,"load_ms":366.5,"mel_ms":14.5,"encode_ms":596.3,"decode_ms":3.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1336.0,"xrt_compute":26.45,"wall_ms":1512.3,"xrt_wall":23.36,"load_ms":130.3,"mel_ms":47.1,"encode_ms":1273.9,"decode_ms":15.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":605.0,"xrt_compute":18.18,"wall_ms":668.1,"xrt_wall":16.47,"load_ms":131.7,"mel_ms":14.5,"encode_ms":587.1,"decode_ms":3.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1319.9,"xrt_compute":26.77,"wall_ms":1498.2,"xrt_wall":23.58,"load_ms":219.8,"mel_ms":46.4,"encode_ms":1258.4,"decode_ms":15.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":582.6,"xrt_compute":18.88,"wall_ms":628.5,"xrt_wall":17.5,"load_ms":219.4,"mel_ms":14.1,"encode_ms":565.1,"decode_ms":3.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/granite-speech-5.0-470m-turboctc.json b/catalog/granite-speech-5.0-470m-turboctc.json new file mode 100644 index 00000000..9d6ca32a --- /dev/null +++ b/catalog/granite-speech-5.0-470m-turboctc.json @@ -0,0 +1,70 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "granite-speech-5.0-470m-turboctc", + "family": "granite5_ctc", + "display_name": "Granite Speech 5.0 470M TurboCTC", + "params": 473014752, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "ibm-granite/granite-speech-5.0-470m-turboctc", + "upstream_commit": "18ca3c1", + "published_repo": "handy-computer/granite-speech-5.0-470m-turboctc-gguf", + "docs_page": "granite-speech-5.0-turboctc.md", + "languages": ["en"], + "language_tag_form": "bare-bcp47", + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"granite-speech-5.0-470m-turboctc-BF16.gguf","size_bytes":947824480}, + {"quant":"F16","filename":"granite-speech-5.0-470m-turboctc-F16.gguf","size_bytes":947824480}, + {"quant":"Q8_0","filename":"granite-speech-5.0-470m-turboctc-Q8_0.gguf","size_bytes":505606496}, + {"quant":"Q6_K","filename":"granite-speech-5.0-470m-turboctc-Q6_K.gguf","size_bytes":391836000}, + {"quant":"Q5_K_M","filename":"granite-speech-5.0-470m-turboctc-Q5_K_M.gguf","size_bytes":335737184}, + {"quant":"Q4_K_M","filename":"granite-speech-5.0-470m-turboctc-Q4_K_M.gguf","size_bytes":279114080} + ], + "accuracy_benchmarks": [ + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"BF16","metric":"wer","err_pct":1.34,"ci95":[1.21,1.47],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":529,"del":87,"ins":92},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"F16","metric":"wer","err_pct":1.33,"ci95":[1.2,1.47],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":527,"del":87,"ins":92},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":1.33,"ci95":[1.2,1.46],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":527,"del":85,"ins":92},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q6_K","metric":"wer","err_pct":1.33,"ci95":[1.2,1.46],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":524,"del":88,"ins":93},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q5_K_M","metric":"wer","err_pct":1.34,"ci95":[1.21,1.47],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":531,"del":86,"ins":93},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q4_K_M","metric":"wer","err_pct":1.35,"ci95":[1.22,1.49],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":533,"del":91,"ins":92},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"fleurs","split":"test","language":"en","language_hint":"en","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":4.61,"ci95":[4.19,5.07],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":"9daf396","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":477,"del":115,"ins":79},"empty_hyp":0,"utts_over_50pct":1} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":769.2,"xrt_compute":45.93,"wall_ms":771.4,"xrt_wall":45.8,"load_ms":100.3,"mel_ms":2.3,"encode_ms":753.1,"decode_ms":13.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":254.1,"xrt_compute":43.29,"wall_ms":255.5,"xrt_wall":43.06,"load_ms":88.0,"mel_ms":0.7,"encode_ms":249.2,"decode_ms":4.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":752.5,"xrt_compute":46.95,"wall_ms":754.7,"xrt_wall":46.82,"load_ms":148.4,"mel_ms":2.2,"encode_ms":736.4,"decode_ms":13.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":243.4,"xrt_compute":45.19,"wall_ms":244.7,"xrt_wall":44.95,"load_ms":136.4,"mel_ms":0.8,"encode_ms":239.0,"decode_ms":3.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":95.2,"xrt_compute":371.09,"wall_ms":100.0,"xrt_wall":353.45,"load_ms":105.0,"mel_ms":2.1,"encode_ms":80.6,"decode_ms":12.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":38.7,"xrt_compute":284.51,"wall_ms":40.9,"xrt_wall":268.73,"load_ms":91.1,"mel_ms":0.8,"encode_ms":34.5,"decode_ms":3.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":86.1,"xrt_compute":410.2,"wall_ms":90.9,"xrt_wall":388.76,"load_ms":151.2,"mel_ms":2.1,"encode_ms":71.6,"decode_ms":12.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":36.5,"xrt_compute":301.3,"wall_ms":38.6,"xrt_wall":284.88,"load_ms":138.9,"mel_ms":0.7,"encode_ms":32.5,"decode_ms":3.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2139.7,"xrt_compute":16.51,"wall_ms":2158.6,"xrt_wall":16.37,"load_ms":354.8,"mel_ms":43.4,"encode_ms":2077.4,"decode_ms":18.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":651.9,"xrt_compute":16.87,"wall_ms":657.9,"xrt_wall":16.72,"load_ms":211.1,"mel_ms":13.3,"encode_ms":635.1,"decode_ms":3.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2131.2,"xrt_compute":16.58,"wall_ms":2150.7,"xrt_wall":16.43,"load_ms":643.3,"mel_ms":43.4,"encode_ms":2068.7,"decode_ms":19.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":614.7,"xrt_compute":17.9,"wall_ms":621.0,"xrt_wall":17.71,"load_ms":365.4,"mel_ms":13.5,"encode_ms":597.5,"decode_ms":3.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1313.9,"xrt_compute":26.89,"wall_ms":1487.8,"xrt_wall":23.75,"load_ms":132.6,"mel_ms":45.0,"encode_ms":1254.0,"decode_ms":15.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":587.5,"xrt_compute":18.72,"wall_ms":648.6,"xrt_wall":16.96,"load_ms":131.1,"mel_ms":14.8,"encode_ms":569.2,"decode_ms":3.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1306.7,"xrt_compute":27.04,"wall_ms":1484.3,"xrt_wall":23.8,"load_ms":219.5,"mel_ms":44.9,"encode_ms":1247.1,"decode_ms":14.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":588.5,"xrt_compute":18.69,"wall_ms":649.3,"xrt_wall":16.94,"load_ms":217.4,"mel_ms":14.1,"encode_ms":571.0,"decode_ms":3.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/medasr.json b/catalog/medasr.json new file mode 100644 index 00000000..66822b71 --- /dev/null +++ b/catalog/medasr.json @@ -0,0 +1,71 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "medasr", + "family": "medasr", + "display_name": "medasr", + "params": 105316112, + "license": { + "spdx": "other", + "display": "Health AI Developer Foundations", + "name": "health-ai-developer-foundations", + "link": "https://developers.google.com/health-ai-developer-foundations/terms" + }, + "upstream_repo": "google/medasr", + "upstream_commit": "ae1e484", + "published_repo": "handy-computer/medasr-gguf", + "docs_page": "medasr.md", + "languages": ["en"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"medasr-F32.gguf","size_bytes":421301440}, + {"quant":"F16","filename":"medasr-F16.gguf","size_bytes":211455168}, + {"quant":"Q8_0","filename":"medasr-Q8_0.gguf","size_bytes":127712448}, + {"quant":"Q6_K","filename":"medasr-Q6_K.gguf","size_bytes":106094784}, + {"quant":"Q5_K_M","filename":"medasr-Q5_K_M.gguf","size_bytes":94239936}, + {"quant":"Q4_K_M","filename":"medasr-Q4_K_M.gguf","size_bytes":83082432} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":37.48,"ci95":[35.83,39.09],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3705,"del":1440,"ins":311},"empty_hyp":6,"utts_over_50pct":153,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":17.88,"ci95":[17.34,18.45],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7183,"del":1617,"ins":679},"empty_hyp":14,"utts_over_50pct":168,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":17.88,"ci95":[17.34,18.46],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7184,"del":1616,"ins":680},"empty_hyp":14,"utts_over_50pct":168,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":17.86,"ci95":[17.33,18.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7180,"del":1616,"ins":673},"empty_hyp":14,"utts_over_50pct":168,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":17.93,"ci95":[17.4,18.53],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7189,"del":1637,"ins":683},"empty_hyp":13,"utts_over_50pct":166,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":17.91,"ci95":[17.38,18.49],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7201,"del":1613,"ins":682},"empty_hyp":13,"utts_over_50pct":165,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":18.14,"ci95":[17.6,18.73],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7241,"del":1686,"ins":693},"empty_hyp":13,"utts_over_50pct":166,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":632.3,"xrt_compute":55.88,"load_ms":47.4,"mel_ms":2.2,"encode_ms":629.6,"decode_ms":0.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":633.0,"xrt_wall":55.81,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":179.3,"xrt_compute":61.33,"load_ms":44.5,"mel_ms":0.9,"encode_ms":178.3,"decode_ms":0.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":179.8,"xrt_wall":61.16,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":564.4,"xrt_compute":62.6,"load_ms":58.5,"mel_ms":2.2,"encode_ms":561.6,"decode_ms":0.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":565.1,"xrt_wall":62.52,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":161.6,"xrt_compute":68.09,"load_ms":52.5,"mel_ms":0.8,"encode_ms":160.6,"decode_ms":0.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":162.1,"xrt_wall":67.87,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":39.5,"xrt_compute":893.98,"load_ms":54.6,"mel_ms":2.1,"encode_ms":36.9,"decode_ms":0.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":41.1,"xrt_wall":859.99,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":16.5,"xrt_compute":667.76,"load_ms":44.9,"mel_ms":0.9,"encode_ms":15.4,"decode_ms":0.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":17.5,"xrt_wall":627.42,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":38.6,"xrt_compute":915.05,"load_ms":61.0,"mel_ms":2.1,"encode_ms":36.0,"decode_ms":0.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":40.1,"xrt_wall":881.47,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":16.7,"xrt_compute":658.45,"load_ms":55.3,"mel_ms":0.9,"encode_ms":15.7,"decode_ms":0.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":17.8,"xrt_wall":619.58,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1498.5,"xrt_compute":23.58,"load_ms":128.3,"mel_ms":59.1,"encode_ms":1438.7,"decode_ms":0.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1501.0,"xrt_wall":23.54,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":466.4,"xrt_compute":23.58,"load_ms":92.4,"mel_ms":18.3,"encode_ms":448.0,"decode_ms":0.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":468.9,"xrt_wall":23.46,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1456.8,"xrt_compute":24.25,"load_ms":184.9,"mel_ms":59.2,"encode_ms":1397.0,"decode_ms":0.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1459.4,"xrt_wall":24.21,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":400.2,"xrt_compute":27.48,"load_ms":115.8,"mel_ms":18.5,"encode_ms":381.6,"decode_ms":0.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":402.7,"xrt_wall":27.32,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":484.3,"xrt_compute":72.95,"load_ms":121.1,"mel_ms":62.6,"encode_ms":409.8,"decode_ms":11.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":488.0,"xrt_wall":72.4,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":157.0,"xrt_compute":70.04,"load_ms":71.7,"mel_ms":24.3,"encode_ms":127.5,"decode_ms":5.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":160.9,"xrt_wall":68.35,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":480.1,"xrt_compute":73.59,"load_ms":173.7,"mel_ms":60.9,"encode_ms":407.1,"decode_ms":12.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":483.9,"xrt_wall":73.01,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":159.6,"xrt_compute":68.93,"load_ms":88.0,"mel_ms":26.4,"encode_ms":127.6,"decode_ms":5.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":163.0,"xrt_wall":67.48,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/moonshine-base-ar.json b/catalog/moonshine-base-ar.json new file mode 100644 index 00000000..ac70ea68 --- /dev/null +++ b/catalog/moonshine-base-ar.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-base-ar", + "family": "moonshine", + "display_name": "moonshine-base-ar", + "params": 61513920, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-base-ar", + "upstream_commit": "264cc18", + "published_repo": "handy-computer/moonshine-base-ar-gguf", + "docs_page": "moonshine.md", + "languages": ["ar"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-base-ar-F32.gguf","size_bytes":247657088}, + {"quant":"F16","filename":"moonshine-base-ar-F16.gguf","size_bytes":131789440}, + {"quant":"Q8_0","filename":"moonshine-base-ar-Q8_0.gguf","size_bytes":77476480} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":24.62,"ci95":[21.58,27.85],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1316,"del":332,"ins":380},"empty_hyp":0,"utts_over_50pct":40,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"F32","metric":"wer","err_pct":24.45,"ci95":[21.6,27.71],"n_utts":428,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1302,"del":335,"ins":377},"empty_hyp":0,"utts_over_50pct":38,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"F16","metric":"wer","err_pct":24.45,"ci95":[21.6,27.71],"n_utts":428,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1302,"del":335,"ins":377},"empty_hyp":0,"utts_over_50pct":38,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ar", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":520.1,"xrt_compute":49.49,"wall_ms":520.6,"xrt_wall":49.44,"load_ms":54.6,"mel_ms":0.0,"encode_ms":169.4,"decode_ms":350.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":224.7,"xrt_compute":48.96,"wall_ms":224.8,"xrt_wall":48.93,"load_ms":51.9,"mel_ms":0.0,"encode_ms":63.9,"decode_ms":160.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":568.8,"xrt_compute":45.25,"wall_ms":569.0,"xrt_wall":45.24,"load_ms":59.5,"mel_ms":0.0,"encode_ms":130.5,"decode_ms":438.3,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":275.6,"xrt_compute":39.91,"wall_ms":275.7,"xrt_wall":39.9,"load_ms":51.5,"mel_ms":0.0,"encode_ms":35.7,"decode_ms":239.9,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":2452.2,"xrt_compute":10.5,"wall_ms":2457.7,"xrt_wall":10.47,"load_ms":115.3,"mel_ms":0.0,"encode_ms":419.4,"decode_ms":2032.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":956.0,"xrt_compute":11.51,"wall_ms":956.1,"xrt_wall":11.51,"load_ms":118.6,"mel_ms":0.0,"encode_ms":150.0,"decode_ms":806.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":1314.4,"xrt_compute":19.58,"wall_ms":1314.8,"xrt_wall":19.58,"load_ms":100.6,"mel_ms":0.0,"encode_ms":209.0,"decode_ms":1105.4,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":674.8,"xrt_compute":16.3,"wall_ms":675.1,"xrt_wall":16.29,"load_ms":101.2,"mel_ms":0.0,"encode_ms":84.7,"decode_ms":590.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-base-ja.json b/catalog/moonshine-base-ja.json new file mode 100644 index 00000000..06b60680 --- /dev/null +++ b/catalog/moonshine-base-ja.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-base-ja", + "family": "moonshine", + "display_name": "moonshine-base-ja", + "params": 61513920, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-base-ja", + "upstream_commit": "f9d4e6a", + "published_repo": "handy-computer/moonshine-base-ja-gguf", + "docs_page": "moonshine.md", + "languages": ["ja"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-base-ja-F32.gguf","size_bytes":247657088}, + {"quant":"F16","filename":"moonshine-base-ja-F16.gguf","size_bytes":131789440}, + {"quant":"Q8_0","filename":"moonshine-base-ja-Q8_0.gguf","size_bytes":77476480} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":11.11,"ci95":[10.05,12.36],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2075,"del":750,"ins":803},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"F32","metric":"cer","err_pct":10.84,"ci95":[9.82,11.93],"n_utts":650,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2046,"del":765,"ins":727},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"F16","metric":"cer","err_pct":10.81,"ci95":[9.8,11.89],"n_utts":650,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2048,"del":764,"ins":718},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ja", + "metric": "cer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":561.4,"xrt_compute":50.23,"wall_ms":561.8,"xrt_wall":50.2,"load_ms":55.5,"mel_ms":0.0,"encode_ms":186.1,"decode_ms":375.3,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":148.0,"xrt_compute":74.19,"wall_ms":148.2,"xrt_wall":74.1,"load_ms":50.9,"mel_ms":0.0,"encode_ms":57.4,"decode_ms":90.6,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":587.1,"xrt_compute":48.03,"wall_ms":587.3,"xrt_wall":48.02,"load_ms":51.8,"mel_ms":0.0,"encode_ms":141.9,"decode_ms":445.2,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":164.8,"xrt_compute":66.62,"wall_ms":164.9,"xrt_wall":66.58,"load_ms":51.2,"mel_ms":0.0,"encode_ms":34.5,"decode_ms":130.3,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":2510.6,"xrt_compute":11.23,"wall_ms":2516.8,"xrt_wall":11.2,"load_ms":176.5,"mel_ms":0.0,"encode_ms":421.8,"decode_ms":2088.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":592.1,"xrt_compute":18.54,"wall_ms":592.2,"xrt_wall":18.54,"load_ms":115.7,"mel_ms":0.0,"encode_ms":147.4,"decode_ms":444.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":1409.8,"xrt_compute":20.0,"wall_ms":1410.3,"xrt_wall":20.0,"load_ms":100.5,"mel_ms":0.0,"encode_ms":231.2,"decode_ms":1178.6,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":409.5,"xrt_compute":26.82,"wall_ms":409.7,"xrt_wall":26.8,"load_ms":101.2,"mel_ms":0.0,"encode_ms":81.4,"decode_ms":328.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-base-ko.json b/catalog/moonshine-base-ko.json new file mode 100644 index 00000000..e6ed2a8c --- /dev/null +++ b/catalog/moonshine-base-ko.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-base-ko", + "family": "moonshine", + "display_name": "moonshine-base-ko", + "params": 61513920, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-base-ko", + "upstream_commit": "fff6035", + "published_repo": "handy-computer/moonshine-base-ko-gguf", + "docs_page": "moonshine.md", + "languages": ["ko"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-base-ko-F32.gguf","size_bytes":247657088}, + {"quant":"F16","filename":"moonshine-base-ko-F16.gguf","size_bytes":131789440}, + {"quant":"Q8_0","filename":"moonshine-base-ko-Q8_0.gguf","size_bytes":77476480} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":8.12,"ci95":[6.89,9.5],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":701,"del":646,"ins":88},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"F32","metric":"cer","err_pct":8.0,"ci95":[6.76,9.28],"n_utts":382,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":700,"del":620,"ins":95},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"F16","metric":"cer","err_pct":8.0,"ci95":[6.76,9.28],"n_utts":382,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":700,"del":620,"ins":95},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ko", + "metric": "cer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":661.9,"xrt_compute":38.98,"wall_ms":662.2,"xrt_wall":38.96,"load_ms":55.9,"mel_ms":0.0,"encode_ms":166.6,"decode_ms":495.2,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":180.6,"xrt_compute":60.81,"wall_ms":180.7,"xrt_wall":60.76,"load_ms":51.5,"mel_ms":0.0,"encode_ms":56.9,"decode_ms":123.6,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":724.0,"xrt_compute":35.64,"wall_ms":724.2,"xrt_wall":35.63,"load_ms":52.3,"mel_ms":0.0,"encode_ms":126.3,"decode_ms":597.7,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":217.3,"xrt_compute":50.52,"wall_ms":217.5,"xrt_wall":50.49,"load_ms":53.3,"mel_ms":0.0,"encode_ms":34.5,"decode_ms":182.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":3194.2,"xrt_compute":8.08,"wall_ms":3200.5,"xrt_wall":8.06,"load_ms":183.4,"mel_ms":0.0,"encode_ms":387.7,"decode_ms":2806.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":764.8,"xrt_compute":14.36,"wall_ms":764.8,"xrt_wall":14.36,"load_ms":117.1,"mel_ms":0.0,"encode_ms":145.1,"decode_ms":619.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":1743.3,"xrt_compute":14.8,"wall_ms":1743.6,"xrt_wall":14.8,"load_ms":99.8,"mel_ms":0.0,"encode_ms":207.2,"decode_ms":1536.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":545.1,"xrt_compute":20.14,"wall_ms":545.4,"xrt_wall":20.13,"load_ms":99.8,"mel_ms":0.0,"encode_ms":85.0,"decode_ms":460.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-base-uk.json b/catalog/moonshine-base-uk.json new file mode 100644 index 00000000..7fa0315e --- /dev/null +++ b/catalog/moonshine-base-uk.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-base-uk", + "family": "moonshine", + "display_name": "moonshine-base-uk", + "params": 61513920, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-base-uk", + "upstream_commit": "a52b2e9", + "published_repo": "handy-computer/moonshine-base-uk-gguf", + "docs_page": "moonshine.md", + "languages": ["uk"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-base-uk-F32.gguf","size_bytes":247657120}, + {"quant":"F16","filename":"moonshine-base-uk-F16.gguf","size_bytes":131789472}, + {"quant":"Q8_0","filename":"moonshine-base-uk-Q8_0.gguf","size_bytes":77476512} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":14.38,"ci95":[13.42,15.32],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1608,"del":194,"ins":265},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"F32","metric":"wer","err_pct":14.49,"ci95":[13.45,15.47],"n_utts":750,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1615,"del":203,"ins":264},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"F16","metric":"wer","err_pct":14.49,"ci95":[13.45,15.47],"n_utts":750,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1615,"del":203,"ins":264},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "uk", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":590.5,"xrt_compute":48.97,"wall_ms":590.9,"xrt_wall":48.94,"load_ms":55.4,"mel_ms":0.0,"encode_ms":189.5,"decode_ms":401.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":121.6,"xrt_compute":90.33,"wall_ms":121.7,"xrt_wall":90.2,"load_ms":56.5,"mel_ms":0.0,"encode_ms":56.7,"decode_ms":64.9,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":595.3,"xrt_compute":48.58,"wall_ms":595.5,"xrt_wall":48.57,"load_ms":53.1,"mel_ms":0.0,"encode_ms":148.5,"decode_ms":446.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":131.1,"xrt_compute":83.76,"wall_ms":131.2,"xrt_wall":83.69,"load_ms":53.3,"mel_ms":0.0,"encode_ms":35.0,"decode_ms":96.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":2596.0,"xrt_compute":11.14,"wall_ms":2602.4,"xrt_wall":11.11,"load_ms":156.2,"mel_ms":0.0,"encode_ms":430.0,"decode_ms":2166.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":465.1,"xrt_compute":23.61,"wall_ms":465.1,"xrt_wall":23.61,"load_ms":117.5,"mel_ms":0.0,"encode_ms":144.1,"decode_ms":321.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":1427.3,"xrt_compute":20.26,"wall_ms":1427.8,"xrt_wall":20.26,"load_ms":99.9,"mel_ms":0.0,"encode_ms":237.6,"decode_ms":1189.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":329.5,"xrt_compute":33.32,"wall_ms":329.8,"xrt_wall":33.29,"load_ms":102.5,"mel_ms":0.0,"encode_ms":82.4,"decode_ms":247.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-base-vi.json b/catalog/moonshine-base-vi.json new file mode 100644 index 00000000..5c5025a9 --- /dev/null +++ b/catalog/moonshine-base-vi.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-base-vi", + "family": "moonshine", + "display_name": "moonshine-base-vi", + "params": 61513920, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-base-vi", + "upstream_commit": "0ee9ce9", + "published_repo": "handy-computer/moonshine-base-vi-gguf", + "docs_page": "moonshine.md", + "languages": ["vi"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-base-vi-F32.gguf","size_bytes":247657120}, + {"quant":"F16","filename":"moonshine-base-vi-F16.gguf","size_bytes":131789472}, + {"quant":"Q8_0","filename":"moonshine-base-vi-Q8_0.gguf","size_bytes":77476512} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":9.96,"ci95":[9.25,10.79],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2019,"del":295,"ins":277},"empty_hyp":1,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"F32","metric":"wer","err_pct":9.67,"ci95":[9.06,10.33],"n_utts":857,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2013,"del":228,"ins":274},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"F16","metric":"wer","err_pct":9.66,"ci95":[9.06,10.33],"n_utts":857,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2013,"del":227,"ins":274},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "vi", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":460.8,"xrt_compute":54.43,"wall_ms":461.2,"xrt_wall":54.39,"load_ms":50.3,"mel_ms":0.0,"encode_ms":162.8,"decode_ms":297.9,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":149.5,"xrt_compute":73.42,"wall_ms":149.8,"xrt_wall":73.32,"load_ms":50.7,"mel_ms":0.0,"encode_ms":56.9,"decode_ms":92.7,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":500.9,"xrt_compute":50.07,"wall_ms":501.0,"xrt_wall":50.06,"load_ms":51.1,"mel_ms":0.0,"encode_ms":130.7,"decode_ms":370.2,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":163.5,"xrt_compute":67.16,"wall_ms":163.6,"xrt_wall":67.12,"load_ms":56.9,"mel_ms":0.0,"encode_ms":34.5,"decode_ms":129.0,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":2093.4,"xrt_compute":11.98,"wall_ms":2099.6,"xrt_wall":11.95,"load_ms":146.1,"mel_ms":0.0,"encode_ms":371.7,"decode_ms":1721.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":607.1,"xrt_compute":18.09,"wall_ms":607.1,"xrt_wall":18.08,"load_ms":117.7,"mel_ms":0.0,"encode_ms":146.9,"decode_ms":460.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":1158.7,"xrt_compute":21.65,"wall_ms":1159.1,"xrt_wall":21.64,"load_ms":133.1,"mel_ms":0.0,"encode_ms":203.6,"decode_ms":955.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":418.9,"xrt_compute":26.21,"wall_ms":419.0,"xrt_wall":26.2,"load_ms":98.7,"mel_ms":0.0,"encode_ms":83.0,"decode_ms":335.9,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-base-zh.json b/catalog/moonshine-base-zh.json new file mode 100644 index 00000000..b5a93377 --- /dev/null +++ b/catalog/moonshine-base-zh.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-base-zh", + "family": "moonshine", + "display_name": "moonshine-base-zh", + "params": 61513920, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-base-zh", + "upstream_commit": "1df4f95", + "published_repo": "handy-computer/moonshine-base-zh-gguf", + "docs_page": "moonshine.md", + "languages": ["zh"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-base-zh-F32.gguf","size_bytes":247657088}, + {"quant":"F16","filename":"moonshine-base-zh-F16.gguf","size_bytes":131789440}, + {"quant":"Q8_0","filename":"moonshine-base-zh-Q8_0.gguf","size_bytes":77476480} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":17.79,"ci95":[15.81,19.77],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2587,"del":1557,"ins":2200},"empty_hyp":0,"utts_over_50pct":59,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"F32","metric":"cer","err_pct":16.65,"ci95":[14.89,18.64],"n_utts":945,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2450,"del":1561,"ins":1927},"empty_hyp":0,"utts_over_50pct":47,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"F16","metric":"cer","err_pct":16.65,"ci95":[14.89,18.64],"n_utts":945,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2450,"del":1561,"ins":1927},"empty_hyp":0,"utts_over_50pct":47,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "zh", + "metric": "cer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":643.5,"xrt_compute":48.36,"wall_ms":644.0,"xrt_wall":48.32,"load_ms":56.9,"mel_ms":0.0,"encode_ms":208.3,"decode_ms":435.2,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":123.5,"xrt_compute":89.07,"wall_ms":123.7,"xrt_wall":88.95,"load_ms":55.3,"mel_ms":0.0,"encode_ms":64.7,"decode_ms":58.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":666.7,"xrt_compute":46.68,"wall_ms":666.8,"xrt_wall":46.67,"load_ms":55.1,"mel_ms":0.0,"encode_ms":159.4,"decode_ms":507.3,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":126.2,"xrt_compute":87.18,"wall_ms":126.3,"xrt_wall":87.12,"load_ms":51.5,"mel_ms":0.0,"encode_ms":36.6,"decode_ms":89.6,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":2837.3,"xrt_compute":10.97,"wall_ms":2844.4,"xrt_wall":10.94,"load_ms":151.7,"mel_ms":0.0,"encode_ms":517.9,"decode_ms":2319.3,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":429.5,"xrt_compute":25.61,"wall_ms":429.5,"xrt_wall":25.61,"load_ms":116.4,"mel_ms":0.0,"encode_ms":147.5,"decode_ms":282.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":1627.9,"xrt_compute":19.12,"wall_ms":1628.3,"xrt_wall":19.11,"load_ms":100.7,"mel_ms":0.0,"encode_ms":261.5,"decode_ms":1366.3,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":297.9,"xrt_compute":36.92,"wall_ms":298.1,"xrt_wall":36.9,"load_ms":99.6,"mel_ms":0.0,"encode_ms":83.8,"decode_ms":214.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-base.json b/catalog/moonshine-base.json new file mode 100644 index 00000000..1ab8b7ad --- /dev/null +++ b/catalog/moonshine-base.json @@ -0,0 +1,55 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-base", + "family": "moonshine", + "display_name": "moonshine-base", + "params": 61513920, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-base", + "upstream_commit": "7a73d8d", + "published_repo": "handy-computer/moonshine-base-gguf", + "docs_page": "moonshine.md", + "languages": ["en"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-base-F32.gguf","size_bytes":247657088}, + {"quant":"F16","filename":"moonshine-base-F16.gguf","size_bytes":131789440}, + {"quant":"Q8_0","filename":"moonshine-base-Q8_0.gguf","size_bytes":77476480} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":12.25,"ci95":[10.67,14.04],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":897,"del":634,"ins":252},"empty_hyp":20,"utts_over_50pct":23,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":3.28,"ci95":[3.07,3.5],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1291,"del":204,"ins":245},"empty_hyp":2,"utts_over_50pct":19,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":3.28,"ci95":[3.07,3.5],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1290,"del":204,"ins":244},"empty_hyp":2,"utts_over_50pct":19,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.26,"ci95":[3.04,3.49],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1285,"del":205,"ins":240},"empty_hyp":2,"utts_over_50pct":18,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":690.5,"xrt_compute":51.17,"load_ms":56.7,"mel_ms":0.0,"encode_ms":224.9,"decode_ms":465.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":691.0,"xrt_wall":51.13,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":100.2,"xrt_compute":109.83,"load_ms":52.4,"mel_ms":0.0,"encode_ms":65.1,"decode_ms":35.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":100.4,"xrt_wall":109.61,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":739.8,"xrt_compute":47.76,"load_ms":60.6,"mel_ms":0.0,"encode_ms":200.7,"decode_ms":539.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":740.1,"xrt_wall":47.74,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":89.2,"xrt_compute":123.33,"load_ms":58.2,"mel_ms":0.0,"encode_ms":37.8,"decode_ms":51.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":89.3,"xrt_wall":123.17,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3204.5,"xrt_compute":11.03,"load_ms":112.2,"mel_ms":0.0,"encode_ms":607.3,"decode_ms":2597.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3213.3,"xrt_wall":11.0,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":306.0,"xrt_compute":35.95,"load_ms":151.9,"mel_ms":0.0,"encode_ms":145.3,"decode_ms":160.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":306.0,"xrt_wall":35.95,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1742.4,"xrt_compute":20.28,"load_ms":98.4,"mel_ms":0.0,"encode_ms":299.9,"decode_ms":1442.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1743.1,"xrt_wall":20.27,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":216.1,"xrt_compute":50.9,"load_ms":127.8,"mel_ms":0.0,"encode_ms":82.8,"decode_ms":133.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":216.3,"xrt_wall":50.86,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/moonshine-streaming-medium.json b/catalog/moonshine-streaming-medium.json new file mode 100644 index 00000000..4832ac36 --- /dev/null +++ b/catalog/moonshine-streaming-medium.json @@ -0,0 +1,55 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-streaming-medium", + "family": "moonshine_streaming", + "display_name": "moonshine-streaming-medium", + "params": 265898113, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-streaming-medium", + "upstream_commit": "57b8436", + "published_repo": "handy-computer/moonshine-streaming-medium-gguf", + "docs_page": "moonshine-streaming.md", + "languages": ["en"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-streaming-medium-F32.gguf","size_bytes":1065204640}, + {"quant":"F16","filename":"moonshine-streaming-medium-F16.gguf","size_bytes":533781408}, + {"quant":"Q8_0","filename":"moonshine-streaming-medium-Q8_0.gguf","size_bytes":295793568} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":7.87,"ci95":[6.97,8.85],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":698,"del":301,"ins":147},"empty_hyp":7,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.16,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.16,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.16,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1120.8,"xrt_compute":31.52,"load_ms":104.8,"mel_ms":0.0,"encode_ms":653.9,"decode_ms":466.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1122.5,"xrt_wall":31.48,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":235.2,"xrt_compute":46.77,"load_ms":101.4,"mel_ms":0.0,"encode_ms":167.6,"decode_ms":67.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":235.5,"xrt_wall":46.7,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":978.3,"xrt_compute":36.12,"load_ms":113.6,"mel_ms":0.0,"encode_ms":114.3,"decode_ms":864.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":981.1,"xrt_wall":36.01,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":127.0,"xrt_compute":86.61,"load_ms":102.4,"mel_ms":0.0,"encode_ms":27.9,"decode_ms":99.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":127.4,"xrt_wall":86.36,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":6032.8,"xrt_compute":5.86,"load_ms":260.9,"mel_ms":0.0,"encode_ms":2284.3,"decode_ms":3748.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6043.4,"xrt_wall":5.85,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":816.9,"xrt_compute":13.47,"load_ms":412.0,"mel_ms":0.0,"encode_ms":498.2,"decode_ms":318.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":817.0,"xrt_wall":13.46,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3968.6,"xrt_compute":8.9,"load_ms":155.1,"mel_ms":0.0,"encode_ms":735.8,"decode_ms":3232.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3970.3,"xrt_wall":8.9,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":559.6,"xrt_compute":19.66,"load_ms":303.8,"mel_ms":0.0,"encode_ms":228.4,"decode_ms":331.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":560.0,"xrt_wall":19.64,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/moonshine-streaming-small.json b/catalog/moonshine-streaming-small.json new file mode 100644 index 00000000..a8580849 --- /dev/null +++ b/catalog/moonshine-streaming-small.json @@ -0,0 +1,55 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-streaming-small", + "family": "moonshine_streaming", + "display_name": "moonshine-streaming-small", + "params": 140135225, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-streaming-small", + "upstream_commit": "2c03650", + "published_repo": "handy-computer/moonshine-streaming-small-gguf", + "docs_page": "moonshine-streaming.md", + "languages": ["en"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-streaming-small-F32.gguf","size_bytes":562146912}, + {"quant":"F16","filename":"moonshine-streaming-small-F16.gguf","size_bytes":282092128}, + {"quant":"Q8_0","filename":"moonshine-streaming-small-Q8_0.gguf","size_bytes":198506848} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":8.55,"ci95":[7.84,9.34],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":880,"del":189,"ins":175},"empty_hyp":1,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.53,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.53,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.54,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":722.4,"xrt_compute":48.91,"load_ms":79.2,"mel_ms":0.0,"encode_ms":457.3,"decode_ms":265.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":723.7,"xrt_wall":48.82,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":172.5,"xrt_compute":63.75,"load_ms":82.9,"mel_ms":0.0,"encode_ms":135.8,"decode_ms":36.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":172.8,"xrt_wall":63.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":603.0,"xrt_compute":58.59,"load_ms":89.0,"mel_ms":0.0,"encode_ms":77.8,"decode_ms":525.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":604.6,"xrt_wall":58.44,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":85.0,"xrt_compute":129.36,"load_ms":80.9,"mel_ms":0.0,"encode_ms":21.0,"decode_ms":64.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":85.4,"xrt_wall":128.84,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3791.5,"xrt_compute":9.32,"load_ms":279.7,"mel_ms":0.0,"encode_ms":1742.0,"decode_ms":2049.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3812.4,"xrt_wall":9.27,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":619.7,"xrt_compute":17.75,"load_ms":192.4,"mel_ms":0.0,"encode_ms":461.3,"decode_ms":158.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":619.8,"xrt_wall":17.75,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2495.8,"xrt_compute":14.16,"load_ms":234.7,"mel_ms":0.0,"encode_ms":629.1,"decode_ms":1866.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2522.7,"xrt_wall":14.01,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":374.4,"xrt_compute":29.38,"load_ms":134.6,"mel_ms":0.0,"encode_ms":172.7,"decode_ms":201.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":375.2,"xrt_wall":29.32,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/moonshine-streaming-tiny.json b/catalog/moonshine-streaming-tiny.json new file mode 100644 index 00000000..0c7bfdfa --- /dev/null +++ b/catalog/moonshine-streaming-tiny.json @@ -0,0 +1,55 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-streaming-tiny", + "family": "moonshine_streaming", + "display_name": "moonshine-streaming-tiny", + "params": 44054721, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-streaming-tiny", + "upstream_commit": "f8e9dfd", + "published_repo": "handy-computer/moonshine-streaming-tiny-gguf", + "docs_page": "moonshine-streaming.md", + "languages": ["en"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-streaming-tiny-F32.gguf","size_bytes":177817696}, + {"quant":"F16","filename":"moonshine-streaming-tiny-F16.gguf","size_bytes":89784416}, + {"quant":"Q8_0","filename":"moonshine-streaming-tiny-Q8_0.gguf","size_bytes":50462816} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":18.18,"ci95":[15.79,22.01],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1732,"del":297,"ins":617},"empty_hyp":1,"utts_over_50pct":25,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":4.53,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":4.53,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.52,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":209.6,"xrt_compute":168.59,"load_ms":47.9,"mel_ms":0.0,"encode_ms":89.6,"decode_ms":120.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":210.4,"xrt_wall":167.9,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":40.6,"xrt_compute":270.63,"load_ms":47.8,"mel_ms":0.0,"encode_ms":22.5,"decode_ms":18.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":40.8,"xrt_wall":269.37,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":315.8,"xrt_compute":111.87,"load_ms":51.7,"mel_ms":0.0,"encode_ms":33.4,"decode_ms":282.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":316.8,"xrt_wall":111.51,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":45.9,"xrt_compute":239.48,"load_ms":48.7,"mel_ms":0.0,"encode_ms":8.4,"decode_ms":37.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":46.1,"xrt_wall":238.43,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":948.5,"xrt_compute":37.25,"load_ms":126.2,"mel_ms":0.0,"encode_ms":341.1,"decode_ms":607.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":958.1,"xrt_wall":36.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":146.1,"xrt_compute":75.29,"load_ms":96.1,"mel_ms":0.0,"encode_ms":78.1,"decode_ms":68.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":148.0,"xrt_wall":74.31,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":922.5,"xrt_compute":38.3,"load_ms":106.9,"mel_ms":0.0,"encode_ms":199.7,"decode_ms":722.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":932.7,"xrt_wall":37.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":137.5,"xrt_compute":80.0,"load_ms":80.7,"mel_ms":0.0,"encode_ms":46.1,"decode_ms":91.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":138.0,"xrt_wall":79.72,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/moonshine-tiny-ar.json b/catalog/moonshine-tiny-ar.json new file mode 100644 index 00000000..a111f096 --- /dev/null +++ b/catalog/moonshine-tiny-ar.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-tiny-ar", + "family": "moonshine", + "display_name": "moonshine-tiny-ar", + "params": 27092736, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-tiny-ar", + "upstream_commit": "99e7fee", + "published_repo": "handy-computer/moonshine-tiny-ar-gguf", + "docs_page": "moonshine.md", + "languages": ["ar"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-tiny-ar-F32.gguf","size_bytes":109969088}, + {"quant":"F16","filename":"moonshine-tiny-ar-F16.gguf","size_bytes":59244224}, + {"quant":"Q8_0","filename":"moonshine-tiny-ar-Q8_0.gguf","size_bytes":35466944} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":26.7,"ci95":[24.08,29.47],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1597,"del":339,"ins":263},"empty_hyp":0,"utts_over_50pct":42,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"F32","metric":"wer","err_pct":27.11,"ci95":[24.57,29.98],"n_utts":428,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1603,"del":343,"ins":287},"empty_hyp":0,"utts_over_50pct":44,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"F16","metric":"wer","err_pct":27.11,"ci95":[24.57,29.98],"n_utts":428,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1603,"del":343,"ins":287},"empty_hyp":0,"utts_over_50pct":44,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ar", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":290.4,"xrt_compute":88.63,"wall_ms":290.7,"xrt_wall":88.56,"load_ms":60.6,"mel_ms":0.0,"encode_ms":82.9,"decode_ms":207.6,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":125.5,"xrt_compute":87.67,"wall_ms":125.6,"xrt_wall":87.59,"load_ms":42.2,"mel_ms":0.0,"encode_ms":31.1,"decode_ms":94.4,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":351.7,"xrt_compute":73.19,"wall_ms":351.8,"xrt_wall":73.17,"load_ms":45.9,"mel_ms":0.0,"encode_ms":50.7,"decode_ms":300.9,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":178.6,"xrt_compute":61.58,"wall_ms":178.7,"xrt_wall":61.56,"load_ms":43.8,"mel_ms":0.0,"encode_ms":14.5,"decode_ms":164.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":1392.2,"xrt_compute":18.49,"wall_ms":1395.9,"xrt_wall":18.44,"load_ms":101.8,"mel_ms":0.0,"encode_ms":210.4,"decode_ms":1181.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":524.9,"xrt_compute":20.95,"wall_ms":525.0,"xrt_wall":20.95,"load_ms":86.7,"mel_ms":0.0,"encode_ms":75.0,"decode_ms":450.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ar-long","sample_duration_s":25.74,"total_ms":849.1,"xrt_compute":30.31,"wall_ms":849.4,"xrt_wall":30.3,"load_ms":79.8,"mel_ms":0.0,"encode_ms":110.6,"decode_ms":738.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ar-short","sample_duration_s":11.0,"total_ms":381.4,"xrt_compute":28.84,"wall_ms":381.5,"xrt_wall":28.83,"load_ms":80.3,"mel_ms":0.0,"encode_ms":44.8,"decode_ms":336.6,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-tiny-ja.json b/catalog/moonshine-tiny-ja.json new file mode 100644 index 00000000..b62728a4 --- /dev/null +++ b/catalog/moonshine-tiny-ja.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-tiny-ja", + "family": "moonshine", + "display_name": "moonshine-tiny-ja", + "params": 27092736, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-tiny-ja", + "upstream_commit": "02ca41b", + "published_repo": "handy-computer/moonshine-tiny-ja-gguf", + "docs_page": "moonshine.md", + "languages": ["ja"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-tiny-ja-F32.gguf","size_bytes":109969088}, + {"quant":"F16","filename":"moonshine-tiny-ja-F16.gguf","size_bytes":59244224}, + {"quant":"Q8_0","filename":"moonshine-tiny-ja-Q8_0.gguf","size_bytes":35466944} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":13.44,"ci95":[12.64,14.35],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3039,"del":712,"ins":638},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"F32","metric":"cer","err_pct":13.2,"ci95":[12.42,14.14],"n_utts":650,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3010,"del":699,"ins":601},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"F16","metric":"cer","err_pct":13.2,"ci95":[12.41,14.15],"n_utts":650,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3010,"del":697,"ins":602},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ja", + "metric": "cer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":288.6,"xrt_compute":97.7,"wall_ms":288.9,"xrt_wall":97.62,"load_ms":45.1,"mel_ms":0.0,"encode_ms":90.8,"decode_ms":197.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":83.3,"xrt_compute":131.76,"wall_ms":83.5,"xrt_wall":131.56,"load_ms":43.0,"mel_ms":0.0,"encode_ms":28.4,"decode_ms":55.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":328.3,"xrt_compute":85.89,"wall_ms":328.4,"xrt_wall":85.86,"load_ms":43.8,"mel_ms":0.0,"encode_ms":57.3,"decode_ms":271.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":106.5,"xrt_compute":103.05,"wall_ms":106.6,"xrt_wall":103.0,"load_ms":43.4,"mel_ms":0.0,"encode_ms":14.5,"decode_ms":92.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":1338.6,"xrt_compute":21.07,"wall_ms":1342.1,"xrt_wall":21.01,"load_ms":87.2,"mel_ms":0.0,"encode_ms":213.0,"decode_ms":1125.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":333.1,"xrt_compute":32.96,"wall_ms":333.1,"xrt_wall":32.96,"load_ms":86.4,"mel_ms":0.0,"encode_ms":79.2,"decode_ms":253.9,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ja-long","sample_duration_s":28.2,"total_ms":769.2,"xrt_compute":36.66,"wall_ms":769.6,"xrt_wall":36.64,"load_ms":78.7,"mel_ms":0.0,"encode_ms":115.2,"decode_ms":654.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ja-short","sample_duration_s":10.98,"total_ms":240.3,"xrt_compute":45.69,"wall_ms":240.5,"xrt_wall":45.65,"load_ms":80.7,"mel_ms":0.0,"encode_ms":44.5,"decode_ms":195.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-tiny-ko.json b/catalog/moonshine-tiny-ko.json new file mode 100644 index 00000000..11380a03 --- /dev/null +++ b/catalog/moonshine-tiny-ko.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-tiny-ko", + "family": "moonshine", + "display_name": "moonshine-tiny-ko", + "params": 27092736, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-tiny-ko", + "upstream_commit": "80995f0", + "published_repo": "handy-computer/moonshine-tiny-ko-gguf", + "docs_page": "moonshine.md", + "languages": ["ko"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-tiny-ko-F32.gguf","size_bytes":109969088}, + {"quant":"F16","filename":"moonshine-tiny-ko-F16.gguf","size_bytes":59244224}, + {"quant":"Q8_0","filename":"moonshine-tiny-ko-Q8_0.gguf","size_bytes":35466944} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":9.0,"ci95":[7.86,10.19],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":934,"del":561,"ins":96},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"F32","metric":"cer","err_pct":8.97,"ci95":[7.8,10.21],"n_utts":382,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":930,"del":561,"ins":95},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"F16","metric":"cer","err_pct":8.97,"ci95":[7.8,10.21],"n_utts":382,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":930,"del":561,"ins":95},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "ko", + "metric": "cer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":366.5,"xrt_compute":70.4,"wall_ms":366.7,"xrt_wall":70.36,"load_ms":46.2,"mel_ms":0.0,"encode_ms":81.7,"decode_ms":284.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":101.1,"xrt_compute":108.57,"wall_ms":101.3,"xrt_wall":108.42,"load_ms":41.9,"mel_ms":0.0,"encode_ms":28.6,"decode_ms":72.5,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":445.1,"xrt_compute":57.96,"wall_ms":445.2,"xrt_wall":57.95,"load_ms":45.9,"mel_ms":0.0,"encode_ms":55.9,"decode_ms":389.2,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":139.4,"xrt_compute":78.77,"wall_ms":139.4,"xrt_wall":78.74,"load_ms":43.3,"mel_ms":0.0,"encode_ms":14.4,"decode_ms":125.0,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":1795.8,"xrt_compute":14.37,"wall_ms":1799.4,"xrt_wall":14.34,"load_ms":133.5,"mel_ms":0.0,"encode_ms":198.4,"decode_ms":1597.4,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":412.4,"xrt_compute":26.62,"wall_ms":412.4,"xrt_wall":26.62,"load_ms":86.5,"mel_ms":0.0,"encode_ms":72.4,"decode_ms":340.0,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ko-long","sample_duration_s":25.8,"total_ms":1043.9,"xrt_compute":24.72,"wall_ms":1044.2,"xrt_wall":24.71,"load_ms":80.5,"mel_ms":0.0,"encode_ms":105.7,"decode_ms":938.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"ko-short","sample_duration_s":10.98,"total_ms":305.2,"xrt_compute":35.98,"wall_ms":305.5,"xrt_wall":35.94,"load_ms":79.6,"mel_ms":0.0,"encode_ms":43.7,"decode_ms":261.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-tiny-uk.json b/catalog/moonshine-tiny-uk.json new file mode 100644 index 00000000..5502b5e9 --- /dev/null +++ b/catalog/moonshine-tiny-uk.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-tiny-uk", + "family": "moonshine", + "display_name": "moonshine-tiny-uk", + "params": 27092736, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-tiny-uk", + "upstream_commit": "8cf70f4", + "published_repo": "handy-computer/moonshine-tiny-uk-gguf", + "docs_page": "moonshine.md", + "languages": ["uk"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-tiny-uk-F32.gguf","size_bytes":109969088}, + {"quant":"F16","filename":"moonshine-tiny-uk-F16.gguf","size_bytes":59244224}, + {"quant":"Q8_0","filename":"moonshine-tiny-uk-Q8_0.gguf","size_bytes":35466944} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":18.76,"ci95":[17.7,19.93],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2065,"del":229,"ins":402},"empty_hyp":0,"utts_over_50pct":40,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"F32","metric":"wer","err_pct":18.83,"ci95":[17.77,20.0],"n_utts":750,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2075,"del":225,"ins":406},"empty_hyp":0,"utts_over_50pct":36,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"F16","metric":"wer","err_pct":18.82,"ci95":[17.76,19.99],"n_utts":750,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2074,"del":224,"ins":407},"empty_hyp":0,"utts_over_50pct":36,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "uk", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":325.6,"xrt_compute":88.82,"wall_ms":325.9,"xrt_wall":88.75,"load_ms":45.8,"mel_ms":0.0,"encode_ms":93.4,"decode_ms":232.2,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":69.3,"xrt_compute":158.48,"wall_ms":69.4,"xrt_wall":158.19,"load_ms":43.1,"mel_ms":0.0,"encode_ms":28.1,"decode_ms":41.2,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":362.2,"xrt_compute":79.84,"wall_ms":362.3,"xrt_wall":79.82,"load_ms":42.9,"mel_ms":0.0,"encode_ms":58.9,"decode_ms":303.3,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":88.6,"xrt_compute":123.92,"wall_ms":88.7,"xrt_wall":123.83,"load_ms":43.8,"mel_ms":0.0,"encode_ms":14.5,"decode_ms":74.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":1458.1,"xrt_compute":19.83,"wall_ms":1462.0,"xrt_wall":19.78,"load_ms":99.6,"mel_ms":0.0,"encode_ms":219.0,"decode_ms":1239.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":264.0,"xrt_compute":41.6,"wall_ms":264.0,"xrt_wall":41.59,"load_ms":88.1,"mel_ms":0.0,"encode_ms":71.9,"decode_ms":192.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"uk-long","sample_duration_s":28.92,"total_ms":843.5,"xrt_compute":34.28,"wall_ms":843.9,"xrt_wall":34.27,"load_ms":80.6,"mel_ms":0.0,"encode_ms":117.4,"decode_ms":726.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"uk-short","sample_duration_s":10.98,"total_ms":197.3,"xrt_compute":55.64,"wall_ms":197.5,"xrt_wall":55.58,"load_ms":81.7,"mel_ms":0.0,"encode_ms":43.8,"decode_ms":153.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-tiny-vi.json b/catalog/moonshine-tiny-vi.json new file mode 100644 index 00000000..67226de9 --- /dev/null +++ b/catalog/moonshine-tiny-vi.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-tiny-vi", + "family": "moonshine", + "display_name": "moonshine-tiny-vi", + "params": 27092736, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-tiny-vi", + "upstream_commit": "d4d20da", + "published_repo": "handy-computer/moonshine-tiny-vi-gguf", + "docs_page": "moonshine.md", + "languages": ["vi"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-tiny-vi-F32.gguf","size_bytes":109969088}, + {"quant":"F16","filename":"moonshine-tiny-vi-F16.gguf","size_bytes":59244224}, + {"quant":"Q8_0","filename":"moonshine-tiny-vi-Q8_0.gguf","size_bytes":35466944} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":13.37,"ci95":[12.57,14.23],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2801,"del":279,"ins":400},"empty_hyp":1,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"F32","metric":"wer","err_pct":13.16,"ci95":[12.42,13.96],"n_utts":857,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2793,"del":227,"ins":404},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"F16","metric":"wer","err_pct":13.17,"ci95":[12.44,13.97],"n_utts":857,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2793,"del":227,"ins":407},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "vi", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":254.5,"xrt_compute":98.56,"wall_ms":254.7,"xrt_wall":98.47,"load_ms":42.1,"mel_ms":0.0,"encode_ms":79.6,"decode_ms":174.9,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":84.7,"xrt_compute":129.65,"wall_ms":84.8,"xrt_wall":129.47,"load_ms":43.8,"mel_ms":0.0,"encode_ms":28.2,"decode_ms":56.5,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":302.7,"xrt_compute":82.85,"wall_ms":302.8,"xrt_wall":82.82,"load_ms":43.2,"mel_ms":0.0,"encode_ms":46.7,"decode_ms":256.1,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":119.1,"xrt_compute":92.21,"wall_ms":119.1,"xrt_wall":92.16,"load_ms":42.5,"mel_ms":0.0,"encode_ms":16.7,"decode_ms":102.4,"engine_sha":"522ccd6","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":1149.2,"xrt_compute":21.82,"wall_ms":1152.9,"xrt_wall":21.75,"load_ms":102.4,"mel_ms":0.0,"encode_ms":186.0,"decode_ms":963.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":342.1,"xrt_compute":32.09,"wall_ms":342.2,"xrt_wall":32.09,"load_ms":87.5,"mel_ms":0.0,"encode_ms":71.4,"decode_ms":270.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"vi-long","sample_duration_s":25.08,"total_ms":699.3,"xrt_compute":35.86,"wall_ms":699.7,"xrt_wall":35.84,"load_ms":95.5,"mel_ms":0.0,"encode_ms":101.1,"decode_ms":598.2,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"vi-short","sample_duration_s":10.98,"total_ms":254.0,"xrt_compute":43.22,"wall_ms":254.3,"xrt_wall":43.19,"load_ms":82.5,"mel_ms":0.0,"encode_ms":44.2,"decode_ms":209.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-tiny-zh.json b/catalog/moonshine-tiny-zh.json new file mode 100644 index 00000000..5a68e66d --- /dev/null +++ b/catalog/moonshine-tiny-zh.json @@ -0,0 +1,54 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-tiny-zh", + "family": "moonshine", + "display_name": "moonshine-tiny-zh", + "params": 27092736, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-tiny-zh", + "upstream_commit": "3306270", + "published_repo": "handy-computer/moonshine-tiny-zh-gguf", + "docs_page": "moonshine.md", + "languages": ["zh"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-tiny-zh-F32.gguf","size_bytes":109969088}, + {"quant":"F16","filename":"moonshine-tiny-zh-F16.gguf","size_bytes":59244224}, + {"quant":"Q8_0","filename":"moonshine-tiny-zh-Q8_0.gguf","size_bytes":35466944} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":13.88,"ci95":[12.83,14.93],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2880,"del":1579,"ins":489},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"F32","metric":"cer","err_pct":13.7,"ci95":[12.7,14.68],"n_utts":945,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2843,"del":1574,"ins":467},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"F16","metric":"cer","err_pct":13.7,"ci95":[12.7,14.68],"n_utts":945,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2843,"del":1574,"ins":467},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "zh", + "metric": "cer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":293.3,"xrt_compute":106.09,"wall_ms":293.6,"xrt_wall":105.98,"load_ms":45.1,"mel_ms":0.0,"encode_ms":103.0,"decode_ms":190.3,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":85.3,"xrt_compute":128.93,"wall_ms":85.4,"xrt_wall":128.76,"load_ms":42.7,"mel_ms":0.0,"encode_ms":31.2,"decode_ms":54.1,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":329.6,"xrt_compute":94.42,"wall_ms":329.7,"xrt_wall":94.39,"load_ms":44.0,"mel_ms":0.0,"encode_ms":64.7,"decode_ms":264.8,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":109.4,"xrt_compute":100.53,"wall_ms":109.5,"xrt_wall":100.47,"load_ms":46.6,"mel_ms":0.0,"encode_ms":14.6,"decode_ms":94.9,"engine_sha":"94f1f45","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":1342.2,"xrt_compute":23.19,"wall_ms":1346.1,"xrt_wall":23.12,"load_ms":117.9,"mel_ms":0.0,"encode_ms":261.7,"decode_ms":1080.5,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":326.9,"xrt_compute":33.65,"wall_ms":326.9,"xrt_wall":33.65,"load_ms":88.8,"mel_ms":0.0,"encode_ms":74.8,"decode_ms":252.1,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"zh-long","sample_duration_s":31.12,"total_ms":749.5,"xrt_compute":41.52,"wall_ms":749.8,"xrt_wall":41.5,"load_ms":84.4,"mel_ms":0.0,"encode_ms":128.7,"decode_ms":620.7,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"zh-short","sample_duration_s":11.0,"total_ms":248.1,"xrt_compute":44.34,"wall_ms":248.3,"xrt_wall":44.31,"load_ms":81.8,"mel_ms":0.0,"encode_ms":44.3,"decode_ms":203.8,"engine_sha":"522ccd68","publication_profile":"asr-publication-v2","measured_on":"2026-09-15","thermal_gated":null} + ] +} diff --git a/catalog/moonshine-tiny.json b/catalog/moonshine-tiny.json new file mode 100644 index 00000000..8e89b1a6 --- /dev/null +++ b/catalog/moonshine-tiny.json @@ -0,0 +1,55 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moonshine-tiny", + "family": "moonshine", + "display_name": "moonshine-tiny", + "params": 27092736, + "license": { + "spdx": "mit", + "display": "MIT" + }, + "upstream_repo": "UsefulSensors/moonshine-tiny", + "upstream_commit": "390624e", + "published_repo": "handy-computer/moonshine-tiny-gguf", + "docs_page": "moonshine.md", + "languages": ["en"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"moonshine-tiny-F32.gguf","size_bytes":109969056}, + {"quant":"F16","filename":"moonshine-tiny-F16.gguf","size_bytes":59244192}, + {"quant":"Q8_0","filename":"moonshine-tiny-Q8_0.gguf","size_bytes":35466912} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":14.13,"ci95":[12.44,16.27],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1372,"del":184,"ins":501},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":4.58,"ci95":[4.3,4.85],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1858,"del":248,"ins":321},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":4.58,"ci95":[4.3,4.85],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1858,"del":248,"ins":321},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.6,"ci95":[4.32,4.86],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1863,"del":250,"ins":324},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":372.8,"xrt_compute":94.78,"load_ms":48.4,"mel_ms":0.0,"encode_ms":115.5,"decode_ms":257.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":373.2,"xrt_wall":94.68,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":54.5,"xrt_compute":201.68,"load_ms":45.1,"mel_ms":0.0,"encode_ms":31.5,"decode_ms":23.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":54.7,"xrt_wall":201.12,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":421.6,"xrt_compute":83.81,"load_ms":56.7,"mel_ms":0.0,"encode_ms":90.2,"decode_ms":331.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":421.7,"xrt_wall":83.78,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":55.8,"xrt_compute":197.19,"load_ms":48.6,"mel_ms":0.0,"encode_ms":16.0,"decode_ms":39.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":55.9,"xrt_wall":196.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1792.2,"xrt_compute":19.71,"load_ms":86.3,"mel_ms":0.0,"encode_ms":292.9,"decode_ms":1499.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1797.3,"xrt_wall":19.66,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":174.8,"xrt_compute":62.93,"load_ms":110.0,"mel_ms":0.0,"encode_ms":73.7,"decode_ms":101.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":174.8,"xrt_wall":62.92,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":938.4,"xrt_compute":37.65,"load_ms":79.7,"mel_ms":0.0,"encode_ms":147.9,"decode_ms":790.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":938.8,"xrt_wall":37.64,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":132.2,"xrt_compute":83.21,"load_ms":138.0,"mel_ms":0.0,"encode_ms":46.3,"decode_ms":85.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":132.4,"xrt_wall":83.11,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/moss-transcribe-diarize.json b/catalog/moss-transcribe-diarize.json new file mode 100644 index 00000000..48fd9641 --- /dev/null +++ b/catalog/moss-transcribe-diarize.json @@ -0,0 +1,70 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "moss-transcribe-diarize", + "family": "moss", + "display_name": "moss-transcribe-diarize", + "params": 908529760, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "OpenMOSS-Team/MOSS-Transcribe-Diarize", + "upstream_commit": "d7231bb", + "published_repo": "handy-computer/MOSS-Transcribe-Diarize-gguf", + "docs_page": "moss-transcribe-diarize.md", + "languages": ["en","zh"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":true,"verified":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"MOSS-Transcribe-Diarize-BF16.gguf","size_bytes":1826882720}, + {"quant":"F16","filename":"MOSS-Transcribe-Diarize-F16.gguf","size_bytes":1833665696}, + {"quant":"Q8_0","filename":"MOSS-Transcribe-Diarize-Q8_0.gguf","size_bytes":986899616}, + {"quant":"Q6_K","filename":"MOSS-Transcribe-Diarize-Q6_K.gguf","size_bytes":768151712}, + {"quant":"Q5_K_M","filename":"MOSS-Transcribe-Diarize-Q5_K_M.gguf","size_bytes":700313760}, + {"quant":"Q4_K_M","filename":"MOSS-Transcribe-Diarize-Q4_K_M.gguf","size_bytes":617345184} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.13,"ci95":[4.57,5.78],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":453,"del":103,"ins":191},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":9.23,"ci95":[8.2,10.17],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1404,"del":1504,"ins":382},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":2.08,"ci95":[1.83,2.4],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":779,"del":204,"ins":118},"empty_hyp":3,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.07,"ci95":[1.82,2.4],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":778,"del":204,"ins":118},"empty_hyp":3,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.93,"ci95":[1.77,2.1],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":770,"del":137,"ins":116},"empty_hyp":2,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.96,"ci95":[1.8,2.12],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":775,"del":142,"ins":121},"empty_hyp":2,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.99,"ci95":[1.82,2.16],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":794,"del":139,"ins":120},"empty_hyp":2,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.59,"ci95":[2.15,3.18],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":837,"del":325,"ins":210},"empty_hyp":6,"utts_over_50pct":13,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":5467.1,"xrt_compute":6.46,"load_ms":215.8,"mel_ms":8.4,"encode_ms":3088.1,"decode_ms":2370.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5468.9,"xrt_wall":6.46,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2201.5,"xrt_compute":5.0,"load_ms":192.3,"mel_ms":4.1,"encode_ms":1577.3,"decode_ms":620.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2202.7,"xrt_wall":4.99,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":5432.4,"xrt_compute":6.5,"load_ms":293.4,"mel_ms":8.5,"encode_ms":2867.5,"decode_ms":2556.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5434.2,"xrt_wall":6.5,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2084.6,"xrt_compute":5.28,"load_ms":271.5,"mel_ms":4.0,"encode_ms":1432.8,"decode_ms":647.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2085.7,"xrt_wall":5.27,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1217.8,"xrt_compute":29.01,"load_ms":219.0,"mel_ms":10.8,"encode_ms":252.1,"decode_ms":954.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1222.1,"xrt_wall":28.91,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":381.7,"xrt_compute":28.82,"load_ms":200.6,"mel_ms":6.0,"encode_ms":127.9,"decode_ms":247.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":384.4,"xrt_wall":28.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1403.8,"xrt_compute":25.17,"load_ms":310.3,"mel_ms":15.7,"encode_ms":249.8,"decode_ms":1138.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1409.0,"xrt_wall":25.08,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":393.2,"xrt_compute":27.98,"load_ms":278.4,"mel_ms":5.5,"encode_ms":123.4,"decode_ms":264.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":395.8,"xrt_wall":27.79,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":19242.5,"xrt_compute":1.84,"load_ms":619.1,"mel_ms":63.8,"encode_ms":9246.0,"decode_ms":9932.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":19272.7,"xrt_wall":1.83,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":6899.5,"xrt_compute":1.59,"load_ms":904.8,"mel_ms":29.5,"encode_ms":4614.5,"decode_ms":2255.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6911.6,"xrt_wall":1.59,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":21084.5,"xrt_compute":1.68,"load_ms":915.3,"mel_ms":56.4,"encode_ms":9587.5,"decode_ms":11440.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":21115.8,"xrt_wall":1.67,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":7542.8,"xrt_compute":1.46,"load_ms":1409.4,"mel_ms":34.1,"encode_ms":4835.1,"decode_ms":2673.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7554.7,"xrt_wall":1.46,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":9948.6,"xrt_compute":3.55,"load_ms":768.0,"mel_ms":68.9,"encode_ms":4300.4,"decode_ms":5579.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":10022.6,"xrt_wall":3.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3483.8,"xrt_compute":3.16,"load_ms":1228.4,"mel_ms":37.4,"encode_ms":2040.4,"decode_ms":1406.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3518.3,"xrt_wall":3.13,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":11093.6,"xrt_compute":3.18,"load_ms":1323.6,"mel_ms":72.3,"encode_ms":4357.6,"decode_ms":6663.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":11167.7,"xrt_wall":3.16,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3733.6,"xrt_compute":2.95,"load_ms":2142.8,"mel_ms":37.9,"encode_ms":2005.4,"decode_ms":1690.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3768.5,"xrt_wall":2.92,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/multitalker-parakeet-streaming-0.6b-v1.json b/catalog/multitalker-parakeet-streaming-0.6b-v1.json new file mode 100644 index 00000000..34f145f6 --- /dev/null +++ b/catalog/multitalker-parakeet-streaming-0.6b-v1.json @@ -0,0 +1,73 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "multitalker-parakeet-streaming-0.6b-v1", + "family": "parakeet", + "display_name": "multitalker-parakeet-streaming-0.6b-v1", + "params": 739989513, + "license": { + "spdx": "other", + "display": "NVIDIA Open Model License", + "name": "nvidia-open-model-license", + "link": "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/" + }, + "upstream_repo": "nvidia/multitalker-parakeet-streaming-0.6b-v1", + "upstream_commit": "8749fc7", + "published_repo": "handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf", + "docs_page": "multitalker-parakeet-streaming-0.6b-v1.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":true,"verified":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"bundle/multitalker-parakeet-streaming-0.6b-v1-F32.gguf","size_bytes":2960100448}, + {"quant":"F16","filename":"bundle/multitalker-parakeet-streaming-0.6b-v1-F16.gguf","size_bytes":1482674272}, + {"quant":"Q8_0","filename":"bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf","size_bytes":873443456}, + {"quant":"Q6_K","filename":"bundle/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf","size_bytes":743197824}, + {"quant":"Q5_K_M","filename":"bundle/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf","size_bytes":681209984}, + {"quant":"Q4_K_M","filename":"bundle/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf","size_bytes":617132160} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":6.52,"ci95":[5.95,7.12],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":672,"del":172,"ins":105},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.19,"ci95":[2.03,2.37],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":874,"del":174,"ins":115},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.19,"ci95":[2.03,2.37],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":871,"del":176,"ins":115},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.18,"ci95":[2.02,2.36],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":869,"del":173,"ins":114},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.2,"ci95":[2.04,2.38],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":875,"del":174,"ins":119},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.18,"ci95":[2.01,2.36],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":869,"del":169,"ins":116},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.18,"ci95":[2.02,2.36],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":864,"del":176,"ins":117},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"ami","split":"ihm-test","language":"en","backend":"cuda","quant":"F32","metric":"cpwer","err_pct":19.35,"ci95":[null,null],"n_utts":16,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","mode":"kernel"}, + {"dataset":"ami","split":"ihm-test","language":"en","backend":"cuda","quant":"F32","metric":"cpwer","err_pct":23.73,"ci95":[null,null],"n_utts":16,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published","mode":"masked"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1108.8,"xrt_compute":31.87,"load_ms":211.4,"mel_ms":1.6,"encode_ms":1071.0,"decode_ms":36.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1113.3,"xrt_wall":31.74,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":337.6,"xrt_compute":32.58,"load_ms":193.2,"mel_ms":0.6,"encode_ms":328.8,"decode_ms":8.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":339.4,"xrt_wall":32.41,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1098.1,"xrt_compute":32.18,"load_ms":202.1,"mel_ms":1.6,"encode_ms":1059.8,"decode_ms":36.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1102.6,"xrt_wall":32.04,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":330.8,"xrt_compute":33.25,"load_ms":204.7,"mel_ms":0.7,"encode_ms":321.7,"decode_ms":8.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":332.7,"xrt_wall":33.07,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":156.0,"xrt_compute":226.47,"load_ms":164.5,"mel_ms":1.7,"encode_ms":117.6,"decode_ms":36.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":166.5,"xrt_wall":212.19,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":54.5,"xrt_compute":201.85,"load_ms":170.6,"mel_ms":0.6,"encode_ms":45.7,"decode_ms":8.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":58.9,"xrt_wall":186.9,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":155.1,"xrt_compute":227.85,"load_ms":178.8,"mel_ms":1.6,"encode_ms":116.1,"decode_ms":37.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":165.8,"xrt_wall":213.05,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":53.1,"xrt_compute":207.08,"load_ms":177.9,"mel_ms":0.6,"encode_ms":44.2,"decode_ms":8.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":57.7,"xrt_wall":190.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2956.1,"xrt_compute":11.95,"load_ms":986.2,"mel_ms":55.9,"encode_ms":2653.2,"decode_ms":247.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3007.2,"xrt_wall":11.75,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":790.3,"xrt_compute":13.92,"load_ms":708.6,"mel_ms":17.5,"encode_ms":728.0,"decode_ms":44.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":806.0,"xrt_wall":13.65,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2911.6,"xrt_compute":12.13,"load_ms":1261.3,"mel_ms":57.2,"encode_ms":2609.5,"decode_ms":244.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2962.0,"xrt_wall":11.93,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":748.3,"xrt_compute":14.7,"load_ms":879.9,"mel_ms":19.5,"encode_ms":683.4,"decode_ms":45.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":763.9,"xrt_wall":14.4,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1303.0,"xrt_compute":27.12,"load_ms":341.6,"mel_ms":57.2,"encode_ms":1007.4,"decode_ms":238.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1322.3,"xrt_wall":26.72,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":438.1,"xrt_compute":25.11,"load_ms":326.5,"mel_ms":17.6,"encode_ms":374.7,"decode_ms":45.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":447.6,"xrt_wall":24.57,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1279.5,"xrt_compute":27.61,"load_ms":421.5,"mel_ms":57.4,"encode_ms":985.7,"decode_ms":236.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1298.5,"xrt_wall":27.21,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":438.6,"xrt_compute":25.08,"load_ms":415.0,"mel_ms":17.8,"encode_ms":375.3,"decode_ms":45.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":448.1,"xrt_wall":24.55,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/nemotron-3.5-asr-streaming-0.6b.json b/catalog/nemotron-3.5-asr-streaming-0.6b.json new file mode 100644 index 00000000..6bb11a55 --- /dev/null +++ b/catalog/nemotron-3.5-asr-streaming-0.6b.json @@ -0,0 +1,108 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "nemotron-3.5-asr-streaming-0.6b", + "family": "parakeet", + "display_name": "nemotron-3.5-asr-streaming-0.6b", + "params": 637991968, + "license": { + "spdx": "other", + "display": "OpenMDW-1.1", + "name": "openmdw-1.1", + "link": "https://openmdw.ai/license/1-1/" + }, + "upstream_repo": "nvidia/nemotron-3.5-asr-streaming-0.6b", + "upstream_commit": "24b151a", + "published_repo": "handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf", + "docs_page": "nemotron-3.5-asr-streaming-0.6b.md", + "languages": [ + "en-US", "en-GB", "es-US", "es-ES", "fr-FR", "fr-CA", "it-IT", "pt-BR", + "pt-PT", "nl-NL", "de-DE", "tr-TR", "ru-RU", "ar-AR", "hi-IN", "ja-JP", + "ko-KR", "vi-VN", "uk-UA", "pl-PL", "sv-SE", "cs-CZ", "nb-NO", "da-DK", + "bg-BG", "fi-FI", "hr-HR", "sk-SK", "zh-CN", "hu-HU", "ro-RO", "et-EE" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"nemotron-3.5-asr-streaming-0.6b-F32.gguf","size_bytes":2552277984}, + {"quant":"F16","filename":"nemotron-3.5-asr-streaming-0.6b-F16.gguf","size_bytes":1277750240}, + {"quant":"Q8_0","filename":"nemotron-3.5-asr-streaming-0.6b-Q8_0.gguf","size_bytes":751094240}, + {"quant":"Q6_K","filename":"nemotron-3.5-asr-streaming-0.6b-Q6_K.gguf","size_bytes":621356512}, + {"quant":"Q5_K_M","filename":"nemotron-3.5-asr-streaming-0.6b-Q5_K_M.gguf","size_bytes":559647200}, + {"quant":"Q4_K_M","filename":"nemotron-3.5-asr-streaming-0.6b-Q4_K_M.gguf","size_bytes":495831520} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":15.93,"ci95":[14.47,17.54],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":869,"del":322,"ins":121},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":22.02,"ci95":[21.07,23.05],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2369,"del":551,"ins":201},"empty_hyp":0,"utts_over_50pct":27,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":23.0,"ci95":[21.89,24.02],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2543,"del":283,"ins":324},"empty_hyp":0,"utts_over_50pct":37,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":28.51,"ci95":[27.49,29.47],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4354,"del":786,"ins":606},"empty_hyp":1,"utts_over_50pct":88,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":10.33,"ci95":[9.58,11.13],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1373,"del":254,"ins":306},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":7.9,"ci95":[7.23,8.54],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":811,"del":232,"ins":107},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":6.3,"ci95":[5.75,6.89],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":970,"del":154,"ins":344},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":31.84,"ci95":[30.63,32.85],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3573,"del":513,"ins":652},"empty_hyp":0,"utts_over_50pct":135,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":21.91,"ci95":[20.93,22.94],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2534,"del":394,"ins":314},"empty_hyp":0,"utts_over_50pct":50,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":10.78,"ci95":[10.02,11.57],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1386,"del":201,"ins":352},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":8.61,"ci95":[7.67,9.7],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":810,"del":436,"ins":368},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":26.21,"ci95":[25.13,27.33],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3600,"del":567,"ins":486},"empty_hyp":0,"utts_over_50pct":73,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":32.12,"ci95":[30.89,33.35],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4353,"del":576,"ins":555},"empty_hyp":0,"utts_over_50pct":135,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":5.78,"ci95":[5.24,6.28],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":868,"del":147,"ins":213},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":13.52,"ci95":[12.78,14.27],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3193,"del":816,"ins":404},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":8.89,"ci95":[7.78,10.06],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":888,"del":582,"ins":101},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nb","quant":"Q8_0","metric":"wer","err_pct":19.24,"ci95":[17.92,20.58],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1072,"del":234,"ins":160},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":13.61,"ci95":[12.47,14.7],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":859,"del":165,"ins":119},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":17.54,"ci95":[16.58,18.5],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1903,"del":243,"ins":377},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":8.52,"ci95":[7.77,9.34],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1113,"del":145,"ins":572},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":28.28,"ci95":[27.12,29.36],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4345,"del":917,"ins":621},"empty_hyp":0,"utts_over_50pct":100,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":12.61,"ci95":[11.87,13.4],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1480,"del":156,"ins":268},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":23.25,"ci95":[22.21,24.21],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2870,"del":426,"ins":248},"empty_hyp":3,"utts_over_50pct":36,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":24.32,"ci95":[23.32,25.3],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2776,"del":647,"ins":319},"empty_hyp":0,"utts_over_50pct":40,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":15.4,"ci95":[14.46,16.35],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1482,"del":368,"ins":197},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":14.88,"ci95":[14.02,15.72],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1693,"del":259,"ins":186},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":13.96,"ci95":[13.09,14.83],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2616,"del":365,"ins":652},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":18.87,"ci95":[17.82,19.91],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4197,"del":2147,"ins":385},"empty_hyp":0,"utts_over_50pct":28,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":3.04,"ci95":[2.83,3.24],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1258,"del":200,"ins":153},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":3.04,"ci95":[2.83,3.24],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1257,"del":202,"ins":153},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.05,"ci95":[2.85,3.26],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1267,"del":197,"ins":154},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":3.08,"ci95":[2.88,3.28],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1272,"del":204,"ins":157},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":3.1,"ci95":[2.9,3.3],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1280,"del":203,"ins":159},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":3.3,"ci95":[3.1,3.51],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1372,"del":219,"ins":161},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"F32","metric":"wer","err_pct":7.97,"ci95":[7.33,8.7],"n_utts":647,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":796,"del":265,"ins":99},"empty_hyp":1,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"F16","metric":"wer","err_pct":7.97,"ci95":[7.33,8.7],"n_utts":647,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":796,"del":264,"ins":100},"empty_hyp":1,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q6_K","metric":"wer","err_pct":8.02,"ci95":[7.39,8.71],"n_utts":647,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":824,"del":228,"ins":116},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":8.15,"ci95":[7.46,8.88],"n_utts":647,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":805,"del":282,"ins":99},"empty_hyp":1,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":8.49,"ci95":[7.8,9.23],"n_utts":647,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":840,"del":289,"ins":107},"empty_hyp":1,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1208.6,"xrt_compute":29.23,"load_ms":182.5,"mel_ms":1.6,"encode_ms":1078.8,"decode_ms":128.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1213.1,"xrt_wall":29.13,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":354.7,"xrt_compute":31.01,"load_ms":170.4,"mel_ms":0.6,"encode_ms":322.6,"decode_ms":31.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":356.6,"xrt_wall":30.85,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1188.4,"xrt_compute":29.73,"load_ms":213.6,"mel_ms":1.6,"encode_ms":1059.8,"decode_ms":127.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1192.8,"xrt_wall":29.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":357.6,"xrt_compute":30.76,"load_ms":219.4,"mel_ms":0.6,"encode_ms":325.3,"decode_ms":31.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":359.5,"xrt_wall":30.6,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":256.3,"xrt_compute":137.88,"load_ms":143.5,"mel_ms":1.6,"encode_ms":127.3,"decode_ms":127.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":266.8,"xrt_wall":132.43,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":76.8,"xrt_compute":143.14,"load_ms":137.4,"mel_ms":0.6,"encode_ms":45.3,"decode_ms":30.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":81.5,"xrt_wall":135.05,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":255.5,"xrt_compute":138.26,"load_ms":186.4,"mel_ms":1.6,"encode_ms":127.7,"decode_ms":126.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":266.1,"xrt_wall":132.77,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":76.4,"xrt_compute":143.94,"load_ms":185.7,"mel_ms":0.6,"encode_ms":44.7,"decode_ms":31.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":80.6,"xrt_wall":136.41,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3739.4,"xrt_compute":9.45,"load_ms":693.7,"mel_ms":57.3,"encode_ms":2664.9,"decode_ms":1017.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3774.5,"xrt_wall":9.36,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":993.3,"xrt_compute":11.07,"load_ms":932.3,"mel_ms":18.2,"encode_ms":727.8,"decode_ms":247.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1004.9,"xrt_wall":10.95,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3671.3,"xrt_compute":9.62,"load_ms":897.7,"mel_ms":58.4,"encode_ms":2604.2,"decode_ms":1008.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3706.0,"xrt_wall":9.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":951.4,"xrt_compute":11.56,"load_ms":1217.2,"mel_ms":18.2,"encode_ms":685.8,"decode_ms":247.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":963.7,"xrt_wall":11.41,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2093.3,"xrt_compute":16.88,"load_ms":420.9,"mel_ms":58.0,"encode_ms":1031.2,"decode_ms":1004.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2109.5,"xrt_wall":16.75,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":643.8,"xrt_compute":17.09,"load_ms":669.2,"mel_ms":17.5,"encode_ms":376.0,"decode_ms":250.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":654.0,"xrt_wall":16.82,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2067.0,"xrt_compute":17.09,"load_ms":522.0,"mel_ms":58.3,"encode_ms":1009.2,"decode_ms":999.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2086.5,"xrt_wall":16.93,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":640.3,"xrt_compute":17.18,"load_ms":884.5,"mel_ms":17.8,"encode_ms":372.2,"decode_ms":250.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":649.2,"xrt_wall":16.94,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/nemotron-speech-streaming-en-0.6b.json b/catalog/nemotron-speech-streaming-en-0.6b.json new file mode 100644 index 00000000..7081e5e2 --- /dev/null +++ b/catalog/nemotron-speech-streaming-en-0.6b.json @@ -0,0 +1,71 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "nemotron-speech-streaming-en-0.6b", + "family": "parakeet", + "display_name": "nemotron-speech-streaming-en-0.6b", + "params": 618079745, + "license": { + "spdx": "other", + "display": "NVIDIA Open Model License", + "name": "nvidia-open-model-license", + "link": "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/" + }, + "upstream_repo": "nvidia/nemotron-speech-streaming-en-0.6b", + "upstream_commit": "ef3bf40", + "published_repo": "handy-computer/nemotron-speech-streaming-en-0.6b-gguf", + "docs_page": "nemotron-speech-streaming-en-0.6b.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"nemotron-speech-streaming-en-0.6b-F32.gguf","size_bytes":2472386176}, + {"quant":"F16","filename":"nemotron-speech-streaming-en-0.6b-F16.gguf","size_bytes":1237652608}, + {"quant":"Q8_0","filename":"nemotron-speech-streaming-en-0.6b-Q8_0.gguf","size_bytes":729650176}, + {"quant":"Q6_K","filename":"nemotron-speech-streaming-en-0.6b-Q6_K.gguf","size_bytes":600420352}, + {"quant":"Q5_K_M","filename":"nemotron-speech-streaming-en-0.6b-Q5_K_M.gguf","size_bytes":538989568}, + {"quant":"Q4_K_M","filename":"nemotron-speech-streaming-en-0.6b-Q4_K_M.gguf","size_bytes":475436032} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":6.43,"ci95":[5.81,7.14],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":624,"del":218,"ins":94},"empty_hyp":2,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.31,"ci95":[2.13,2.49],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":939,"del":159,"ins":128},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.31,"ci95":[2.13,2.49],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":939,"del":159,"ins":128},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.31,"ci95":[2.12,2.5],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":935,"del":161,"ins":127},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.29,"ci95":[2.1,2.46],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":935,"del":157,"ins":120},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.34,"ci95":[2.15,2.53],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":952,"del":157,"ins":130},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.38,"ci95":[2.2,2.56],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":960,"del":168,"ins":135},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1108.0,"xrt_compute":31.89,"load_ms":168.9,"mel_ms":1.6,"encode_ms":1069.5,"decode_ms":36.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1112.2,"xrt_wall":31.77,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":335.3,"xrt_compute":32.81,"load_ms":159.5,"mel_ms":0.6,"encode_ms":326.2,"decode_ms":8.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":337.0,"xrt_wall":32.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1103.2,"xrt_compute":32.03,"load_ms":222.4,"mel_ms":1.6,"encode_ms":1064.8,"decode_ms":36.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1107.4,"xrt_wall":31.91,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":327.0,"xrt_compute":33.64,"load_ms":206.5,"mel_ms":0.7,"encode_ms":318.2,"decode_ms":8.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":328.8,"xrt_wall":33.45,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":158.2,"xrt_compute":223.33,"load_ms":147.5,"mel_ms":1.6,"encode_ms":118.6,"decode_ms":38.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":168.5,"xrt_wall":209.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":54.8,"xrt_compute":200.58,"load_ms":131.6,"mel_ms":0.6,"encode_ms":46.0,"decode_ms":8.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":59.3,"xrt_wall":185.48,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":156.9,"xrt_compute":225.2,"load_ms":212.2,"mel_ms":1.6,"encode_ms":117.5,"decode_ms":37.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":167.3,"xrt_wall":211.22,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":53.2,"xrt_compute":206.74,"load_ms":182.5,"mel_ms":0.6,"encode_ms":44.4,"decode_ms":8.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":57.7,"xrt_wall":190.66,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2928.5,"xrt_compute":12.06,"load_ms":622.1,"mel_ms":56.6,"encode_ms":2624.7,"decode_ms":247.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2961.7,"xrt_wall":11.93,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":788.2,"xrt_compute":13.96,"load_ms":862.9,"mel_ms":17.7,"encode_ms":724.8,"decode_ms":45.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":800.5,"xrt_wall":13.74,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2864.8,"xrt_compute":12.33,"load_ms":838.1,"mel_ms":55.6,"encode_ms":2564.3,"decode_ms":244.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2897.5,"xrt_wall":12.19,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":743.0,"xrt_compute":14.81,"load_ms":1138.7,"mel_ms":18.2,"encode_ms":680.4,"decode_ms":44.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":755.8,"xrt_wall":14.55,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1261.2,"xrt_compute":28.01,"load_ms":289.0,"mel_ms":57.5,"encode_ms":967.2,"decode_ms":236.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1277.3,"xrt_wall":27.66,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":425.8,"xrt_compute":25.83,"load_ms":551.7,"mel_ms":17.8,"encode_ms":362.3,"decode_ms":45.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":435.5,"xrt_wall":25.26,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1247.5,"xrt_compute":28.32,"load_ms":381.4,"mel_ms":56.9,"encode_ms":954.3,"decode_ms":236.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1267.0,"xrt_wall":27.89,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":423.5,"xrt_compute":25.97,"load_ms":728.8,"mel_ms":17.6,"encode_ms":361.0,"decode_ms":45.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":432.8,"xrt_wall":25.41,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-ctc-0.6b.json b/catalog/parakeet-ctc-0.6b.json new file mode 100644 index 00000000..84c5448e --- /dev/null +++ b/catalog/parakeet-ctc-0.6b.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-ctc-0.6b", + "family": "parakeet", + "display_name": "parakeet-ctc-0.6b", + "params": 608848897, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-ctc-0.6b", + "upstream_commit": "ad09ba1", + "published_repo": "handy-computer/parakeet-ctc-0.6b-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-ctc-0.6b-F32.gguf","size_bytes":2435482816}, + {"quant":"F16","filename":"parakeet-ctc-0.6b-F16.gguf","size_bytes":1220181184}, + {"quant":"Q8_0","filename":"parakeet-ctc-0.6b-Q8_0.gguf","size_bytes":722271424}, + {"quant":"Q6_K","filename":"parakeet-ctc-0.6b-Q6_K.gguf","size_bytes":593644736}, + {"quant":"Q5_K_M","filename":"parakeet-ctc-0.6b-Q5_K_M.gguf","size_bytes":532544704}, + {"quant":"Q4_K_M","filename":"parakeet-ctc-0.6b-Q4_K_M.gguf","size_bytes":469302464} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.53,"ci95":[5.06,6.07],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":578,"del":121,"ins":106},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.87,"ci95":[1.71,2.03],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":770,"del":112,"ins":109},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.87,"ci95":[1.71,2.03],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":769,"del":112,"ins":108},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.87,"ci95":[1.71,2.03],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":769,"del":111,"ins":110},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.84,"ci95":[1.69,2.0],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":767,"del":107,"ins":104},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.87,"ci95":[1.71,2.04],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":768,"del":111,"ins":115},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.9,"ci95":[1.73,2.06],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":781,"del":109,"ins":115},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":986.8,"xrt_compute":35.8,"load_ms":170.1,"mel_ms":1.7,"encode_ms":982.9,"decode_ms":2.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":989.8,"xrt_wall":35.7,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":293.5,"xrt_compute":37.48,"load_ms":154.1,"mel_ms":0.7,"encode_ms":292.0,"decode_ms":0.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":295.1,"xrt_wall":37.28,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":921.4,"xrt_compute":38.35,"load_ms":201.5,"mel_ms":1.8,"encode_ms":917.4,"decode_ms":2.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":924.5,"xrt_wall":38.22,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":270.5,"xrt_compute":40.66,"load_ms":200.2,"mel_ms":0.7,"encode_ms":269.1,"decode_ms":0.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":272.1,"xrt_wall":40.43,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":113.4,"xrt_compute":311.61,"load_ms":126.5,"mel_ms":1.8,"encode_ms":109.4,"decode_ms":2.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":119.4,"xrt_wall":296.0,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":48.5,"xrt_compute":226.86,"load_ms":124.1,"mel_ms":0.7,"encode_ms":46.9,"decode_ms":0.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":52.0,"xrt_wall":211.37,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":111.4,"xrt_compute":317.21,"load_ms":171.3,"mel_ms":1.8,"encode_ms":107.4,"decode_ms":2.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":117.3,"xrt_wall":301.19,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":48.3,"xrt_compute":227.64,"load_ms":172.7,"mel_ms":0.7,"encode_ms":46.8,"decode_ms":0.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":51.7,"xrt_wall":212.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2644.1,"xrt_compute":13.36,"load_ms":821.1,"mel_ms":40.9,"encode_ms":2536.9,"decode_ms":66.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2663.9,"xrt_wall":13.26,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":723.3,"xrt_compute":15.21,"load_ms":607.6,"mel_ms":12.6,"encode_ms":691.2,"decode_ms":19.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":727.9,"xrt_wall":15.11,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2571.5,"xrt_compute":13.74,"load_ms":1129.3,"mel_ms":38.9,"encode_ms":2465.9,"decode_ms":66.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2591.5,"xrt_wall":13.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":663.5,"xrt_compute":16.58,"load_ms":783.6,"mel_ms":13.1,"encode_ms":629.1,"decode_ms":21.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":668.1,"xrt_wall":16.47,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1155.6,"xrt_compute":30.57,"load_ms":439.3,"mel_ms":40.4,"encode_ms":1054.0,"decode_ms":61.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1172.8,"xrt_wall":30.13,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":426.8,"xrt_compute":25.77,"load_ms":202.5,"mel_ms":13.6,"encode_ms":385.3,"decode_ms":27.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":438.6,"xrt_wall":25.08,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1142.9,"xrt_compute":30.91,"load_ms":642.1,"mel_ms":40.8,"encode_ms":1039.5,"decode_ms":62.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1162.9,"xrt_wall":30.38,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":418.4,"xrt_compute":26.29,"load_ms":295.7,"mel_ms":14.0,"encode_ms":378.2,"decode_ms":26.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":430.0,"xrt_wall":25.58,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-ctc-1.1b.json b/catalog/parakeet-ctc-1.1b.json new file mode 100644 index 00000000..69846813 --- /dev/null +++ b/catalog/parakeet-ctc-1.1b.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-ctc-1.1b", + "family": "parakeet", + "display_name": "parakeet-ctc-1.1b", + "params": 1062626305, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-ctc-1.1b", + "upstream_commit": "a707e81", + "published_repo": "handy-computer/parakeet-ctc-1.1b-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-ctc-1.1b-F32.gguf","size_bytes":4250639392}, + {"quant":"F16","filename":"parakeet-ctc-1.1b-F16.gguf","size_bytes":2129368096}, + {"quant":"Q8_0","filename":"parakeet-ctc-1.1b-Q8_0.gguf","size_bytes":1259869216}, + {"quant":"Q6_K","filename":"parakeet-ctc-1.1b-Q6_K.gguf","size_bytes":1035248672}, + {"quant":"Q5_K_M","filename":"parakeet-ctc-1.1b-Q5_K_M.gguf","size_bytes":928584736}, + {"quant":"Q4_K_M","filename":"parakeet-ctc-1.1b-Q4_K_M.gguf","size_bytes":818156576} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.61,"ci95":[5.04,6.21],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":567,"del":106,"ins":143},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.85,"ci95":[1.68,2.01],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":766,"del":98,"ins":116},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.85,"ci95":[1.68,2.02],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":768,"del":98,"ins":116},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.85,"ci95":[1.68,2.02],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":769,"del":98,"ins":115},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.85,"ci95":[1.69,2.02],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":762,"del":101,"ins":119},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.84,"ci95":[1.68,2.01],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":763,"del":97,"ins":118},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.9,"ci95":[1.74,2.07],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":788,"del":105,"ins":116},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1698.3,"xrt_compute":20.8,"load_ms":276.7,"mel_ms":1.8,"encode_ms":1694.3,"decode_ms":2.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1701.9,"xrt_wall":20.76,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":501.8,"xrt_compute":21.92,"load_ms":256.3,"mel_ms":0.7,"encode_ms":500.4,"decode_ms":0.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":504.1,"xrt_wall":21.82,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1583.3,"xrt_compute":22.32,"load_ms":334.7,"mel_ms":1.7,"encode_ms":1579.2,"decode_ms":2.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1587.4,"xrt_wall":22.26,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":458.4,"xrt_compute":24.0,"load_ms":341.0,"mel_ms":0.7,"encode_ms":456.9,"decode_ms":0.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":460.9,"xrt_wall":23.87,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":180.3,"xrt_compute":195.92,"load_ms":201.9,"mel_ms":1.8,"encode_ms":176.2,"decode_ms":2.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":188.1,"xrt_wall":187.83,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":80.4,"xrt_compute":136.77,"load_ms":203.6,"mel_ms":0.8,"encode_ms":78.9,"decode_ms":0.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":85.7,"xrt_wall":128.34,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":176.6,"xrt_compute":200.12,"load_ms":284.5,"mel_ms":1.8,"encode_ms":172.5,"decode_ms":2.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":184.5,"xrt_wall":191.54,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":78.6,"xrt_compute":140.04,"load_ms":288.0,"mel_ms":0.8,"encode_ms":77.0,"decode_ms":0.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":83.7,"xrt_wall":131.38,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4627.7,"xrt_compute":7.63,"load_ms":1364.2,"mel_ms":38.2,"encode_ms":4520.6,"decode_ms":68.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4650.1,"xrt_wall":7.6,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1213.9,"xrt_compute":9.06,"load_ms":1002.2,"mel_ms":14.4,"encode_ms":1179.8,"decode_ms":19.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1221.3,"xrt_wall":9.01,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4484.5,"xrt_compute":7.88,"load_ms":1842.1,"mel_ms":41.8,"encode_ms":4372.0,"decode_ms":70.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4507.3,"xrt_wall":7.84,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1103.5,"xrt_compute":9.97,"load_ms":1303.9,"mel_ms":13.8,"encode_ms":1068.9,"decode_ms":20.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1110.5,"xrt_wall":9.91,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2019.8,"xrt_compute":17.49,"load_ms":692.9,"mel_ms":41.7,"encode_ms":1914.5,"decode_ms":63.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2065.6,"xrt_wall":17.11,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":711.0,"xrt_compute":15.47,"load_ms":300.4,"mel_ms":13.7,"encode_ms":670.4,"decode_ms":26.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":736.1,"xrt_wall":14.94,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2002.0,"xrt_compute":17.65,"load_ms":1049.1,"mel_ms":41.3,"encode_ms":1897.2,"decode_ms":63.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2047.6,"xrt_wall":17.26,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":691.0,"xrt_compute":15.92,"load_ms":468.5,"mel_ms":14.1,"encode_ms":649.7,"decode_ms":27.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":712.7,"xrt_wall":15.43,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-primeline.json b/catalog/parakeet-primeline.json new file mode 100644 index 00000000..cd54ae77 --- /dev/null +++ b/catalog/parakeet-primeline.json @@ -0,0 +1,102 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-primeline", + "family": "parakeet", + "display_name": "parakeet-primeline", + "params": 627052166, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "primeline/parakeet-primeline", + "upstream_commit": "3f1a9bc", + "published_repo": "handy-computer/parakeet-primeline-gguf", + "docs_page": "parakeet.md", + "languages": [ + "bg", "hr", "cs", "da", "nl", "en", "et", "fi", "fr", "de", "el", "hu", + "it", "lv", "lt", "mt", "pl", "pt", "ro", "ru", "sk", "sl", "es", "sv", + "uk" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-primeline-F32.gguf","size_bytes":2508435680}, + {"quant":"F16","filename":"parakeet-primeline-F16.gguf","size_bytes":1255869920}, + {"quant":"Q8_0","filename":"parakeet-primeline-Q8_0.gguf","size_bytes":739508640}, + {"quant":"Q6_K","filename":"parakeet-primeline-Q6_K.gguf","size_bytes":610342304}, + {"quant":"Q5_K_M","filename":"parakeet-primeline-Q5_K_M.gguf","size_bytes":548946336}, + {"quant":"Q4_K_M","filename":"parakeet-primeline-Q4_K_M.gguf","size_bytes":485425568} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":16.33,"ci95":[15.43,17.24],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1790,"del":414,"ins":110},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":15.05,"ci95":[14.18,16.05],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1757,"del":212,"ins":92},"empty_hyp":0,"utts_over_50pct":28,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":20.52,"ci95":[19.78,21.34],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3315,"del":603,"ins":219},"empty_hyp":1,"utts_over_50pct":40,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":5.98,"ci95":[5.53,6.44],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":903,"del":145,"ins":71},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":34.76,"ci95":[33.6,35.92],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4853,"del":279,"ins":142},"empty_hyp":0,"utts_over_50pct":83,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.82,"ci95":[4.39,5.34],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":467,"del":187,"ins":48},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.85,"ci95":[3.48,4.21],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":640,"del":181,"ins":76},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":17.17,"ci95":[16.36,18.0],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1936,"del":303,"ins":316},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":13.39,"ci95":[12.6,14.2],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1580,"del":244,"ins":157},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":6.35,"ci95":[5.84,6.89],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":877,"del":214,"ins":52},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":13.68,"ci95":[12.94,14.51],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1981,"del":318,"ins":130},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":17.52,"ci95":[16.7,18.3],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2386,"del":366,"ins":239},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":3.17,"ci95":[2.88,3.48],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":473,"del":147,"ins":54},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":23.08,"ci95":[22.26,23.89],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3484,"del":347,"ins":124},"empty_hyp":0,"utts_over_50pct":41,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":28.83,"ci95":[27.83,29.95],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3892,"del":362,"ins":191},"empty_hyp":0,"utts_over_50pct":73,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":24.74,"ci95":[23.87,25.63],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4469,"del":800,"ins":231},"empty_hyp":0,"utts_over_50pct":45,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":8.49,"ci95":[7.76,9.26],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":526,"del":123,"ins":64},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":8.19,"ci95":[7.45,9.07],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":931,"del":189,"ins":59},"empty_hyp":1,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":5.17,"ci95":[4.72,5.66],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":766,"del":264,"ins":80},"empty_hyp":2,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":13.8,"ci95":[13.12,14.49],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2247,"del":501,"ins":123},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":7.81,"ci95":[7.21,8.42],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":961,"del":177,"ins":42},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":12.36,"ci95":[11.65,13.08],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1563,"del":250,"ins":71},"empty_hyp":3,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":51.07,"ci95":[49.27,52.72],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6883,"del":1319,"ins":233},"empty_hyp":4,"utts_over_50pct":408,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":16.42,"ci95":[15.53,17.29],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1898,"del":502,"ins":126},"empty_hyp":0,"utts_over_50pct":21,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":8.11,"ci95":[7.55,8.7],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":948,"del":159,"ins":58},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":2.2,"ci95":[2.04,2.37],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":866,"del":231,"ins":67},"empty_hyp":0,"utts_over_50pct":5}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"F32","metric":"wer","err_pct":2.2,"ci95":[2.04,2.37],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":867,"del":228,"ins":69},"empty_hyp":0,"utts_over_50pct":5}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"F16","metric":"wer","err_pct":2.19,"ci95":[2.04,2.37],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":866,"del":227,"ins":69},"empty_hyp":0,"utts_over_50pct":5}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q6_K","metric":"wer","err_pct":2.19,"ci95":[2.04,2.36],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":870,"del":225,"ins":67},"empty_hyp":0,"utts_over_50pct":5}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q5_K_M","metric":"wer","err_pct":2.2,"ci95":[2.04,2.37],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":868,"del":228,"ins":70},"empty_hyp":0,"utts_over_50pct":4}, + {"dataset":"librispeech","split":"test-clean","language":"en","language_hint":"en","backend":"cuda","quant":"Q4_K_M","metric":"wer","err_pct":2.26,"ci95":[2.09,2.43],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":"bcd4a0f","measured_on":null,"errors":{"sub":899,"del":231,"ins":70},"empty_hyp":0,"utts_over_50pct":5}, + {"dataset":"fleurs","split":"test","language":"de","quant":"F32","metric":"wer","err_pct":6.0,"ci95":[5.51,6.48],"n_utts":862,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":903,"del":145,"ins":74},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"F16","metric":"wer","err_pct":6.0,"ci95":[5.51,6.48],"n_utts":862,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":903,"del":145,"ins":74},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q6_K","metric":"wer","err_pct":5.96,"ci95":[5.5,6.45],"n_utts":862,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":900,"del":142,"ins":74},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q5_K_M","metric":"wer","err_pct":5.99,"ci95":[5.5,6.48],"n_utts":862,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":896,"del":148,"ins":77},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q4_K_M","metric":"wer","err_pct":5.98,"ci95":[5.5,6.48],"n_utts":862,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":905,"del":137,"ins":78},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "fleurs", + "split": "test", + "language": "de", + "metric": "wer", + "batch_size": null, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1072.9,"xrt_compute":32.93,"load_ms":176.5,"mel_ms":2.3,"encode_ms":1019.9,"decode_ms":50.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1076.2,"xrt_wall":32.83,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":308.6,"xrt_compute":35.65,"load_ms":167.2,"mel_ms":0.9,"encode_ms":296.3,"decode_ms":11.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":310.2,"xrt_wall":35.46,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1006.2,"xrt_compute":35.11,"load_ms":231.5,"mel_ms":2.3,"encode_ms":955.6,"decode_ms":48.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1009.5,"xrt_wall":35.0,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":286.7,"xrt_compute":38.36,"load_ms":209.9,"mel_ms":0.8,"encode_ms":274.6,"decode_ms":11.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":288.3,"xrt_wall":38.15,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":165.1,"xrt_compute":214.02,"load_ms":160.9,"mel_ms":2.2,"encode_ms":114.6,"decode_ms":48.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":171.5,"xrt_wall":206.06,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":61.6,"xrt_compute":178.53,"load_ms":138.4,"mel_ms":0.8,"encode_ms":49.4,"decode_ms":11.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":64.8,"xrt_wall":169.63,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":164.4,"xrt_compute":214.98,"load_ms":213.9,"mel_ms":2.3,"encode_ms":114.0,"decode_ms":48.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":170.8,"xrt_wall":206.89,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":59.9,"xrt_compute":183.77,"load_ms":185.2,"mel_ms":0.8,"encode_ms":47.7,"decode_ms":11.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":63.1,"xrt_wall":174.41,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2962.2,"xrt_compute":11.93,"load_ms":662.9,"mel_ms":56.0,"encode_ms":2565.7,"decode_ms":340.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2988.3,"xrt_wall":11.82,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":788.7,"xrt_compute":13.95,"load_ms":891.1,"mel_ms":18.6,"encode_ms":698.4,"decode_ms":71.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":798.0,"xrt_wall":13.78,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2900.0,"xrt_compute":12.18,"load_ms":869.5,"mel_ms":56.8,"encode_ms":2514.1,"decode_ms":329.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2926.1,"xrt_wall":12.07,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":728.8,"xrt_compute":15.09,"load_ms":1178.6,"mel_ms":17.5,"encode_ms":639.2,"decode_ms":72.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":738.5,"xrt_wall":14.9,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1392.0,"xrt_compute":25.38,"load_ms":364.7,"mel_ms":59.6,"encode_ms":1005.2,"decode_ms":327.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1411.8,"xrt_wall":25.03,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":454.0,"xrt_compute":24.23,"load_ms":612.1,"mel_ms":17.9,"encode_ms":364.2,"decode_ms":71.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":464.0,"xrt_wall":23.71,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1375.1,"xrt_compute":25.69,"load_ms":460.9,"mel_ms":59.2,"encode_ms":985.9,"decode_ms":330.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1395.2,"xrt_wall":25.32,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":453.5,"xrt_compute":24.25,"load_ms":825.5,"mel_ms":17.9,"encode_ms":360.6,"decode_ms":75.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":464.6,"xrt_wall":23.68,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-rnnt-0.6b.json b/catalog/parakeet-rnnt-0.6b.json new file mode 100644 index 00000000..3e54e03b --- /dev/null +++ b/catalog/parakeet-rnnt-0.6b.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-rnnt-0.6b", + "family": "parakeet", + "display_name": "parakeet-rnnt-0.6b", + "params": 616736257, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-rnnt-0.6b", + "upstream_commit": "c0c1f09", + "published_repo": "handy-computer/parakeet-rnnt-0.6b-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-rnnt-0.6b-F32.gguf","size_bytes":2467033120}, + {"quant":"F16","filename":"parakeet-rnnt-0.6b-F16.gguf","size_bytes":1235969568}, + {"quant":"Q8_0","filename":"parakeet-rnnt-0.6b-Q8_0.gguf","size_bytes":729687456}, + {"quant":"Q6_K","filename":"parakeet-rnnt-0.6b-Q6_K.gguf","size_bytes":600902048}, + {"quant":"Q5_K_M","filename":"parakeet-rnnt-0.6b-Q5_K_M.gguf","size_bytes":539714976}, + {"quant":"Q4_K_M","filename":"parakeet-rnnt-0.6b-Q4_K_M.gguf","size_bytes":476390816} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.57,"ci95":[4.13,5.05],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":446,"del":147,"ins":72},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.62,"ci95":[1.48,1.77],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":649,"del":109,"ins":102},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.62,"ci95":[1.48,1.77],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":649,"del":109,"ins":102},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.62,"ci95":[1.48,1.76],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":649,"del":107,"ins":101},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.62,"ci95":[1.48,1.76],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":645,"del":112,"ins":101},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.62,"ci95":[1.49,1.77],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":649,"del":111,"ins":100},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.66,"ci95":[1.51,1.8],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":660,"del":118,"ins":101},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1038.5,"xrt_compute":34.02,"load_ms":171.9,"mel_ms":1.8,"encode_ms":1005.6,"decode_ms":31.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1041.6,"xrt_wall":33.92,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":301.2,"xrt_compute":36.52,"load_ms":167.7,"mel_ms":0.7,"encode_ms":293.6,"decode_ms":6.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":302.8,"xrt_wall":36.32,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":977.0,"xrt_compute":36.16,"load_ms":232.8,"mel_ms":1.8,"encode_ms":944.0,"decode_ms":31.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":980.1,"xrt_wall":36.05,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":280.6,"xrt_compute":39.19,"load_ms":219.7,"mel_ms":0.7,"encode_ms":272.7,"decode_ms":7.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":282.3,"xrt_wall":38.96,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":142.2,"xrt_compute":248.43,"load_ms":146.7,"mel_ms":1.7,"encode_ms":109.8,"decode_ms":30.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":148.2,"xrt_wall":238.33,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":55.5,"xrt_compute":198.17,"load_ms":132.3,"mel_ms":0.6,"encode_ms":48.0,"decode_ms":6.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":59.1,"xrt_wall":186.24,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":138.4,"xrt_compute":255.29,"load_ms":201.3,"mel_ms":1.7,"encode_ms":107.5,"decode_ms":29.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":144.2,"xrt_wall":245.01,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":54.7,"xrt_compute":201.01,"load_ms":187.0,"mel_ms":0.7,"encode_ms":46.8,"decode_ms":7.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":58.4,"xrt_wall":188.44,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2708.4,"xrt_compute":13.05,"load_ms":620.1,"mel_ms":38.1,"encode_ms":2494.2,"decode_ms":176.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2728.6,"xrt_wall":12.95,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":719.6,"xrt_compute":15.29,"load_ms":824.1,"mel_ms":12.2,"encode_ms":675.0,"decode_ms":32.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":724.2,"xrt_wall":15.19,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2637.8,"xrt_compute":13.39,"load_ms":810.4,"mel_ms":38.1,"encode_ms":2425.6,"decode_ms":174.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2658.4,"xrt_wall":13.29,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":674.7,"xrt_compute":16.3,"load_ms":1158.2,"mel_ms":13.2,"encode_ms":628.9,"decode_ms":32.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":679.2,"xrt_wall":16.2,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1268.4,"xrt_compute":27.86,"load_ms":280.3,"mel_ms":40.1,"encode_ms":1057.3,"decode_ms":171.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1288.4,"xrt_wall":27.42,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":431.2,"xrt_compute":25.51,"load_ms":521.2,"mel_ms":12.7,"encode_ms":385.4,"decode_ms":33.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":441.6,"xrt_wall":24.91,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1247.4,"xrt_compute":28.33,"load_ms":384.6,"mel_ms":39.2,"encode_ms":1036.4,"decode_ms":171.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1267.6,"xrt_wall":27.87,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":425.2,"xrt_compute":25.87,"load_ms":780.1,"mel_ms":12.9,"encode_ms":379.1,"decode_ms":33.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":435.6,"xrt_wall":25.25,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-rnnt-1.1b.json b/catalog/parakeet-rnnt-1.1b.json new file mode 100644 index 00000000..ef0e2efb --- /dev/null +++ b/catalog/parakeet-rnnt-1.1b.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-rnnt-1.1b", + "family": "parakeet", + "display_name": "parakeet-rnnt-1.1b", + "params": 1070513665, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-rnnt-1.1b", + "upstream_commit": "a07b19e", + "published_repo": "handy-computer/parakeet-rnnt-1.1b-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-rnnt-1.1b-F32.gguf","size_bytes":4282189696}, + {"quant":"F16","filename":"parakeet-rnnt-1.1b-F16.gguf","size_bytes":2145156480}, + {"quant":"Q8_0","filename":"parakeet-rnnt-1.1b-Q8_0.gguf","size_bytes":1267285248}, + {"quant":"Q6_K","filename":"parakeet-rnnt-1.1b-Q6_K.gguf","size_bytes":1042505984}, + {"quant":"Q5_K_M","filename":"parakeet-rnnt-1.1b-Q5_K_M.gguf","size_bytes":935755008}, + {"quant":"Q4_K_M","filename":"parakeet-rnnt-1.1b-Q4_K_M.gguf","size_bytes":825244928} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.45,"ci95":[3.99,4.91],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":428,"del":137,"ins":83},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.45,"ci95":[1.31,1.59],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":552,"del":115,"ins":101},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.45,"ci95":[1.31,1.59],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":552,"del":115,"ins":101},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.46,"ci95":[1.32,1.6],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":555,"del":118,"ins":101},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.43,"ci95":[1.29,1.57],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":552,"del":111,"ins":97},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.43,"ci95":[1.29,1.57],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":555,"del":107,"ins":97},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.41,"ci95":[1.27,1.54],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":546,"del":110,"ins":90},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1855.7,"xrt_compute":19.04,"load_ms":297.5,"mel_ms":1.9,"encode_ms":1822.0,"decode_ms":31.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1860.1,"xrt_wall":18.99,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":515.6,"xrt_compute":21.34,"load_ms":273.2,"mel_ms":0.7,"encode_ms":508.0,"decode_ms":6.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":518.4,"xrt_wall":21.22,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1640.7,"xrt_compute":21.53,"load_ms":377.6,"mel_ms":1.8,"encode_ms":1608.1,"decode_ms":30.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1645.1,"xrt_wall":21.48,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":491.8,"xrt_compute":22.37,"load_ms":347.4,"mel_ms":0.7,"encode_ms":483.7,"decode_ms":7.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":494.1,"xrt_wall":22.26,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":207.1,"xrt_compute":170.62,"load_ms":238.2,"mel_ms":1.7,"encode_ms":175.7,"decode_ms":29.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":215.0,"xrt_wall":164.35,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":85.8,"xrt_compute":128.14,"load_ms":218.4,"mel_ms":0.7,"encode_ms":78.3,"decode_ms":6.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":91.3,"xrt_wall":120.51,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":201.3,"xrt_compute":175.51,"load_ms":338.5,"mel_ms":1.7,"encode_ms":170.0,"decode_ms":29.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":209.4,"xrt_wall":168.69,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":83.6,"xrt_compute":131.63,"load_ms":322.1,"mel_ms":0.7,"encode_ms":76.2,"decode_ms":6.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":88.8,"xrt_wall":123.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4725.3,"xrt_compute":7.48,"load_ms":1388.7,"mel_ms":40.8,"encode_ms":4505.6,"decode_ms":179.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4748.2,"xrt_wall":7.44,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1212.0,"xrt_compute":9.08,"load_ms":1029.8,"mel_ms":13.8,"encode_ms":1165.6,"decode_ms":32.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1218.9,"xrt_wall":9.02,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4789.2,"xrt_compute":7.38,"load_ms":1943.4,"mel_ms":39.2,"encode_ms":4576.1,"decode_ms":173.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4813.0,"xrt_wall":7.34,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1121.9,"xrt_compute":9.8,"load_ms":1353.4,"mel_ms":12.1,"encode_ms":1075.7,"decode_ms":34.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1128.7,"xrt_wall":9.75,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2181.1,"xrt_compute":16.2,"load_ms":813.9,"mel_ms":41.7,"encode_ms":1966.2,"decode_ms":173.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2223.3,"xrt_wall":15.89,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":709.5,"xrt_compute":15.5,"load_ms":386.8,"mel_ms":12.1,"encode_ms":664.5,"decode_ms":32.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":733.7,"xrt_wall":14.99,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2145.2,"xrt_compute":16.47,"load_ms":1178.5,"mel_ms":40.4,"encode_ms":1924.8,"decode_ms":180.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2187.1,"xrt_wall":16.15,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":700.8,"xrt_compute":15.7,"load_ms":556.3,"mel_ms":12.1,"encode_ms":654.3,"decode_ms":34.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":730.4,"xrt_wall":15.06,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-tdt-0.6b-v2.json b/catalog/parakeet-tdt-0.6b-v2.json new file mode 100644 index 00000000..7ae1c465 --- /dev/null +++ b/catalog/parakeet-tdt-0.6b-v2.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-tdt-0.6b-v2", + "family": "parakeet", + "display_name": "parakeet-tdt-0.6b-v2", + "params": 617869958, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-tdt-0.6b-v2", + "upstream_commit": "1b149a3", + "published_repo": "handy-computer/parakeet-tdt-0.6b-v2-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-tdt-0.6b-v2-F32.gguf","size_bytes":2471550272}, + {"quant":"F16","filename":"parakeet-tdt-0.6b-v2-F16.gguf","size_bytes":1237334592}, + {"quant":"Q8_0","filename":"parakeet-tdt-0.6b-v2-Q8_0.gguf","size_bytes":729574912}, + {"quant":"Q6_K","filename":"parakeet-tdt-0.6b-v2-Q6_K.gguf","size_bytes":600408576}, + {"quant":"Q5_K_M","filename":"parakeet-tdt-0.6b-v2-Q5_K_M.gguf","size_bytes":539012608}, + {"quant":"Q4_K_M","filename":"parakeet-tdt-0.6b-v2-Q4_K_M.gguf","size_bytes":475491840} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.11,"ci95":[3.71,4.53],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":406,"del":125,"ins":68},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.68,"ci95":[1.54,1.83],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":670,"del":123,"ins":99},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.68,"ci95":[1.54,1.83],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":670,"del":123,"ins":99},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.69,"ci95":[1.54,1.83],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":670,"del":124,"ins":100},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.7,"ci95":[1.56,1.85],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":676,"del":124,"ins":102},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.7,"ci95":[1.56,1.85],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":677,"del":126,"ins":98},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.72,"ci95":[1.57,1.87],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":682,"del":129,"ins":101},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1086.3,"xrt_compute":32.53,"load_ms":189.6,"mel_ms":2.5,"encode_ms":1055.6,"decode_ms":28.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1090.0,"xrt_wall":32.42,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":317.5,"xrt_compute":34.64,"load_ms":171.8,"mel_ms":0.9,"encode_ms":310.2,"decode_ms":6.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":319.2,"xrt_wall":34.46,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":987.0,"xrt_compute":35.8,"load_ms":245.9,"mel_ms":2.4,"encode_ms":957.2,"decode_ms":27.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":990.5,"xrt_wall":35.67,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":277.5,"xrt_compute":39.64,"load_ms":205.3,"mel_ms":0.8,"encode_ms":271.1,"decode_ms":5.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":279.0,"xrt_wall":39.42,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":145.2,"xrt_compute":243.28,"load_ms":144.8,"mel_ms":2.3,"encode_ms":116.4,"decode_ms":26.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":152.2,"xrt_wall":232.14,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":56.1,"xrt_compute":196.11,"load_ms":133.9,"mel_ms":0.8,"encode_ms":49.7,"decode_ms":5.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":59.4,"xrt_wall":185.1,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":142.3,"xrt_compute":248.28,"load_ms":204.6,"mel_ms":2.3,"encode_ms":112.9,"decode_ms":27.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":149.6,"xrt_wall":236.17,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":55.0,"xrt_compute":199.95,"load_ms":184.8,"mel_ms":0.8,"encode_ms":48.7,"decode_ms":5.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":58.3,"xrt_wall":188.74,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2831.1,"xrt_compute":12.48,"load_ms":621.4,"mel_ms":57.0,"encode_ms":2563.9,"decode_ms":210.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2857.2,"xrt_wall":12.37,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":748.7,"xrt_compute":14.69,"load_ms":845.0,"mel_ms":17.8,"encode_ms":694.9,"decode_ms":36.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":758.7,"xrt_wall":14.5,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2774.6,"xrt_compute":12.73,"load_ms":803.3,"mel_ms":57.0,"encode_ms":2510.7,"decode_ms":206.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2801.3,"xrt_wall":12.61,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":695.8,"xrt_compute":15.81,"load_ms":1122.5,"mel_ms":17.9,"encode_ms":641.7,"decode_ms":36.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":706.0,"xrt_wall":15.58,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1257.9,"xrt_compute":28.09,"load_ms":283.3,"mel_ms":57.4,"encode_ms":1003.8,"decode_ms":196.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1277.6,"xrt_wall":27.66,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":417.6,"xrt_compute":26.34,"load_ms":509.3,"mel_ms":17.1,"encode_ms":363.8,"decode_ms":36.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":429.1,"xrt_wall":25.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1243.3,"xrt_compute":28.42,"load_ms":371.8,"mel_ms":57.9,"encode_ms":986.2,"decode_ms":199.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1259.3,"xrt_wall":28.06,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":415.9,"xrt_compute":26.45,"load_ms":716.2,"mel_ms":17.4,"encode_ms":361.2,"decode_ms":37.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":426.2,"xrt_wall":25.81,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-tdt-0.6b-v3.json b/catalog/parakeet-tdt-0.6b-v3.json new file mode 100644 index 00000000..53b67bcf --- /dev/null +++ b/catalog/parakeet-tdt-0.6b-v3.json @@ -0,0 +1,97 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-tdt-0.6b-v3", + "family": "parakeet", + "display_name": "parakeet-tdt-0.6b-v3", + "params": 627052166, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-tdt-0.6b-v3", + "upstream_commit": "6d590f7", + "published_repo": "handy-computer/parakeet-tdt-0.6b-v3-gguf", + "docs_page": "parakeet.md", + "languages": [ + "bg", "hr", "cs", "da", "nl", "en", "et", "fi", "fr", "de", "el", "hu", + "it", "lv", "lt", "mt", "pl", "pt", "ro", "ru", "sk", "sl", "es", "sv", + "uk" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-tdt-0.6b-v3-F32.gguf","size_bytes":2508435616}, + {"quant":"F16","filename":"parakeet-tdt-0.6b-v3-F16.gguf","size_bytes":1255869856}, + {"quant":"Q8_0","filename":"parakeet-tdt-0.6b-v3-Q8_0.gguf","size_bytes":739508576}, + {"quant":"Q6_K","filename":"parakeet-tdt-0.6b-v3-Q6_K.gguf","size_bytes":610342240}, + {"quant":"Q5_K_M","filename":"parakeet-tdt-0.6b-v3-Q5_K_M.gguf","size_bytes":548946272}, + {"quant":"Q4_K_M","filename":"parakeet-tdt-0.6b-v3-Q4_K_M.gguf","size_bytes":485425504} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":12.81,"ci95":[12.01,13.64],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1420,"del":278,"ins":118},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":12.31,"ci95":[11.42,13.18],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1383,"del":185,"ins":117},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":18.64,"ci95":[17.84,19.43],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3009,"del":457,"ins":292},"empty_hyp":1,"utts_over_50pct":25,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":5.24,"ci95":[4.83,5.66],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":755,"del":147,"ins":79},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":35.33,"ci95":[34.07,36.55],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4839,"del":374,"ins":147},"empty_hyp":0,"utts_over_50pct":97,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.83,"ci95":[4.39,5.3],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":484,"del":158,"ins":61},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.65,"ci95":[3.32,4.0],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":625,"del":143,"ins":83},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":17.96,"ci95":[17.06,18.8],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1998,"del":301,"ins":374},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":13.3,"ci95":[12.48,14.1],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1588,"del":226,"ins":153},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":5.3,"ci95":[4.77,5.78],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":748,"del":124,"ins":81},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":12.59,"ci95":[11.85,13.37],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1794,"del":266,"ins":175},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":16.06,"ci95":[15.24,16.89],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2211,"del":303,"ins":228},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":3.02,"ci95":[2.72,3.34],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":490,"del":92,"ins":60},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":22.2,"ci95":[21.36,23.02],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3363,"del":325,"ins":115},"empty_hyp":0,"utts_over_50pct":38,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":23.77,"ci95":[22.87,24.77],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3224,"del":255,"ins":185},"empty_hyp":0,"utts_over_50pct":41,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":20.63,"ci95":[19.86,21.43],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3672,"del":603,"ins":312},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":7.66,"ci95":[6.83,8.46],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":489,"del":87,"ins":67},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":7.37,"ci95":[6.75,7.97],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":883,"del":120,"ins":57},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":4.96,"ci95":[4.49,5.52],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":736,"del":169,"ins":160},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":12.62,"ci95":[11.95,13.33],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2052,"del":408,"ins":165},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":6.54,"ci95":[6.08,7.06],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":837,"del":104,"ins":47},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":10.19,"ci95":[9.43,10.91],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1230,"del":241,"ins":82},"empty_hyp":4,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":24.3,"ci95":[23.22,25.5],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3219,"del":446,"ins":348},"empty_hyp":0,"utts_over_50pct":44,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":15.25,"ci95":[14.38,16.05],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1824,"del":349,"ins":174},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":6.84,"ci95":[6.25,7.46],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":794,"del":124,"ins":65},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.95,"ci95":[1.79,2.12],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":768,"del":129,"ins":138},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.95,"ci95":[1.79,2.12],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":767,"del":129,"ins":138},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.94,"ci95":[1.78,2.1],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":766,"del":125,"ins":137},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.93,"ci95":[1.77,2.09],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":760,"del":127,"ins":134},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.92,"ci95":[1.77,2.09],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":766,"del":121,"ins":133},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.98,"ci95":[1.82,2.15],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":788,"del":128,"ins":135},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1080.2,"xrt_compute":32.71,"load_ms":177.1,"mel_ms":2.2,"encode_ms":1028.8,"decode_ms":49.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1083.6,"xrt_wall":32.61,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":309.6,"xrt_compute":35.53,"load_ms":166.9,"mel_ms":0.8,"encode_ms":297.4,"decode_ms":11.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":311.2,"xrt_wall":35.35,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1004.9,"xrt_compute":35.16,"load_ms":236.0,"mel_ms":2.3,"encode_ms":954.0,"decode_ms":48.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1008.3,"xrt_wall":35.04,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":285.9,"xrt_compute":38.48,"load_ms":216.9,"mel_ms":0.9,"encode_ms":273.6,"decode_ms":11.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":287.5,"xrt_wall":38.27,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":166.6,"xrt_compute":212.12,"load_ms":157.2,"mel_ms":2.2,"encode_ms":116.1,"decode_ms":48.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":173.8,"xrt_wall":203.31,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":61.6,"xrt_compute":178.45,"load_ms":139.4,"mel_ms":0.8,"encode_ms":49.5,"decode_ms":11.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":65.0,"xrt_wall":169.32,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":164.0,"xrt_compute":215.45,"load_ms":217.9,"mel_ms":2.2,"encode_ms":113.1,"decode_ms":48.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":170.9,"xrt_wall":206.75,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":60.5,"xrt_compute":181.74,"load_ms":196.6,"mel_ms":0.8,"encode_ms":48.6,"decode_ms":11.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":64.0,"xrt_wall":171.84,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2972.3,"xrt_compute":11.89,"load_ms":658.8,"mel_ms":57.8,"encode_ms":2584.2,"decode_ms":330.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2998.3,"xrt_wall":11.78,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":793.7,"xrt_compute":13.86,"load_ms":886.9,"mel_ms":18.1,"encode_ms":702.1,"decode_ms":73.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":802.9,"xrt_wall":13.7,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2891.7,"xrt_compute":12.22,"load_ms":907.0,"mel_ms":55.8,"encode_ms":2509.9,"decode_ms":326.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2917.8,"xrt_wall":12.11,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":729.1,"xrt_compute":15.09,"load_ms":1184.1,"mel_ms":18.1,"encode_ms":639.3,"decode_ms":71.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":738.6,"xrt_wall":14.89,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1385.4,"xrt_compute":25.5,"load_ms":374.0,"mel_ms":56.7,"encode_ms":1006.7,"decode_ms":322.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1405.7,"xrt_wall":25.14,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":458.2,"xrt_compute":24.01,"load_ms":595.2,"mel_ms":19.5,"encode_ms":366.5,"decode_ms":72.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":469.2,"xrt_wall":23.44,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1367.5,"xrt_compute":25.84,"load_ms":447.9,"mel_ms":57.2,"encode_ms":983.3,"decode_ms":327.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1387.8,"xrt_wall":25.46,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":452.9,"xrt_compute":24.29,"load_ms":892.4,"mel_ms":18.4,"encode_ms":362.2,"decode_ms":72.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":463.1,"xrt_wall":23.75,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-tdt-1.1b.json b/catalog/parakeet-tdt-1.1b.json new file mode 100644 index 00000000..2ddab8a9 --- /dev/null +++ b/catalog/parakeet-tdt-1.1b.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-tdt-1.1b", + "family": "parakeet", + "display_name": "parakeet-tdt-1.1b", + "params": 1070516870, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-tdt-1.1b", + "upstream_commit": "53276c6", + "published_repo": "handy-computer/parakeet-tdt-1.1b-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-tdt-1.1b-F32.gguf","size_bytes":4282202592}, + {"quant":"F16","filename":"parakeet-tdt-1.1b-F16.gguf","size_bytes":2145162976}, + {"quant":"Q8_0","filename":"parakeet-tdt-1.1b-Q8_0.gguf","size_bytes":1267288736}, + {"quant":"Q6_K","filename":"parakeet-tdt-1.1b-Q6_K.gguf","size_bytes":1042509472}, + {"quant":"Q5_K_M","filename":"parakeet-tdt-1.1b-Q5_K_M.gguf","size_bytes":935758496}, + {"quant":"Q4_K_M","filename":"parakeet-tdt-1.1b-Q4_K_M.gguf","size_bytes":825248416} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.24,"ci95":[3.78,4.71],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":414,"del":124,"ins":79},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.39,"ci95":[1.24,1.52],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":548,"del":98,"ins":89},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.39,"ci95":[1.24,1.52],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":548,"del":98,"ins":89},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.38,"ci95":[1.24,1.52],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":548,"del":98,"ins":88},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.4,"ci95":[1.26,1.53],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":552,"del":98,"ins":90},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.39,"ci95":[1.25,1.53],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":549,"del":98,"ins":88},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.42,"ci95":[1.28,1.55],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":562,"del":105,"ins":86},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1761.1,"xrt_compute":20.06,"load_ms":290.5,"mel_ms":1.9,"encode_ms":1735.5,"decode_ms":23.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1765.2,"xrt_wall":20.02,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":517.7,"xrt_compute":21.25,"load_ms":260.5,"mel_ms":0.6,"encode_ms":512.2,"decode_ms":4.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":519.9,"xrt_wall":21.16,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1761.1,"xrt_compute":20.06,"load_ms":393.5,"mel_ms":1.9,"encode_ms":1732.0,"decode_ms":27.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1765.9,"xrt_wall":20.01,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":674.6,"xrt_compute":16.31,"load_ms":365.7,"mel_ms":0.7,"encode_ms":668.4,"decode_ms":5.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":677.0,"xrt_wall":16.25,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":201.2,"xrt_compute":175.57,"load_ms":253.9,"mel_ms":1.7,"encode_ms":176.2,"decode_ms":23.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":209.7,"xrt_wall":168.46,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":84.0,"xrt_compute":130.98,"load_ms":216.2,"mel_ms":0.6,"encode_ms":78.5,"decode_ms":4.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":89.5,"xrt_wall":122.86,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":195.8,"xrt_compute":180.48,"load_ms":342.6,"mel_ms":1.7,"encode_ms":171.4,"decode_ms":22.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":203.9,"xrt_wall":173.28,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":81.3,"xrt_compute":135.38,"load_ms":319.2,"mel_ms":0.7,"encode_ms":75.7,"decode_ms":4.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":86.9,"xrt_wall":126.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4664.5,"xrt_compute":7.57,"load_ms":1270.3,"mel_ms":39.3,"encode_ms":4458.4,"decode_ms":166.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4688.5,"xrt_wall":7.54,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1211.7,"xrt_compute":9.08,"load_ms":1433.3,"mel_ms":11.7,"encode_ms":1169.8,"decode_ms":30.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1218.5,"xrt_wall":9.03,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4537.2,"xrt_compute":7.79,"load_ms":1921.7,"mel_ms":38.7,"encode_ms":4332.9,"decode_ms":165.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4560.1,"xrt_wall":7.75,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1120.1,"xrt_compute":9.82,"load_ms":1894.2,"mel_ms":12.1,"encode_ms":1077.2,"decode_ms":30.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1126.4,"xrt_wall":9.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2121.5,"xrt_compute":16.65,"load_ms":414.5,"mel_ms":41.5,"encode_ms":1913.9,"decode_ms":166.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2171.3,"xrt_wall":16.27,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":706.0,"xrt_compute":15.58,"load_ms":833.3,"mel_ms":14.1,"encode_ms":661.5,"decode_ms":30.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":738.4,"xrt_wall":14.9,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2074.9,"xrt_compute":17.03,"load_ms":553.8,"mel_ms":41.4,"encode_ms":1869.4,"decode_ms":164.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2120.0,"xrt_wall":16.67,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":693.6,"xrt_compute":15.86,"load_ms":1180.2,"mel_ms":12.4,"encode_ms":650.8,"decode_ms":30.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":724.2,"xrt_wall":15.19,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-tdt_ctc-1.1b.json b/catalog/parakeet-tdt_ctc-1.1b.json new file mode 100644 index 00000000..05353067 --- /dev/null +++ b/catalog/parakeet-tdt_ctc-1.1b.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-tdt_ctc-1.1b", + "family": "parakeet", + "display_name": "parakeet-tdt_ctc-1.1b", + "params": 1070516870, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-tdt_ctc-1.1b", + "upstream_commit": "675e786", + "published_repo": "handy-computer/parakeet-tdt_ctc-1.1b-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-tdt_ctc-1.1b-F32.gguf","size_bytes":4282202176}, + {"quant":"F16","filename":"parakeet-tdt_ctc-1.1b-F16.gguf","size_bytes":2145162560}, + {"quant":"Q8_0","filename":"parakeet-tdt_ctc-1.1b-Q8_0.gguf","size_bytes":1267288320}, + {"quant":"Q6_K","filename":"parakeet-tdt_ctc-1.1b-Q6_K.gguf","size_bytes":1042509056}, + {"quant":"Q5_K_M","filename":"parakeet-tdt_ctc-1.1b-Q5_K_M.gguf","size_bytes":935758080}, + {"quant":"Q4_K_M","filename":"parakeet-tdt_ctc-1.1b-Q4_K_M.gguf","size_bytes":825248000} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.68,"ci95":[4.22,5.15],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":481,"del":127,"ins":73},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.87,"ci95":[1.72,2.02],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":752,"del":151,"ins":89},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.87,"ci95":[1.72,2.02],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":752,"del":151,"ins":89},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.87,"ci95":[1.72,2.02],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":754,"del":151,"ins":88},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.87,"ci95":[1.71,2.02],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":754,"del":151,"ins":85},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.87,"ci95":[1.71,2.01],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":750,"del":157,"ins":82},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.91,"ci95":[1.74,2.06],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":769,"del":158,"ins":85},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1566.1,"xrt_compute":22.56,"load_ms":284.7,"mel_ms":1.8,"encode_ms":1537.3,"decode_ms":27.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1570.5,"xrt_wall":22.5,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":508.8,"xrt_compute":21.62,"load_ms":265.3,"mel_ms":0.7,"encode_ms":502.2,"decode_ms":5.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":510.5,"xrt_wall":21.55,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1439.5,"xrt_compute":24.54,"load_ms":378.2,"mel_ms":1.7,"encode_ms":1410.3,"decode_ms":27.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1443.7,"xrt_wall":24.47,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":458.1,"xrt_compute":24.01,"load_ms":349.9,"mel_ms":0.7,"encode_ms":451.6,"decode_ms":5.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":459.8,"xrt_wall":23.92,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":187.7,"xrt_compute":188.19,"load_ms":243.3,"mel_ms":1.8,"encode_ms":158.8,"decode_ms":27.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":200.8,"xrt_wall":175.96,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":73.0,"xrt_compute":150.6,"load_ms":219.5,"mel_ms":0.6,"encode_ms":66.4,"decode_ms":6.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":77.0,"xrt_wall":142.87,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":184.2,"xrt_compute":191.86,"load_ms":343.0,"mel_ms":1.8,"encode_ms":156.2,"decode_ms":26.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":196.7,"xrt_wall":179.63,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":71.6,"xrt_compute":153.68,"load_ms":304.1,"mel_ms":0.7,"encode_ms":65.1,"decode_ms":5.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":75.4,"xrt_wall":145.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4230.9,"xrt_compute":8.35,"load_ms":1190.8,"mel_ms":41.1,"encode_ms":4001.0,"decode_ms":188.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4288.7,"xrt_wall":8.24,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1197.6,"xrt_compute":9.18,"load_ms":1408.5,"mel_ms":12.4,"encode_ms":1149.8,"decode_ms":35.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1202.6,"xrt_wall":9.15,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4107.3,"xrt_compute":8.6,"load_ms":1895.9,"mel_ms":41.7,"encode_ms":3876.3,"decode_ms":189.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4163.7,"xrt_wall":8.49,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1096.4,"xrt_compute":10.03,"load_ms":1919.6,"mel_ms":12.4,"encode_ms":1048.8,"decode_ms":35.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1101.5,"xrt_wall":9.99,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1956.7,"xrt_compute":18.06,"load_ms":377.9,"mel_ms":39.2,"encode_ms":1727.3,"decode_ms":190.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1978.2,"xrt_wall":17.86,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":705.7,"xrt_compute":15.59,"load_ms":796.3,"mel_ms":12.3,"encode_ms":656.5,"decode_ms":36.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":718.3,"xrt_wall":15.31,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1929.1,"xrt_compute":18.32,"load_ms":589.8,"mel_ms":40.4,"encode_ms":1699.8,"decode_ms":188.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1950.2,"xrt_wall":18.12,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":697.8,"xrt_compute":15.76,"load_ms":1198.4,"mel_ms":12.1,"encode_ms":650.3,"decode_ms":35.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":711.0,"xrt_wall":15.47,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-tdt_ctc-110m.json b/catalog/parakeet-tdt_ctc-110m.json new file mode 100644 index 00000000..249bd651 --- /dev/null +++ b/catalog/parakeet-tdt_ctc-110m.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-tdt_ctc-110m", + "family": "parakeet", + "display_name": "parakeet-tdt_ctc-110m", + "params": 114113670, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-tdt_ctc-110m", + "upstream_commit": "431a349", + "published_repo": "handy-computer/parakeet-tdt_ctc-110m-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["token"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-tdt_ctc-110m-F32.gguf","size_bytes":456524064}, + {"quant":"F16","filename":"parakeet-tdt_ctc-110m-F16.gguf","size_bytes":229334560}, + {"quant":"Q8_0","filename":"parakeet-tdt_ctc-110m-Q8_0.gguf","size_bytes":135373280}, + {"quant":"Q6_K","filename":"parakeet-tdt_ctc-110m-Q6_K.gguf","size_bytes":112311264}, + {"quant":"Q5_K_M","filename":"parakeet-tdt_ctc-110m-Q5_K_M.gguf","size_bytes":101335520}, + {"quant":"Q4_K_M","filename":"parakeet-tdt_ctc-110m-Q4_K_M.gguf","size_bytes":89989600} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":6.11,"ci95":[5.59,6.66],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":630,"del":164,"ins":95},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.43,"ci95":[2.25,2.6],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1007,"del":154,"ins":127},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.43,"ci95":[2.26,2.6],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1006,"del":154,"ins":128},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.43,"ci95":[2.26,2.6],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1003,"del":156,"ins":130},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.44,"ci95":[2.26,2.61],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1005,"del":154,"ins":133},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.47,"ci95":[2.29,2.65],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1029,"del":149,"ins":131},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.53,"ci95":[2.35,2.72],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1050,"del":160,"ins":133},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":279.3,"xrt_compute":126.5,"load_ms":60.4,"mel_ms":1.7,"encode_ms":263.3,"decode_ms":14.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":281.4,"xrt_wall":125.56,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":78.2,"xrt_compute":140.61,"load_ms":55.4,"mel_ms":0.6,"encode_ms":74.3,"decode_ms":3.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":79.4,"xrt_wall":138.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":252.0,"xrt_compute":140.19,"load_ms":67.7,"mel_ms":1.8,"encode_ms":236.5,"decode_ms":13.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":254.1,"xrt_wall":139.02,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":70.1,"xrt_compute":156.94,"load_ms":64.1,"mel_ms":0.7,"encode_ms":65.8,"decode_ms":3.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":71.2,"xrt_wall":154.55,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":69.3,"xrt_compute":510.03,"load_ms":57.1,"mel_ms":1.7,"encode_ms":53.1,"decode_ms":14.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":74.1,"xrt_wall":476.69,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":27.4,"xrt_compute":401.4,"load_ms":52.6,"mel_ms":0.7,"encode_ms":23.4,"decode_ms":3.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":30.1,"xrt_wall":365.67,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":69.3,"xrt_compute":509.69,"load_ms":65.2,"mel_ms":1.7,"encode_ms":53.2,"decode_ms":14.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":74.4,"xrt_wall":474.96,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":27.1,"xrt_compute":405.32,"load_ms":61.1,"mel_ms":0.6,"encode_ms":23.2,"decode_ms":3.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":29.8,"xrt_wall":369.55,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":869.9,"xrt_compute":40.62,"load_ms":199.5,"mel_ms":38.5,"encode_ms":727.8,"decode_ms":103.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":884.7,"xrt_wall":39.94,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":231.4,"xrt_compute":47.53,"load_ms":163.9,"mel_ms":12.0,"encode_ms":197.9,"decode_ms":21.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":234.8,"xrt_wall":46.85,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":790.4,"xrt_compute":44.7,"load_ms":238.6,"mel_ms":37.4,"encode_ms":640.9,"decode_ms":112.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":805.7,"xrt_wall":43.85,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":198.7,"xrt_compute":55.36,"load_ms":178.0,"mel_ms":13.3,"encode_ms":163.6,"decode_ms":21.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":202.1,"xrt_wall":54.44,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":468.1,"xrt_compute":75.48,"load_ms":167.1,"mel_ms":39.3,"encode_ms":317.4,"decode_ms":111.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":480.3,"xrt_wall":73.56,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":144.1,"xrt_compute":76.32,"load_ms":125.8,"mel_ms":14.1,"encode_ms":108.3,"decode_ms":21.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":151.7,"xrt_wall":72.5,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":467.2,"xrt_compute":75.62,"load_ms":201.5,"mel_ms":39.1,"encode_ms":313.4,"decode_ms":114.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":480.7,"xrt_wall":73.5,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":142.3,"xrt_compute":77.29,"load_ms":140.9,"mel_ms":11.8,"encode_ms":106.8,"decode_ms":23.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":149.6,"xrt_wall":73.51,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/parakeet-unified-en-0.6b.json b/catalog/parakeet-unified-en-0.6b.json new file mode 100644 index 00000000..5bbc230e --- /dev/null +++ b/catalog/parakeet-unified-en-0.6b.json @@ -0,0 +1,84 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "parakeet-unified-en-0.6b", + "family": "parakeet", + "display_name": "Parakeet Unified EN 0.6B", + "params": 618309121, + "license": { + "spdx": "cc-by-4.0", + "display": "CC-BY-4.0" + }, + "upstream_repo": "nvidia/parakeet-unified-en-0.6b", + "upstream_commit": "d4ac992", + "published_repo": "handy-computer/parakeet-unified-en-0.6b-gguf", + "docs_page": "parakeet.md", + "languages": ["en"], + "language_tag_form": "bare-bcp47", + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":true}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["token"]}, + "streaming": { + "supported": true, + "verified": true, + "note": "byte-equal transcript vs NeMo speech_to_text_streaming_infer_rnnt.py at the default (70, 13, 13)", + "mode": "buffered", + "presets": [ + {"name":"70-1-0","left_ms":5600,"chunk_ms":80,"right_ms":0,"lookahead_ms":80,"is_default":false}, + {"name":"70-1-1","left_ms":5600,"chunk_ms":80,"right_ms":80,"lookahead_ms":160,"is_default":false}, + {"name":"70-2-2","left_ms":5600,"chunk_ms":160,"right_ms":160,"lookahead_ms":320,"is_default":false}, + {"name":"70-2-4","left_ms":5600,"chunk_ms":160,"right_ms":320,"lookahead_ms":480,"is_default":false}, + {"name":"70-7-7","left_ms":5600,"chunk_ms":560,"right_ms":560,"lookahead_ms":1120,"is_default":false}, + {"name":"70-13-13","left_ms":5600,"chunk_ms":1040,"right_ms":1040,"lookahead_ms":2080,"is_default":true} + ] + }, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false}, + "punctuation": {"supported":true,"verified":true} + }, + "downloads": [ + {"quant":"F32","filename":"parakeet-unified-en-0.6b-F32.gguf","size_bytes":2473323520}, + {"quant":"F16","filename":"parakeet-unified-en-0.6b-F16.gguf","size_bytes":1239114240}, + {"quant":"Q8_0","filename":"parakeet-unified-en-0.6b-Q8_0.gguf","size_bytes":731357568}, + {"quant":"Q6_K","filename":"parakeet-unified-en-0.6b-Q6_K.gguf","size_bytes":602191232}, + {"quant":"Q5_K_M","filename":"parakeet-unified-en-0.6b-Q5_K_M.gguf","size_bytes":540795264}, + {"quant":"Q4_K_M","filename":"parakeet-unified-en-0.6b-Q4_K_M.gguf","size_bytes":477274496} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.99,"ci95":[3.6,4.42],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":405,"del":116,"ins":60},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":1.59,"ci95":[1.45,1.73],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":635,"del":118,"ins":91},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.59,"ci95":[1.45,1.74],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":636,"del":118,"ins":91},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.6,"ci95":[1.46,1.75],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":641,"del":117,"ins":92},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.61,"ci95":[1.47,1.76],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":645,"del":119,"ins":91},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.58,"ci95":[1.44,1.73],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":631,"del":118,"ins":90},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.62,"ci95":[1.47,1.76],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":643,"del":123,"ins":92},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1046.2,"xrt_compute":33.77,"load_ms":171.7,"mel_ms":2.2,"encode_ms":1007.3,"decode_ms":36.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1050.0,"xrt_wall":33.65,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":306.8,"xrt_compute":35.86,"load_ms":158.8,"mel_ms":0.8,"encode_ms":297.7,"decode_ms":8.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":308.6,"xrt_wall":35.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":979.0,"xrt_compute":36.09,"load_ms":206.7,"mel_ms":2.2,"encode_ms":940.5,"decode_ms":36.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":982.7,"xrt_wall":35.96,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":285.3,"xrt_compute":38.55,"load_ms":203.6,"mel_ms":0.8,"encode_ms":276.4,"decode_ms":8.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":287.1,"xrt_wall":38.31,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":161.1,"xrt_compute":219.3,"load_ms":128.3,"mel_ms":2.2,"encode_ms":122.8,"decode_ms":36.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":168.0,"xrt_wall":210.31,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":60.0,"xrt_compute":183.34,"load_ms":130.0,"mel_ms":0.8,"encode_ms":50.5,"decode_ms":8.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":63.8,"xrt_wall":172.45,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":154.7,"xrt_compute":228.39,"load_ms":177.2,"mel_ms":2.2,"encode_ms":115.9,"decode_ms":36.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":161.5,"xrt_wall":218.76,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":58.6,"xrt_compute":187.72,"load_ms":182.3,"mel_ms":0.8,"encode_ms":49.7,"decode_ms":8.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":62.5,"xrt_wall":176.1,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2951.7,"xrt_compute":11.97,"load_ms":829.4,"mel_ms":55.2,"encode_ms":2647.6,"decode_ms":248.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":2978.1,"xrt_wall":11.86,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":787.9,"xrt_compute":13.96,"load_ms":621.8,"mel_ms":17.9,"encode_ms":723.6,"decode_ms":46.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":798.0,"xrt_wall":13.78,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2836.3,"xrt_compute":12.46,"load_ms":1111.0,"mel_ms":57.5,"encode_ms":2539.9,"decode_ms":238.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":2864.1,"xrt_wall":12.34,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":711.8,"xrt_compute":15.45,"load_ms":807.7,"mel_ms":18.3,"encode_ms":646.9,"decode_ms":46.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":722.6,"xrt_wall":15.22,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1372.9,"xrt_compute":25.74,"load_ms":506.0,"mel_ms":58.0,"encode_ms":1077.8,"decode_ms":237.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":1392.7,"xrt_wall":25.37,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":455.4,"xrt_compute":24.15,"load_ms":284.1,"mel_ms":17.3,"encode_ms":391.1,"decode_ms":47.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":467.2,"xrt_wall":23.54,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1358.3,"xrt_compute":26.01,"load_ms":729.5,"mel_ms":57.2,"encode_ms":1060.5,"decode_ms":240.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":1378.1,"xrt_wall":25.64,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":448.1,"xrt_compute":24.55,"load_ms":371.5,"mel_ms":16.7,"encode_ms":385.2,"decode_ms":46.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":true,"wall_ms":459.0,"xrt_wall":23.96,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/qwen3-asr-0.6b.json b/catalog/qwen3-asr-0.6b.json new file mode 100644 index 00000000..72767f1e --- /dev/null +++ b/catalog/qwen3-asr-0.6b.json @@ -0,0 +1,102 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "qwen3-asr-0.6b", + "family": "qwen3_asr", + "display_name": "qwen3-asr-0.6b", + "params": 782452240, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "Qwen/Qwen3-ASR-0.6B", + "upstream_commit": "5eb1441", + "published_repo": "handy-computer/Qwen3-ASR-0.6B-gguf", + "docs_page": "qwen3-asr.md", + "languages": [ + "zh", "en", "yue", "ar", "de", "fr", "es", "pt", "id", "it", "ko", "ru", + "th", "vi", "ja", "tr", "hi", "ms", "nl", "sv", "da", "fi", "pl", "cs", + "fil", "fa", "el", "ro", "hu", "mk" + ], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Qwen3-ASR-0.6B-BF16.gguf","size_bytes":1571490016}, + {"quant":"F16","filename":"Qwen3-ASR-0.6B-F16.gguf","size_bytes":1579793056}, + {"quant":"Q8_0","filename":"Qwen3-ASR-0.6B-Q8_0.gguf","size_bytes":850423456}, + {"quant":"Q6_K","filename":"Qwen3-ASR-0.6B-Q6_K.gguf","size_bytes":690417824}, + {"quant":"Q5_K_M","filename":"Qwen3-ASR-0.6B-Q5_K_M.gguf","size_bytes":645356192}, + {"quant":"Q4_K_M","filename":"Qwen3-ASR-0.6B-Q4_K_M.gguf","size_bytes":589560480} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":24.51,"ci95":[22.15,28.02],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1508,"del":348,"ins":163},"empty_hyp":0,"utts_over_50pct":23,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":44.5,"ci95":[43.23,45.72],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5071,"del":649,"ins":373},"empty_hyp":0,"utts_over_50pct":246,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":36.07,"ci95":[35.05,37.08],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5679,"del":828,"ins":763},"empty_hyp":0,"utts_over_50pct":160,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":6.8,"ci95":[6.33,7.3],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":989,"del":169,"ins":115},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":49.12,"ci95":[47.74,50.55],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5693,"del":1534,"ins":226},"empty_hyp":0,"utts_over_50pct":286,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.23,"ci95":[3.76,4.69],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":406,"del":126,"ins":84},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":4.88,"ci95":[4.5,5.29],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":808,"del":207,"ins":124},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":50.3,"ci95":[49.3,51.39],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7893,"del":1201,"ins":1016},"empty_hyp":0,"utts_over_50pct":419,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":46.49,"ci95":[45.2,47.9],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5591,"del":719,"ins":568},"empty_hyp":0,"utts_over_50pct":356,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":35.43,"ci95":[34.47,36.38],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7114,"del":1033,"ins":1216},"empty_hyp":0,"utts_over_50pct":161,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":7.76,"ci95":[7.14,8.43],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1013,"del":206,"ins":178},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":12.68,"ci95":[11.63,13.83],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1357,"del":737,"ins":284},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":56.24,"ci95":[54.94,57.67],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7508,"del":1386,"ins":709},"empty_hyp":1,"utts_over_50pct":547,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":8.49,"ci95":[7.85,9.21],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":894,"del":113,"ins":144},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":5.19,"ci95":[4.78,5.64],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":812,"del":179,"ins":112},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":8.61,"ci95":[7.98,9.28],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1929,"del":522,"ins":359},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":5.82,"ci95":[4.86,6.83],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":422,"del":553,"ins":54},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":35.09,"ci95":[34.14,36.11],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5844,"del":971,"ins":584},"empty_hyp":0,"utts_over_50pct":160,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":17.18,"ci95":[16.28,18.01],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2163,"del":160,"ins":211},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":13.9,"ci95":[12.88,14.93],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":891,"del":155,"ins":121},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":25.06,"ci95":[24.04,26.12],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2993,"del":363,"ins":250},"empty_hyp":0,"utts_over_50pct":52,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":6.57,"ci95":[6.05,7.12],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1016,"del":207,"ins":187},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":40.65,"ci95":[39.48,41.79],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6596,"del":1379,"ins":481},"empty_hyp":0,"utts_over_50pct":254,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":10.3,"ci95":[9.59,11.0],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1258,"del":185,"ins":113},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":35.72,"ci95":[34.53,36.94],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4292,"del":762,"ins":443},"empty_hyp":0,"utts_over_50pct":160,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":8.81,"ci95":[8.04,9.71],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3502,"del":2027,"ins":3058},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":16.74,"ci95":[15.89,17.74],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1778,"del":237,"ins":211},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":9.32,"ci95":[8.67,9.93],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1919,"del":256,"ins":249},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yue","quant":"Q8_0","metric":"cer","err_pct":7.91,"ci95":[7.28,8.52],"n_utts":819,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1836,"del":211,"ins":248},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":7.57,"ci95":[6.7,8.42],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":914,"del":1473,"ins":313},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":2.12,"ci95":[1.96,2.29],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":874,"del":161,"ins":91},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.12,"ci95":[1.96,2.28],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":870,"del":161,"ins":91},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.11,"ci95":[1.95,2.27],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":867,"del":161,"ins":90},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.11,"ci95":[1.95,2.27],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":862,"del":162,"ins":93},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.21,"ci95":[2.05,2.38],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":906,"del":169,"ins":96},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.26,"ci95":[2.08,2.44],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":931,"del":149,"ins":121},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1968.7,"xrt_compute":17.95,"load_ms":202.3,"mel_ms":5.9,"encode_ms":757.3,"decode_ms":1205.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1972.8,"xrt_wall":17.91,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":547.3,"xrt_compute":20.1,"load_ms":184.7,"mel_ms":1.9,"encode_ms":237.3,"decode_ms":308.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":549.0,"xrt_wall":20.03,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2062.9,"xrt_compute":17.13,"load_ms":239.6,"mel_ms":5.9,"encode_ms":753.7,"decode_ms":1303.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2066.9,"xrt_wall":17.09,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":544.6,"xrt_compute":20.2,"load_ms":254.9,"mel_ms":1.9,"encode_ms":234.5,"decode_ms":308.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":546.3,"xrt_wall":20.13,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":510.6,"xrt_compute":69.19,"load_ms":210.8,"mel_ms":7.6,"encode_ms":59.7,"decode_ms":443.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":527.0,"xrt_wall":67.04,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":134.5,"xrt_compute":81.78,"load_ms":197.1,"mel_ms":3.1,"encode_ms":22.8,"decode_ms":108.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":139.9,"xrt_wall":78.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":556.4,"xrt_compute":63.5,"load_ms":280.9,"mel_ms":10.5,"encode_ms":59.7,"decode_ms":486.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":574.2,"xrt_wall":61.54,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":146.6,"xrt_compute":75.02,"load_ms":251.9,"mel_ms":3.1,"encode_ms":22.3,"decode_ms":121.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":151.8,"xrt_wall":72.46,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":6749.8,"xrt_compute":5.23,"load_ms":850.6,"mel_ms":37.3,"encode_ms":2102.7,"decode_ms":4609.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6834.7,"xrt_wall":5.17,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1671.2,"xrt_compute":6.58,"load_ms":601.1,"mel_ms":18.6,"encode_ms":546.7,"decode_ms":1105.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1695.6,"xrt_wall":6.49,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7549.2,"xrt_compute":4.68,"load_ms":1275.7,"mel_ms":38.1,"encode_ms":2106.5,"decode_ms":5404.7,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7632.9,"xrt_wall":4.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1860.4,"xrt_compute":5.91,"load_ms":859.8,"mel_ms":18.4,"encode_ms":563.2,"decode_ms":1278.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1885.1,"xrt_wall":5.84,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3547.6,"xrt_compute":9.96,"load_ms":1140.6,"mel_ms":44.4,"encode_ms":782.9,"decode_ms":2720.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3567.1,"xrt_wall":9.91,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":907.6,"xrt_compute":12.12,"load_ms":1041.3,"mel_ms":20.5,"encode_ms":228.8,"decode_ms":658.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":913.7,"xrt_wall":12.04,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4071.9,"xrt_compute":8.68,"load_ms":1794.5,"mel_ms":47.9,"encode_ms":785.4,"decode_ms":3238.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4090.7,"xrt_wall":8.64,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1025.4,"xrt_compute":10.73,"load_ms":1652.6,"mel_ms":20.6,"encode_ms":230.0,"decode_ms":774.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1033.5,"xrt_wall":10.64,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/qwen3-asr-1.7b.json b/catalog/qwen3-asr-1.7b.json new file mode 100644 index 00000000..0ccd9c19 --- /dev/null +++ b/catalog/qwen3-asr-1.7b.json @@ -0,0 +1,102 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "qwen3-asr-1.7b", + "family": "qwen3_asr", + "display_name": "qwen3-asr-1.7b", + "params": 2038078608, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "Qwen/Qwen3-ASR-1.7B", + "upstream_commit": "7278e1e", + "published_repo": "handy-computer/Qwen3-ASR-1.7B-gguf", + "docs_page": "qwen3-asr.md", + "languages": [ + "zh", "en", "yue", "ar", "de", "fr", "es", "pt", "id", "it", "ko", "ru", + "th", "vi", "ja", "tr", "hi", "ms", "nl", "sv", "da", "fi", "pl", "cs", + "fil", "fa", "el", "ro", "hu", "mk" + ], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Qwen3-ASR-1.7B-BF16.gguf","size_bytes":4083087904}, + {"quant":"F16","filename":"Qwen3-ASR-1.7B-F16.gguf","size_bytes":4091390944}, + {"quant":"Q8_0","filename":"Qwen3-ASR-1.7B-Q8_0.gguf","size_bytes":2185030624}, + {"quant":"Q6_K","filename":"Qwen3-ASR-1.7B-Q6_K.gguf","size_bytes":1692554208}, + {"quant":"Q5_K_M","filename":"Qwen3-ASR-1.7B-Q5_K_M.gguf","size_bytes":1517290464}, + {"quant":"Q4_K_M","filename":"Qwen3-ASR-1.7B-Q4_K_M.gguf","size_bytes":1319830496} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":14.91,"ci95":[13.53,16.42],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":855,"del":305,"ins":68},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":22.97,"ci95":[22.0,24.09],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2591,"del":339,"ins":215},"empty_hyp":0,"utts_over_50pct":37,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":21.23,"ci95":[20.4,22.14],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3367,"del":447,"ins":466},"empty_hyp":0,"utts_over_50pct":35,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":4.25,"ci95":[3.86,4.64],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":593,"del":140,"ins":62},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":29.22,"ci95":[27.87,30.64],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3528,"del":690,"ins":215},"empty_hyp":0,"utts_over_50pct":71,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.23,"ci95":[2.85,3.65],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":283,"del":122,"ins":65},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.31,"ci95":[2.98,3.65],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":508,"del":190,"ins":75},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":28.29,"ci95":[27.34,29.31],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4297,"del":564,"ins":826},"empty_hyp":0,"utts_over_50pct":58,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":25.48,"ci95":[24.58,26.47],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3095,"del":391,"ins":284},"empty_hyp":0,"utts_over_50pct":58,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":24.29,"ci95":[23.49,25.18],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4757,"del":618,"ins":1043},"empty_hyp":0,"utts_over_50pct":40,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":4.52,"ci95":[4.06,5.01],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":579,"del":166,"ins":69},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":7.84,"ci95":[7.09,8.7],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":807,"del":422,"ins":241},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":32.84,"ci95":[31.69,33.99],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4484,"del":667,"ins":457},"empty_hyp":0,"utts_over_50pct":134,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":5.37,"ci95":[4.85,5.93],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":543,"del":89,"ins":96},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":2.68,"ci95":[2.38,2.99],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":402,"del":113,"ins":54},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":5.29,"ci95":[4.81,5.8],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1162,"del":321,"ins":245},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":4.6,"ci95":[3.62,5.65],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":236,"del":542,"ins":36},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":18.22,"ci95":[17.43,19.04],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3043,"del":435,"ins":364},"empty_hyp":0,"utts_over_50pct":23,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":10.42,"ci95":[9.72,11.11],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1292,"del":104,"ins":141},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":7.43,"ci95":[6.73,8.15],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":456,"del":96,"ins":72},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":12.5,"ci95":[11.66,13.28],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1458,"del":222,"ins":118},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":4.37,"ci95":[3.95,4.83],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":651,"del":149,"ins":139},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":20.46,"ci95":[19.57,21.43],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3345,"del":537,"ins":374},"empty_hyp":0,"utts_over_50pct":36,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":6.25,"ci95":[5.74,6.79],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":764,"del":125,"ins":55},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":19.68,"ci95":[18.7,20.62],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2304,"del":472,"ins":252},"empty_hyp":0,"utts_over_50pct":31,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":6.89,"ci95":[6.15,7.73],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2488,"del":1506,"ins":2725},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":9.46,"ci95":[8.72,10.22],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":952,"del":186,"ins":120},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":6.15,"ci95":[5.68,6.61],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1220,"del":190,"ins":189},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yue","quant":"Q8_0","metric":"cer","err_pct":6.13,"ci95":[5.55,6.68],"n_utts":819,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1381,"del":169,"ins":229},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":7.14,"ci95":[6.26,8.12],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":776,"del":1462,"ins":306},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.62,"ci95":[1.47,1.76],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":649,"del":143,"ins":65},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.62,"ci95":[1.48,1.77],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":651,"del":143,"ins":65},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.62,"ci95":[1.47,1.76],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":649,"del":145,"ins":64},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.65,"ci95":[1.51,1.8],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":660,"del":148,"ins":68},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.65,"ci95":[1.51,1.81],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":663,"del":141,"ins":73},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.81,"ci95":[1.66,1.98],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":729,"del":144,"ins":88},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":3796.9,"xrt_compute":9.31,"load_ms":389.8,"mel_ms":6.2,"encode_ms":963.4,"decode_ms":2827.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3801.5,"xrt_wall":9.29,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1227.6,"xrt_compute":8.96,"load_ms":354.0,"mel_ms":2.0,"encode_ms":385.9,"decode_ms":839.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1229.5,"xrt_wall":8.95,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4003.6,"xrt_compute":8.83,"load_ms":589.9,"mel_ms":7.0,"encode_ms":929.2,"decode_ms":3067.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4007.8,"xrt_wall":8.82,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1063.0,"xrt_compute":10.35,"load_ms":523.1,"mel_ms":1.9,"encode_ms":292.4,"decode_ms":768.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1064.8,"xrt_wall":10.33,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":804.0,"xrt_compute":43.95,"load_ms":396.9,"mel_ms":13.8,"encode_ms":75.3,"decode_ms":714.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":824.7,"xrt_wall":42.84,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":208.9,"xrt_compute":52.65,"load_ms":362.7,"mel_ms":3.1,"encode_ms":27.7,"decode_ms":178.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":214.3,"xrt_wall":51.34,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":959.4,"xrt_compute":36.83,"load_ms":619.1,"mel_ms":8.2,"encode_ms":73.9,"decode_ms":877.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":977.1,"xrt_wall":36.16,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":242.6,"xrt_compute":45.33,"load_ms":566.4,"mel_ms":3.1,"encode_ms":27.4,"decode_ms":212.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":248.3,"xrt_wall":44.31,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":13071.6,"xrt_compute":2.7,"load_ms":1994.5,"mel_ms":37.2,"encode_ms":2714.5,"decode_ms":10319.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13152.5,"xrt_wall":2.69,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3508.3,"xrt_compute":3.14,"load_ms":1350.8,"mel_ms":15.7,"encode_ms":761.3,"decode_ms":2731.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3537.6,"xrt_wall":3.11,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":15628.0,"xrt_compute":2.26,"load_ms":3294.1,"mel_ms":38.2,"encode_ms":2680.9,"decode_ms":12908.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15709.9,"xrt_wall":2.25,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":4102.3,"xrt_compute":2.68,"load_ms":3317.5,"mel_ms":15.2,"encode_ms":741.0,"decode_ms":3346.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4132.1,"xrt_wall":2.66,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":7891.9,"xrt_compute":4.48,"load_ms":3039.6,"mel_ms":43.0,"encode_ms":1170.7,"decode_ms":6678.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7977.0,"xrt_wall":4.43,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2019.2,"xrt_compute":5.45,"load_ms":3090.8,"mel_ms":25.3,"encode_ms":334.5,"decode_ms":1659.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2043.7,"xrt_wall":5.38,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":9490.0,"xrt_compute":3.72,"load_ms":3366.4,"mel_ms":43.1,"encode_ms":1154.9,"decode_ms":8292.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":9588.9,"xrt_wall":3.68,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2473.2,"xrt_compute":4.45,"load_ms":3370.3,"mel_ms":25.0,"encode_ms":336.8,"decode_ms":2111.3,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2501.4,"xrt_wall":4.4,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/sensevoice-small.json b/catalog/sensevoice-small.json new file mode 100644 index 00000000..6fac8436 --- /dev/null +++ b/catalog/sensevoice-small.json @@ -0,0 +1,75 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "sensevoice-small", + "family": "sensevoice", + "display_name": "sensevoice-small", + "params": 234000287, + "license": { + "spdx": "other", + "display": "model-license (FunASR MODEL_LICENSE)", + "name": "model-license", + "link": "https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE" + }, + "upstream_repo": "FunAudioLLM/SenseVoiceSmall", + "upstream_commit": "3eb3b4e", + "published_repo": "handy-computer/SenseVoiceSmall-gguf", + "docs_page": "sensevoice-small.md", + "languages": ["zh","yue","en","ja","ko"], + "long_form_strategy": "soft-window", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"SenseVoiceSmall-F32.gguf","size_bytes":936617824}, + {"quant":"F16","filename":"SenseVoiceSmall-F16.gguf","size_bytes":470412128}, + {"quant":"Q8_0","filename":"SenseVoiceSmall-Q8_0.gguf","size_bytes":252684608}, + {"quant":"Q6_K","filename":"SenseVoiceSmall-Q6_K.gguf","size_bytes":196438336}, + {"quant":"Q5_K_M","filename":"SenseVoiceSmall-Q5_K_M.gguf","size_bytes":172474880}, + {"quant":"Q4_K_M","filename":"SenseVoiceSmall-Q4_K_M.gguf","size_bytes":145738304} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":7.14,"ci95":[6.54,7.77],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":763,"del":117,"ins":159},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":7.63,"ci95":[7.1,8.22],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1665,"del":558,"ins":268},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":8.27,"ci95":[7.13,9.45],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":642,"del":667,"ins":153},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yue","quant":"Q8_0","metric":"cer","err_pct":37.44,"ci95":[36.69,38.21],"n_utts":819,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10304,"del":315,"ins":239},"empty_hyp":0,"utts_over_50pct":83,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":10.12,"ci95":[9.16,11.08],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1716,"del":1570,"ins":323},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":3.13,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":3.13,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.13,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":3.14,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":3.18,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":3.45,"ci95":[null,null],"n_utts":2620,"batch_size":null,"timestamps":null,"engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": null, + "timestamps": null + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":756.7,"xrt_compute":46.69,"load_ms":70.0,"mel_ms":38.6,"encode_ms":702.5,"decode_ms":15.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":759.6,"xrt_wall":46.51,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":220.4,"xrt_compute":49.92,"load_ms":65.1,"mel_ms":12.1,"encode_ms":203.1,"decode_ms":5.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":221.8,"xrt_wall":49.59,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":737.3,"xrt_compute":47.92,"load_ms":97.8,"mel_ms":38.9,"encode_ms":682.4,"decode_ms":15.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":740.3,"xrt_wall":47.73,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":215.1,"xrt_compute":51.15,"load_ms":86.3,"mel_ms":12.0,"encode_ms":198.0,"decode_ms":5.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":216.5,"xrt_wall":50.82,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":123.2,"xrt_compute":286.71,"load_ms":70.9,"mel_ms":37.7,"encode_ms":70.4,"decode_ms":15.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":130.7,"xrt_wall":270.36,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":46.1,"xrt_compute":238.72,"load_ms":68.0,"mel_ms":11.6,"encode_ms":29.8,"decode_ms":4.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":49.3,"xrt_wall":223.07,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":122.4,"xrt_compute":288.69,"load_ms":96.2,"mel_ms":37.2,"encode_ms":70.1,"decode_ms":15.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":129.8,"xrt_wall":272.14,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":44.8,"xrt_compute":245.28,"load_ms":89.3,"mel_ms":11.5,"encode_ms":28.5,"decode_ms":4.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":48.0,"xrt_wall":229.32,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2154.6,"xrt_compute":16.4,"load_ms":200.8,"mel_ms":75.4,"encode_ms":2065.8,"decode_ms":13.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2173.9,"xrt_wall":16.25,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":626.6,"xrt_compute":17.55,"load_ms":133.7,"mel_ms":23.3,"encode_ms":599.8,"decode_ms":3.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":632.8,"xrt_wall":17.38,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1883.6,"xrt_compute":18.76,"load_ms":313.5,"mel_ms":74.9,"encode_ms":1796.3,"decode_ms":12.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1903.3,"xrt_wall":18.56,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":531.1,"xrt_compute":20.71,"load_ms":196.5,"mel_ms":23.4,"encode_ms":504.2,"decode_ms":3.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":537.3,"xrt_wall":20.47,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1295.7,"xrt_compute":27.27,"load_ms":151.2,"mel_ms":74.1,"encode_ms":1211.1,"decode_ms":10.5,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1641.1,"xrt_wall":21.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":398.2,"xrt_compute":27.63,"load_ms":92.5,"mel_ms":23.0,"encode_ms":371.8,"decode_ms":3.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":515.2,"xrt_wall":21.35,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1283.5,"xrt_compute":27.53,"load_ms":246.5,"mel_ms":74.3,"encode_ms":1198.3,"decode_ms":10.8,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1629.9,"xrt_wall":21.68,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":395.0,"xrt_compute":27.85,"load_ms":121.5,"mel_ms":22.9,"encode_ms":368.7,"decode_ms":3.4,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":512.3,"xrt_wall":21.47,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/voxtral-mini-3b-2507.json b/catalog/voxtral-mini-3b-2507.json new file mode 100644 index 00000000..80547a78 --- /dev/null +++ b/catalog/voxtral-mini-3b-2507.json @@ -0,0 +1,76 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "voxtral-mini-3b-2507", + "family": "voxtral", + "display_name": "voxtral-mini-3b-2507", + "params": 4676297232, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "mistralai/Voxtral-Mini-3B-2507", + "upstream_commit": "3060fe3", + "published_repo": "handy-computer/Voxtral-Mini-3B-2507-gguf", + "docs_page": "voxtral.md", + "languages": ["en","fr","de","es","it","pt","nl","hi"], + "long_form_strategy": "hard-cap", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en","fr","de","es","it","pt","nl","hi"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Voxtral-Mini-3B-2507-BF16.gguf","size_bytes":9365764768}, + {"quant":"F16","filename":"Voxtral-Mini-3B-2507-F16.gguf","size_bytes":9376578208}, + {"quant":"Q8_0","filename":"Voxtral-Mini-3B-2507-Q8_0.gguf","size_bytes":5000084128}, + {"quant":"Q6_K","filename":"Voxtral-Mini-3B-2507-Q6_K.gguf","size_bytes":3869489824}, + {"quant":"Q5_K_M","filename":"Voxtral-Mini-3B-2507-Q5_K_M.gguf","size_bytes":3464182432}, + {"quant":"Q4_K_M","filename":"Voxtral-Mini-3B-2507-Q4_K_M.gguf","size_bytes":2984721056} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":4.71,"ci95":[4.26,5.21],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":585,"del":203,"ins":93},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.89,"ci95":[3.47,4.35],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":344,"del":137,"ins":85},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.52,"ci95":[3.11,3.95],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":472,"del":268,"ins":82},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":4.51,"ci95":[4.04,5.02],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":581,"del":128,"ins":103},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":8.93,"ci95":[8.13,9.91],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":954,"del":458,"ins":263},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":2.56,"ci95":[2.24,2.9],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":352,"del":149,"ins":42},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":6.57,"ci95":[5.84,7.32],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":378,"del":116,"ins":58},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":3.84,"ci95":[3.44,4.31],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":513,"del":164,"ins":147},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.88,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.89,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.87,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.87,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.91,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.94,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 8, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":14183.7,"xrt_compute":2.49,"load_ms":773.9,"mel_ms":10.7,"encode_ms":6035.5,"decode_ms":8137.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14184.9,"xrt_wall":2.49,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":6118.8,"xrt_compute":1.8,"load_ms":703.0,"mel_ms":5.4,"encode_ms":3018.1,"decode_ms":3095.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6119.6,"xrt_wall":1.8,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":14941.4,"xrt_compute":2.36,"load_ms":1256.8,"mel_ms":11.0,"encode_ms":5637.6,"decode_ms":9292.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14942.6,"xrt_wall":2.36,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":5834.8,"xrt_compute":1.89,"load_ms":1150.0,"mel_ms":5.3,"encode_ms":2830.1,"decode_ms":2999.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5835.5,"xrt_wall":1.89,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2091.0,"xrt_compute":16.9,"load_ms":825.7,"mel_ms":12.6,"encode_ms":584.7,"decode_ms":1493.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2091.6,"xrt_wall":16.89,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":769.4,"xrt_compute":14.3,"load_ms":720.7,"mel_ms":6.3,"encode_ms":290.4,"decode_ms":472.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":769.9,"xrt_wall":14.29,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2509.5,"xrt_compute":14.08,"load_ms":1323.5,"mel_ms":12.1,"encode_ms":629.7,"decode_ms":1867.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2510.1,"xrt_wall":14.08,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":862.3,"xrt_compute":12.76,"load_ms":1187.7,"mel_ms":9.2,"encode_ms":295.3,"decode_ms":557.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":862.9,"xrt_wall":12.75,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":49489.9,"xrt_compute":0.71,"load_ms":4317.4,"mel_ms":57.5,"encode_ms":19202.3,"decode_ms":30230.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":49509.5,"xrt_wall":0.71,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":20878.8,"xrt_compute":0.53,"load_ms":4367.0,"mel_ms":33.8,"encode_ms":9595.2,"decode_ms":11249.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":20889.0,"xrt_wall":0.53,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":55993.4,"xrt_compute":0.63,"load_ms":7735.1,"mel_ms":58.1,"encode_ms":19719.7,"decode_ms":36215.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":56012.8,"xrt_wall":0.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":22372.2,"xrt_compute":0.49,"load_ms":8286.0,"mel_ms":29.9,"encode_ms":9867.1,"decode_ms":12475.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":22382.7,"xrt_wall":0.49,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":27463.8,"xrt_compute":1.29,"load_ms":4330.3,"mel_ms":71.4,"encode_ms":9116.5,"decode_ms":18275.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":27469.2,"xrt_wall":1.29,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":10583.1,"xrt_compute":1.04,"load_ms":5062.7,"mel_ms":48.1,"encode_ms":4536.8,"decode_ms":5998.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":10585.0,"xrt_wall":1.04,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":30044.3,"xrt_compute":1.18,"load_ms":6869.2,"mel_ms":78.4,"encode_ms":8950.2,"decode_ms":21015.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":30049.7,"xrt_wall":1.18,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":11111.9,"xrt_compute":0.99,"load_ms":6923.9,"mel_ms":52.0,"encode_ms":4440.6,"decode_ms":6619.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":11113.9,"xrt_wall":0.99,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/voxtral-mini-4b-realtime-2602.json b/catalog/voxtral-mini-4b-realtime-2602.json new file mode 100644 index 00000000..d5cde426 --- /dev/null +++ b/catalog/voxtral-mini-4b-realtime-2602.json @@ -0,0 +1,81 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "voxtral-mini-4b-realtime-2602", + "family": "voxtral_realtime", + "display_name": "voxtral-mini-4b-realtime-2602", + "params": 4429707024, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "mistralai/Voxtral-Mini-4B-Realtime-2602", + "upstream_commit": "2769294", + "published_repo": "handy-computer/Voxtral-Mini-4B-Realtime-2602-gguf", + "docs_page": "voxtral-realtime.md", + "languages": ["en","fr","es","de","ru","zh","ja","it","pt","nl","ar","hi","ko"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":true,"verified":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Voxtral-Mini-4B-Realtime-2602-BF16.gguf","size_bytes":8868301088}, + {"quant":"F16","filename":"Voxtral-Mini-4B-Realtime-2602-F16.gguf","size_bytes":8879114528}, + {"quant":"Q8_0","filename":"Voxtral-Mini-4B-Realtime-2602-Q8_0.gguf","size_bytes":4731791648}, + {"quant":"Q6_K","filename":"Voxtral-Mini-4B-Realtime-2602-Q6_K.gguf","size_bytes":3661018912}, + {"quant":"Q5_K_M","filename":"Voxtral-Mini-4B-Realtime-2602-Q5_K_M.gguf","size_bytes":3281439008}, + {"quant":"Q4_K_M","filename":"Voxtral-Mini-4B-Realtime-2602-Q4_K_M.gguf","size_bytes":2830493984} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":14.53,"ci95":[13.17,16.13],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":830,"del":322,"ins":45},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":4.24,"ci95":[3.78,4.73],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":577,"del":127,"ins":90},"empty_hyp":1,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":6.27,"ci95":[5.38,7.31],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":519,"del":295,"ins":99},"empty_hyp":6,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":2.67,"ci95":[2.38,2.96],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":454,"del":93,"ins":75},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":6.29,"ci95":[5.67,6.88],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":827,"del":184,"ins":121},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":14.44,"ci95":[13.36,15.68],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1625,"del":793,"ins":290},"empty_hyp":3,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":2.25,"ci95":[1.99,2.53],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":377,"del":52,"ins":50},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":5.44,"ci95":[4.94,5.95],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1227,"del":365,"ins":184},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":5.27,"ci95":[4.3,6.31],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":355,"del":532,"ins":45},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":6.09,"ci95":[5.45,6.73],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":380,"del":63,"ins":68},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":3.87,"ci95":[3.43,4.38],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":572,"del":91,"ins":168},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":5.41,"ci95":[4.95,5.88],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":698,"del":70,"ins":49},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":10.41,"ci95":[9.32,11.52],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1752,"del":1609,"ins":350},"empty_hyp":2,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":2.08,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.09,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.07,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.08,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.08,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.08,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 8, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":13204.9,"xrt_compute":2.68,"load_ms":747.6,"mel_ms":7.7,"encode_ms":5642.3,"decode_ms":7554.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13202.3,"xrt_wall":2.68,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":5040.9,"xrt_compute":2.18,"load_ms":715.4,"mel_ms":3.2,"encode_ms":2349.7,"decode_ms":2688.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5041.4,"xrt_wall":2.18,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":13796.7,"xrt_compute":2.56,"load_ms":1280.9,"mel_ms":8.1,"encode_ms":5326.8,"decode_ms":8461.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13798.0,"xrt_wall":2.56,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":4878.1,"xrt_compute":2.25,"load_ms":1149.8,"mel_ms":3.3,"encode_ms":2100.8,"decode_ms":2774.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4878.8,"xrt_wall":2.25,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4435.8,"xrt_compute":7.97,"load_ms":781.9,"mel_ms":10.6,"encode_ms":654.6,"decode_ms":3770.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4436.9,"xrt_wall":7.96,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1494.8,"xrt_compute":7.36,"load_ms":697.4,"mel_ms":4.5,"encode_ms":255.7,"decode_ms":1234.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1495.5,"xrt_wall":7.36,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":5025.4,"xrt_compute":7.03,"load_ms":1268.7,"mel_ms":13.1,"encode_ms":765.6,"decode_ms":4246.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5026.8,"xrt_wall":7.03,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1763.2,"xrt_compute":6.24,"load_ms":1116.5,"mel_ms":4.5,"encode_ms":293.4,"decode_ms":1465.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1763.7,"xrt_wall":6.24,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":46119.5,"xrt_compute":0.77,"load_ms":2777.3,"mel_ms":51.8,"encode_ms":19170.5,"decode_ms":26897.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":46139.2,"xrt_wall":0.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":16390.6,"xrt_compute":0.67,"load_ms":4170.6,"mel_ms":20.7,"encode_ms":7306.5,"decode_ms":9063.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":16399.7,"xrt_wall":0.67,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":57916.3,"xrt_compute":0.61,"load_ms":7061.6,"mel_ms":43.6,"encode_ms":19739.9,"decode_ms":38132.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":57934.8,"xrt_wall":0.61,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":19763.4,"xrt_compute":0.56,"load_ms":7214.9,"mel_ms":23.1,"encode_ms":7202.2,"decode_ms":12538.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":19771.6,"xrt_wall":0.56,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":39160.0,"xrt_compute":0.9,"load_ms":3558.6,"mel_ms":61.2,"encode_ms":8840.2,"decode_ms":30258.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":39162.2,"xrt_wall":0.9,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":13171.6,"xrt_compute":0.84,"load_ms":5569.3,"mel_ms":37.2,"encode_ms":3376.6,"decode_ms":9757.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13173.1,"xrt_wall":0.84,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":45034.6,"xrt_compute":0.78,"load_ms":6929.3,"mel_ms":54.3,"encode_ms":9089.3,"decode_ms":35891.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":45037.7,"xrt_wall":0.78,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":14951.4,"xrt_compute":0.74,"load_ms":6986.9,"mel_ms":32.2,"encode_ms":3424.5,"decode_ms":11494.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14952.9,"xrt_wall":0.74,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/voxtral-small-24b-2507.json b/catalog/voxtral-small-24b-2507.json new file mode 100644 index 00000000..56fc0b4d --- /dev/null +++ b/catalog/voxtral-small-24b-2507.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "voxtral-small-24b-2507", + "family": "voxtral", + "display_name": "voxtral-small-24b-2507", + "params": 24261827088, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "mistralai/Voxtral-Small-24B-2507", + "upstream_commit": "da5b424", + "published_repo": "handy-computer/Voxtral-Small-24B-2507-gguf", + "docs_page": "voxtral.md", + "languages": ["en","fr","de","es","it","pt","nl","hi"], + "long_form_strategy": "hard-cap", + "benchmark_exceptions": [ + {"kind":"speed","match":{"machine":"m4-max","backend":"cpu"},"reason":"The model card explicitly scopes this 24B GPU-class model to Metal; CPU execution is impractical at its 14.3-25.8 GB publication-quant footprint."}, + {"kind":"speed","match":{"machine":"ryzen-4750u","backend":"cpu"},"reason":"The model card explicitly omits CPU for this 24B GPU-class model; the publication quants are 14.3 GB and 25.8 GB before runtime state."}, + {"kind":"speed","match":{"machine":"ryzen-4750u","backend":"vulkan"},"reason":"The smallest downloadable quant is 14.3 GB, exceeding the reference Ryzen 4750U Vulkan device-memory budget."} + ], + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en","fr","de","es","it","pt","nl","hi"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"BF16","filename":"Voxtral-Small-24B-2507-BF16.gguf","size_bytes":48537285088}, + {"quant":"F16","filename":"Voxtral-Small-24B-2507-F16.gguf","size_bytes":48548098528}, + {"quant":"Q8_0","filename":"Voxtral-Small-24B-2507-Q8_0.gguf","size_bytes":25810383328}, + {"quant":"Q6_K","filename":"Voxtral-Small-24B-2507-Q6_K.gguf","size_bytes":19936473568}, + {"quant":"Q5_K_M","filename":"Voxtral-Small-24B-2507-Q5_K_M.gguf","size_bytes":17138659808}, + {"quant":"Q4_K_M","filename":"Voxtral-Small-24B-2507-Q4_K_M.gguf","size_bytes":14302261728} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":3.29,"ci95":[2.93,3.65],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":361,"del":199,"ins":55},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.55,"ci95":[3.07,4.1],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":259,"del":196,"ins":62},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":2.86,"ci95":[2.47,3.25],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":338,"del":274,"ins":54},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":3.86,"ci95":[3.39,4.31],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":433,"del":192,"ins":69},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":7.4,"ci95":[6.65,8.36],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":759,"del":408,"ins":220},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":2.69,"ci95":[2.19,3.34],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":268,"del":273,"ins":31},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":5.12,"ci95":[4.38,5.96],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":236,"del":144,"ins":50},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":3.74,"ci95":[3.25,4.26],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":401,"del":308,"ins":95},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"BF16","metric":"wer","err_pct":1.56,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.57,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.56,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.58,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.6,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.11,"ci95":[null,null],"n_utts":2620,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 8, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":12934.7,"xrt_compute":2.73,"load_ms":4433.2,"mel_ms":11.3,"encode_ms":1045.2,"decode_ms":11878.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":12935.5,"xrt_wall":2.73,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3964.6,"xrt_compute":2.77,"load_ms":3598.9,"mel_ms":5.0,"encode_ms":401.1,"decode_ms":3558.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3965.1,"xrt_wall":2.77,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":14680.8,"xrt_compute":2.41,"load_ms":7038.4,"mel_ms":12.4,"encode_ms":1068.0,"decode_ms":13600.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14681.8,"xrt_wall":2.41,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":4445.6,"xrt_compute":2.47,"load_ms":7405.4,"mel_ms":5.0,"encode_ms":491.5,"decode_ms":3949.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4446.1,"xrt_wall":2.47,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-base.en.json b/catalog/whisper-base.en.json new file mode 100644 index 00000000..40b47256 --- /dev/null +++ b/catalog/whisper-base.en.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-base.en", + "family": "whisper", + "display_name": "whisper-base.en", + "params": 72609888, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-base.en", + "upstream_commit": "911407f", + "published_repo": "handy-computer/whisper-base.en-gguf", + "docs_page": "whisper.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["segment"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-base.en-F32.gguf","size_bytes":292257728}, + {"quant":"F16","filename":"whisper-base.en-F16.gguf","size_bytes":151068608}, + {"quant":"Q8_0","filename":"whisper-base.en-Q8_0.gguf","size_bytes":84886208}, + {"quant":"Q6_K","filename":"whisper-base.en-Q6_K.gguf","size_bytes":67789088}, + {"quant":"Q5_K_M","filename":"whisper-base.en-Q5_K_M.gguf","size_bytes":63709472}, + {"quant":"Q4_K_M","filename":"whisper-base.en-Q4_K_M.gguf","size_bytes":58794272} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":7.6,"ci95":[6.89,8.32],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":761,"del":165,"ins":180},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":4.3,"ci95":[3.95,4.74],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1622,"del":287,"ins":370},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":4.13,"ci95":[3.89,4.4],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1617,"del":287,"ins":286},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.16,"ci95":[3.92,4.43],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1631,"del":293,"ins":283},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":4.15,"ci95":[3.9,4.4],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1630,"del":287,"ins":282},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":4.16,"ci95":[3.92,4.42],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1641,"del":276,"ins":289},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":4.29,"ci95":[4.05,4.55],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1695,"del":280,"ins":300},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":463.9,"xrt_compute":76.16,"load_ms":56.8,"mel_ms":4.8,"encode_ms":145.6,"decode_ms":313.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":464.2,"xrt_wall":76.12,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":195.3,"xrt_compute":56.31,"load_ms":53.7,"mel_ms":4.1,"encode_ms":146.3,"decode_ms":44.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":195.6,"xrt_wall":56.24,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":423.6,"xrt_compute":83.4,"load_ms":62.8,"mel_ms":4.8,"encode_ms":130.1,"decode_ms":288.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":423.9,"xrt_wall":83.36,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":173.6,"xrt_compute":63.35,"load_ms":58.5,"mel_ms":4.2,"encode_ms":129.9,"decode_ms":39.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":173.9,"xrt_wall":63.26,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":181.2,"xrt_compute":194.94,"load_ms":64.8,"mel_ms":4.8,"encode_ms":15.3,"decode_ms":161.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":181.4,"xrt_wall":194.81,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":51.8,"xrt_compute":212.42,"load_ms":52.3,"mel_ms":4.1,"encode_ms":15.3,"decode_ms":32.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":51.9,"xrt_wall":212.01,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":185.2,"xrt_compute":190.73,"load_ms":69.6,"mel_ms":4.8,"encode_ms":14.5,"decode_ms":165.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":185.4,"xrt_wall":190.59,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":51.8,"xrt_compute":212.45,"load_ms":62.5,"mel_ms":4.2,"encode_ms":14.9,"decode_ms":32.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":51.9,"xrt_wall":212.04,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1497.4,"xrt_compute":23.6,"load_ms":110.7,"mel_ms":35.6,"encode_ms":378.2,"decode_ms":1083.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1497.6,"xrt_wall":23.59,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":574.8,"xrt_compute":19.14,"load_ms":134.4,"mel_ms":30.0,"encode_ms":375.9,"decode_ms":169.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":575.0,"xrt_wall":19.13,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1568.9,"xrt_compute":22.52,"load_ms":128.8,"mel_ms":32.3,"encode_ms":385.8,"decode_ms":1150.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1569.0,"xrt_wall":22.52,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":588.7,"xrt_compute":18.68,"load_ms":164.9,"mel_ms":31.8,"encode_ms":383.9,"decode_ms":173.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":588.9,"xrt_wall":18.68,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1058.3,"xrt_compute":33.38,"load_ms":90.1,"mel_ms":37.3,"encode_ms":181.0,"decode_ms":840.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1058.7,"xrt_wall":33.37,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":385.1,"xrt_compute":28.56,"load_ms":111.8,"mel_ms":31.8,"encode_ms":180.4,"decode_ms":172.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":385.4,"xrt_wall":28.54,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1015.0,"xrt_compute":34.81,"load_ms":104.8,"mel_ms":36.5,"encode_ms":176.6,"decode_ms":801.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1015.3,"xrt_wall":34.8,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":376.5,"xrt_compute":29.22,"load_ms":144.0,"mel_ms":32.2,"encode_ms":176.9,"decode_ms":167.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":376.8,"xrt_wall":29.2,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-base.json b/catalog/whisper-base.json new file mode 100644 index 00000000..1b2553d4 --- /dev/null +++ b/catalog/whisper-base.json @@ -0,0 +1,160 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-base", + "family": "whisper", + "display_name": "whisper-base", + "params": 72610400, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-base", + "upstream_commit": "e37978b", + "published_repo": "handy-computer/whisper-base-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "zh" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-base-F32.gguf","size_bytes":292335904}, + {"quant":"F16","filename":"whisper-base-F16.gguf","size_bytes":151145760}, + {"quant":"Q8_0","filename":"whisper-base-Q8_0.gguf","size_bytes":84962880}, + {"quant":"Q6_K","filename":"whisper-base-Q6_K.gguf","size_bytes":67865664}, + {"quant":"Q5_K_M","filename":"whisper-base-Q5_K_M.gguf","size_bytes":63786048}, + {"quant":"Q4_K_M","filename":"whisper-base-Q4_K_M.gguf","size_bytes":58870848} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":83.05,"ci95":[80.55,86.01],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3815,"del":657,"ins":388},"empty_hyp":0,"utts_over_50pct":252,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":150.97,"ci95":[144.02,159.59],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6390,"del":2173,"ins":4366},"empty_hyp":46,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":52.74,"ci95":[50.65,55.22],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3675,"del":384,"ins":285},"empty_hyp":0,"utts_over_50pct":217,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":100.6,"ci95":[100.3,100.97],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17921,"del":29061,"ins":294},"empty_hyp":13,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":81.22,"ci95":[79.97,82.51],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12129,"del":1412,"ins":1017},"empty_hyp":0,"utts_over_50pct":899,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":92.72,"ci95":[92.02,93.59],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17118,"del":957,"ins":1234},"empty_hyp":0,"utts_over_50pct":967,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":70.53,"ci95":[69.06,71.99],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7792,"del":1006,"ins":1197},"empty_hyp":0,"utts_over_50pct":577,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":100.73,"ci95":[100.32,101.27],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17889,"del":26855,"ins":356},"empty_hyp":2,"utts_over_50pct":920,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":71.53,"ci95":[70.38,72.69],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10784,"del":1212,"ins":1471},"empty_hyp":0,"utts_over_50pct":813,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":29.48,"ci95":[28.52,30.38],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5319,"del":777,"ins":1047},"empty_hyp":1,"utts_over_50pct":88,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":70.14,"ci95":[68.85,71.39],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7820,"del":755,"ins":1029},"empty_hyp":0,"utts_over_50pct":629,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":98.19,"ci95":[96.7,99.94],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":18119,"del":2476,"ins":5135},"empty_hyp":2,"utts_over_50pct":1011,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":63.85,"ci95":[61.94,66.27],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9467,"del":964,"ins":2440},"empty_hyp":1,"utts_over_50pct":676,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":19.69,"ci95":[18.79,20.63],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2750,"del":377,"ins":558},"empty_hyp":0,"utts_over_50pct":23,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":59.14,"ci95":[57.67,60.72],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7225,"del":1026,"ins":722},"empty_hyp":0,"utts_over_50pct":438,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":9.88,"ci95":[9.07,10.7],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1000,"del":171,"ins":267},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":11.15,"ci95":[10.58,11.77],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1946,"del":291,"ins":364},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":81.71,"ci95":[80.47,82.88],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9675,"del":948,"ins":1536},"empty_hyp":0,"utts_over_50pct":867,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":87.72,"ci95":[86.66,88.77],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14266,"del":1375,"ins":1991},"empty_hyp":2,"utts_over_50pct":865,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":49.46,"ci95":[48.18,50.94],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5620,"del":659,"ins":1038},"empty_hyp":0,"utts_over_50pct":419,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":49.32,"ci95":[48.08,50.51],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9679,"del":1094,"ins":2260},"empty_hyp":0,"utts_over_50pct":420,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":27.91,"ci95":[26.68,29.23],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3633,"del":478,"ins":912},"empty_hyp":0,"utts_over_50pct":69,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":50.06,"ci95":[49.09,51.03],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8950,"del":1210,"ins":944},"empty_hyp":0,"utts_over_50pct":425,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":100.4,"ci95":[100.16,100.74],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":20899,"del":24028,"ins":244},"empty_hyp":2,"utts_over_50pct":1000,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":108.15,"ci95":[105.69,110.64],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9957,"del":5196,"ins":2080},"empty_hyp":13,"utts_over_50pct":621,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":65.56,"ci95":[63.95,67.11],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7450,"del":984,"ins":641},"empty_hyp":1,"utts_over_50pct":567,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":100.01,"ci95":[99.26,101.08],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8934,"del":9692,"ins":131},"empty_hyp":0,"utts_over_50pct":418,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":64.23,"ci95":[62.91,65.45],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9190,"del":983,"ins":1229},"empty_hyp":0,"utts_over_50pct":698,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":72.26,"ci95":[70.67,74.05],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9172,"del":1377,"ins":1790},"empty_hyp":2,"utts_over_50pct":776,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":127.56,"ci95":[124.05,131.34],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14135,"del":4020,"ins":5013},"empty_hyp":7,"utts_over_50pct":932,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":38.02,"ci95":[36.34,39.83],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4082,"del":416,"ins":657},"empty_hyp":0,"utts_over_50pct":164,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":99.32,"ci95":[93.94,105.24],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":809,"del":82,"ins":135},"empty_hyp":0,"utts_over_50pct":46,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":17.26,"ci95":[16.46,18.13],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2788,"del":438,"ins":440},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":25.28,"ci95":[24.19,26.38],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6115,"del":997,"ins":1143},"empty_hyp":0,"utts_over_50pct":25,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":117.78,"ci95":[114.84,120.78],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12710,"del":4427,"ins":3050},"empty_hyp":21,"utts_over_50pct":979,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":99.79,"ci95":[97.99,101.89],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12395,"del":1268,"ins":1320},"empty_hyp":2,"utts_over_50pct":856,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":102.88,"ci95":[101.85,104.24],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":19530,"del":22591,"ins":1226},"empty_hyp":12,"utts_over_50pct":838,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":12.98,"ci95":[11.77,14.26],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1569,"del":627,"ins":98},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":107.78,"ci95":[104.95,110.59],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13710,"del":6173,"ins":2708},"empty_hyp":1,"utts_over_50pct":933,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":102.73,"ci95":[100.18,105.9],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7259,"del":1999,"ins":1476},"empty_hyp":4,"utts_over_50pct":478,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":91.78,"ci95":[90.81,92.83],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13273,"del":1400,"ins":1052},"empty_hyp":0,"utts_over_50pct":983,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":84.6,"ci95":[83.48,85.62],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10887,"del":973,"ins":1182},"empty_hyp":0,"utts_over_50pct":840,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","quant":"Q8_0","metric":"wer","err_pct":81.65,"ci95":[80.04,83.57],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17505,"del":5466,"ins":2357},"empty_hyp":2,"utts_over_50pct":983,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":63.95,"ci95":[62.78,65.06],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10583,"del":998,"ins":1901},"empty_hyp":0,"utts_over_50pct":768,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ml","quant":"Q8_0","metric":"wer","err_pct":102.84,"ci95":[102.22,103.53],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":25468,"del":28736,"ins":1576},"empty_hyp":2,"utts_over_50pct":958,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mn","quant":"Q8_0","metric":"wer","err_pct":124.42,"ci95":[120.42,128.66],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10704,"del":7824,"ins":4529},"empty_hyp":8,"utts_over_50pct":949,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mr","quant":"Q8_0","metric":"wer","err_pct":100.42,"ci95":[100.12,100.78],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":20589,"del":30654,"ins":286},"empty_hyp":1,"utts_over_50pct":1015,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":40.87,"ci95":[39.31,42.67],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4850,"del":411,"ins":768},"empty_hyp":1,"utts_over_50pct":215,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":103.46,"ci95":[101.68,105.42],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14173,"del":7061,"ins":1768},"empty_hyp":39,"utts_over_50pct":926,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nb","quant":"Q8_0","metric":"wer","err_pct":49.26,"ci95":[47.5,50.95],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2903,"del":360,"ins":490},"empty_hyp":0,"utts_over_50pct":156,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ne","quant":"Q8_0","metric":"wer","err_pct":101.15,"ci95":[100.48,101.94],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15638,"del":17476,"ins":616},"empty_hyp":15,"utts_over_50pct":726,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":36.75,"ci95":[35.31,38.27],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2283,"del":290,"ins":513},"empty_hyp":0,"utts_over_50pct":62,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"oc","quant":"Q8_0","metric":"wer","err_pct":88.62,"ci95":[86.95,90.5],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17761,"del":2993,"ins":2643},"empty_hyp":0,"utts_over_50pct":989,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pa","quant":"Q8_0","metric":"wer","err_pct":101.13,"ci95":[100.15,102.93],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10050,"del":15449,"ins":294},"empty_hyp":3,"utts_over_50pct":574,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":35.68,"ci95":[34.46,36.98],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3959,"del":442,"ins":732},"empty_hyp":0,"utts_over_50pct":153,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ps","quant":"Q8_0","metric":"wer","err_pct":101.19,"ci95":[100.22,102.37],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8434,"del":4628,"ins":311},"empty_hyp":13,"utts_over_50pct":512,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":13.91,"ci95":[13.11,14.77],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2169,"del":293,"ins":525},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":62.16,"ci95":[60.92,63.47],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10025,"del":1183,"ins":1721},"empty_hyp":0,"utts_over_50pct":640,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":22.92,"ci95":[22.0,23.95],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2744,"del":352,"ins":366},"empty_hyp":0,"utts_over_50pct":39,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sd","quant":"Q8_0","metric":"wer","err_pct":103.23,"ci95":[101.54,105.16],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":20008,"del":2277,"ins":2116},"empty_hyp":1,"utts_over_50pct":980,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":65.77,"ci95":[64.54,67.01],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8217,"del":815,"ins":995},"empty_hyp":3,"utts_over_50pct":653,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":77.9,"ci95":[76.78,79.09],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10382,"del":1143,"ins":1340},"empty_hyp":0,"utts_over_50pct":797,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sn","quant":"Q8_0","metric":"wer","err_pct":134.76,"ci95":[129.9,140.17],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11530,"del":3992,"ins":6039},"empty_hyp":14,"utts_over_50pct":925,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"so","quant":"Q8_0","metric":"wer","err_pct":107.06,"ci95":[105.6,108.9],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17989,"del":6189,"ins":1714},"empty_hyp":4,"utts_over_50pct":1019,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sr","quant":"Q8_0","metric":"wer","err_pct":69.25,"ci95":[67.61,70.81],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7905,"del":864,"ins":823},"empty_hyp":0,"utts_over_50pct":573,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":42.4,"ci95":[41.0,43.92],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4806,"del":598,"ins":1120},"empty_hyp":0,"utts_over_50pct":268,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sw","quant":"Q8_0","metric":"wer","err_pct":100.69,"ci95":[98.1,103.39],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6701,"del":1873,"ins":1492},"empty_hyp":19,"utts_over_50pct":486,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ta","quant":"Q8_0","metric":"wer","err_pct":58.84,"ci95":[56.79,61.23],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10760,"del":3707,"ins":5340},"empty_hyp":0,"utts_over_50pct":313,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"te","quant":"Q8_0","metric":"wer","err_pct":101.77,"ci95":[100.99,102.71],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8390,"del":14193,"ins":424},"empty_hyp":1,"utts_over_50pct":472,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tg","quant":"Q8_0","metric":"wer","err_pct":108.3,"ci95":[106.47,110.51],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11412,"del":1516,"ins":1188},"empty_hyp":9,"utts_over_50pct":600,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":38.1,"ci95":[36.81,39.42],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":18768,"del":10910,"ins":7472},"empty_hyp":17,"utts_over_50pct":112,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":31.09,"ci95":[29.91,32.41],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3257,"del":393,"ins":484},"empty_hyp":0,"utts_over_50pct":94,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":42.03,"ci95":[40.78,43.22],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4827,"del":569,"ins":644},"empty_hyp":0,"utts_over_50pct":249,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ur","quant":"Q8_0","metric":"wer","err_pct":55.42,"ci95":[53.8,57.15],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3818,"del":507,"ins":385},"empty_hyp":0,"utts_over_50pct":182,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uz","quant":"Q8_0","metric":"wer","err_pct":111.42,"ci95":[108.74,114.38],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9508,"del":6212,"ins":2072},"empty_hyp":21,"utts_over_50pct":862,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":42.6,"ci95":[41.42,43.81],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9608,"del":542,"ins":934},"empty_hyp":0,"utts_over_50pct":235,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yo","quant":"Q8_0","metric":"wer","err_pct":103.28,"ci95":[101.84,105.0],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14460,"del":7074,"ins":1352},"empty_hyp":5,"utts_over_50pct":831,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":36.21,"ci95":[35.11,37.29],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10924,"del":1681,"ins":307},"empty_hyp":0,"utts_over_50pct":167,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":5.11,"ci95":[4.83,5.39],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2046,"del":311,"ins":351},"empty_hyp":0,"utts_over_50pct":21,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":5.1,"ci95":[4.83,5.38],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2046,"del":310,"ins":351},"empty_hyp":0,"utts_over_50pct":21,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.12,"ci95":[4.85,5.39],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2057,"del":295,"ins":361},"empty_hyp":0,"utts_over_50pct":22,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":5.11,"ci95":[4.85,5.39],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2048,"del":314,"ins":350},"empty_hyp":0,"utts_over_50pct":21,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":5.19,"ci95":[4.91,5.48],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2071,"del":329,"ins":352},"empty_hyp":0,"utts_over_50pct":21,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":5.36,"ci95":[5.07,5.66],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2126,"del":337,"ins":379},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":134.48,"ci95":[129.24,140.11],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":33931,"del":28350,"ins":23399},"empty_hyp":133,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":104.35,"ci95":[102.73,106.31],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":31866,"del":5210,"ins":1977},"empty_hyp":8,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"my","quant":"Q8_0","metric":"cer","err_pct":130.63,"ci95":[126.3,135.27],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":33930,"del":24069,"ins":18411},"empty_hyp":23,"utts_over_50pct":880,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":93.07,"ci95":[91.72,94.55],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"5d0c03e","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":10363,"del":2258,"ins":1067},"empty_hyp":0,"utts_over_50pct":727} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":482.5,"xrt_compute":73.23,"load_ms":56.5,"mel_ms":4.8,"encode_ms":146.9,"decode_ms":330.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":482.7,"xrt_wall":73.19,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":210.8,"xrt_compute":52.18,"load_ms":53.4,"mel_ms":4.1,"encode_ms":144.5,"decode_ms":62.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":211.0,"xrt_wall":52.12,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":436.6,"xrt_compute":80.93,"load_ms":62.9,"mel_ms":4.9,"encode_ms":129.2,"decode_ms":302.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":436.9,"xrt_wall":80.88,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":185.3,"xrt_compute":59.35,"load_ms":58.1,"mel_ms":4.1,"encode_ms":130.0,"decode_ms":51.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":185.6,"xrt_wall":59.27,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":187.2,"xrt_compute":188.77,"load_ms":56.2,"mel_ms":4.9,"encode_ms":15.2,"decode_ms":167.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":187.3,"xrt_wall":188.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":54.6,"xrt_compute":201.42,"load_ms":52.7,"mel_ms":4.1,"encode_ms":15.3,"decode_ms":35.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":54.7,"xrt_wall":201.05,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":191.6,"xrt_compute":184.36,"load_ms":62.4,"mel_ms":4.9,"encode_ms":15.0,"decode_ms":171.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":191.8,"xrt_wall":184.21,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":54.5,"xrt_compute":201.79,"load_ms":59.3,"mel_ms":4.1,"encode_ms":15.1,"decode_ms":35.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":54.6,"xrt_wall":201.33,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1561.3,"xrt_compute":22.63,"load_ms":109.6,"mel_ms":38.4,"encode_ms":378.5,"decode_ms":1144.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1561.5,"xrt_wall":22.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":635.3,"xrt_compute":17.32,"load_ms":132.0,"mel_ms":33.9,"encode_ms":376.1,"decode_ms":225.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":635.5,"xrt_wall":17.31,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1633.2,"xrt_compute":21.63,"load_ms":125.8,"mel_ms":39.6,"encode_ms":386.6,"decode_ms":1207.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1633.3,"xrt_wall":21.63,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":645.8,"xrt_compute":17.03,"load_ms":161.9,"mel_ms":28.9,"encode_ms":385.6,"decode_ms":231.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":645.9,"xrt_wall":17.03,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1212.6,"xrt_compute":29.14,"load_ms":91.5,"mel_ms":37.4,"encode_ms":179.4,"decode_ms":995.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1213.0,"xrt_wall":29.13,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":407.5,"xrt_compute":27.0,"load_ms":115.1,"mel_ms":33.7,"encode_ms":179.8,"decode_ms":193.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":407.7,"xrt_wall":26.98,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1208.9,"xrt_compute":29.23,"load_ms":103.8,"mel_ms":36.4,"encode_ms":177.6,"decode_ms":994.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1209.3,"xrt_wall":29.22,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":381.9,"xrt_compute":28.8,"load_ms":137.0,"mel_ms":33.5,"encode_ms":177.1,"decode_ms":171.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":382.2,"xrt_wall":28.78,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-large-v2.json b/catalog/whisper-large-v2.json new file mode 100644 index 00000000..5e64a8bc --- /dev/null +++ b/catalog/whisper-large-v2.json @@ -0,0 +1,160 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-large-v2", + "family": "whisper", + "display_name": "whisper-large-v2", + "params": 1543321440, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-large-v2", + "upstream_commit": "ae46427", + "published_repo": "handy-computer/whisper-large-v2-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "zh" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-large-v2-F32.gguf","size_bytes":6175245408}, + {"quant":"F16","filename":"whisper-large-v2-F16.gguf","size_bytes":3106458208}, + {"quant":"Q8_0","filename":"whisper-large-v2-Q8_0.gguf","size_bytes":1667964224}, + {"quant":"Q6_K","filename":"whisper-large-v2-Q6_K.gguf","size_bytes":1296353280}, + {"quant":"Q5_K_M","filename":"whisper-large-v2-Q5_K_M.gguf","size_bytes":1160366080}, + {"quant":"Q4_K_M","filename":"whisper-large-v2-Q4_K_M.gguf","size_bytes":996526080} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":38.45,"ci95":[36.36,40.66],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1787,"del":292,"ins":171},"empty_hyp":1,"utts_over_50pct":59,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":140.81,"ci95":[135.38,146.83],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7487,"del":1050,"ins":3522},"empty_hyp":4,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":17.06,"ci95":[15.68,18.51],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1034,"del":305,"ins":66},"empty_hyp":1,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":104.58,"ci95":[104.19,105.02],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":41441,"del":5499,"ins":2206},"empty_hyp":0,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":24.13,"ci95":[23.27,24.98],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3667,"del":353,"ins":306},"empty_hyp":0,"utts_over_50pct":47,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":46.96,"ci95":[46.01,47.89],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8517,"del":648,"ins":614},"empty_hyp":0,"utts_over_50pct":372,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":15.81,"ci95":[15.0,16.69],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1805,"del":223,"ins":213},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":103.42,"ci95":[102.99,103.87],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":37827,"del":6726,"ins":1747},"empty_hyp":0,"utts_over_50pct":918,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":17.02,"ci95":[16.25,17.87],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2570,"del":398,"ins":236},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":5.56,"ci95":[5.03,6.12],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":961,"del":232,"ins":155},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":14.42,"ci95":[13.62,15.22],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1656,"del":125,"ins":194},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":30.55,"ci95":[29.6,31.55],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6238,"del":804,"ins":964},"empty_hyp":1,"utts_over_50pct":89,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":14.92,"ci95":[14.19,15.63],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2333,"del":319,"ins":356},"empty_hyp":1,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":4.53,"ci95":[4.13,4.93],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":612,"del":91,"ins":144},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":13.51,"ci95":[12.53,14.42],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1774,"del":165,"ins":111},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.21,"ci95":[3.73,4.82],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":405,"del":126,"ins":82},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.3,"ci95":[2.93,3.7],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":556,"del":128,"ins":85},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":23.25,"ci95":[22.23,24.25],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2648,"del":314,"ins":498},"empty_hyp":0,"utts_over_50pct":50,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":34.25,"ci95":[33.18,35.39],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4932,"del":308,"ins":1644},"empty_hyp":0,"utts_over_50pct":122,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":9.58,"ci95":[8.93,10.26],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1141,"del":127,"ins":150},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":13.17,"ci95":[12.48,13.88],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2615,"del":449,"ins":417},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":5.81,"ci95":[5.29,6.33],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":794,"del":150,"ins":102},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":16.57,"ci95":[15.91,17.25],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2991,"del":372,"ins":313},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":103.37,"ci95":[103.0,103.74],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":43235,"del":1553,"ins":1719},"empty_hyp":0,"utts_over_50pct":1000,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":92.22,"ci95":[90.51,94.49],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11413,"del":2003,"ins":1278},"empty_hyp":0,"utts_over_50pct":619,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":27.78,"ci95":[26.13,29.34],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2984,"del":553,"ins":308},"empty_hyp":1,"utts_over_50pct":98,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":23.27,"ci95":[21.44,25.21],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2686,"del":1231,"ins":447},"empty_hyp":0,"utts_over_50pct":23,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":14.18,"ci95":[13.43,14.97],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2055,"del":248,"ins":214},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":17.84,"ci95":[17.0,18.71],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2448,"del":273,"ins":325},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":46.93,"ci95":[45.6,48.18],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6702,"del":1275,"ins":546},"empty_hyp":0,"utts_over_50pct":349,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":7.43,"ci95":[6.82,8.12],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":793,"del":87,"ins":127},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":39.59,"ci95":[35.74,43.66],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":335,"del":19,"ins":55},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":3.59,"ci95":[3.07,4.21],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":511,"del":173,"ins":79},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":5.56,"ci95":[5.09,6.06],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1304,"del":321,"ins":190},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":115.24,"ci95":[112.85,117.95],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15061,"del":2002,"ins":2688},"empty_hyp":1,"utts_over_50pct":979,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":40.13,"ci95":[39.15,41.23],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5091,"del":394,"ins":541},"empty_hyp":0,"utts_over_50pct":201,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":47.64,"ci95":[45.56,49.83],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11402,"del":7003,"ins":1669},"empty_hyp":2,"utts_over_50pct":235,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":4.99,"ci95":[3.99,6.0],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":297,"del":529,"ins":56},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":92.83,"ci95":[91.48,94.35],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":16224,"del":1620,"ins":1613},"empty_hyp":0,"utts_over_50pct":922,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":79.4,"ci95":[77.2,81.78],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6040,"del":453,"ins":1804},"empty_hyp":0,"utts_over_50pct":439,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":30.37,"ci95":[29.39,31.27],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4664,"del":275,"ins":264},"empty_hyp":0,"utts_over_50pct":110,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":24.49,"ci95":[23.58,25.48],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3247,"del":199,"ins":329},"empty_hyp":0,"utts_over_50pct":40,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.67,"ci95":[2.23,3.33],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":866,"del":281,"ins":267},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.68,"ci95":[2.23,3.35],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":869,"del":280,"ins":271},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.97,"ci95":[2.37,3.79],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":868,"del":286,"ins":420},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.83,"ci95":[2.31,3.57],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":873,"del":286,"ins":342},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.71,"ci95":[2.26,3.39],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":869,"del":292,"ins":274},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.46,"ci95":[2.21,2.78],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":870,"del":270,"ins":162},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":150.84,"ci95":[146.99,154.74],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":54849,"del":6541,"ins":34710},"empty_hyp":12,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":101.65,"ci95":[101.07,102.27],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":35553,"del":1166,"ins":1326},"empty_hyp":0,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","language_hint":"mi","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":39.72,"ci95":[38.5,40.98],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":9256,"del":1389,"ins":1677},"empty_hyp":0,"utts_over_50pct":249}, + {"dataset":"fleurs","split":"test","language":"mk","language_hint":"mk","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":18.76,"ci95":[17.94,19.58],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3210,"del":264,"ins":482},"empty_hyp":0,"utts_over_50pct":26}, + {"dataset":"fleurs","split":"test","language":"ml","language_hint":"ml","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":101.85,"ci95":[101.53,102.18],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":41708,"del":12307,"ins":1228},"empty_hyp":0,"utts_over_50pct":958}, + {"dataset":"fleurs","split":"test","language":"mn","language_hint":"mn","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":115.7,"ci95":[112.97,118.72],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":14761,"del":3674,"ins":3006},"empty_hyp":5,"utts_over_50pct":949}, + {"dataset":"fleurs","split":"test","language":"mr","language_hint":"mr","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":39.81,"ci95":[38.87,40.91],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":12433,"del":5512,"ins":2484},"empty_hyp":0,"utts_over_50pct":202}, + {"dataset":"fleurs","split":"test","language":"ms","language_hint":"ms","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":9.38,"ci95":[8.7,10.08],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1144,"del":132,"ins":108},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"fleurs","split":"test","language":"mt","language_hint":"mt","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":73.79,"ci95":[72.78,74.79],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":13495,"del":1341,"ins":1568},"empty_hyp":0,"utts_over_50pct":862}, + {"dataset":"fleurs","split":"test","language":"my","language_hint":"my","backend":"cuda","quant":"Q8_0","metric":"cer","err_pct":149.11,"ci95":[143.42,155.7],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":39444,"del":18579,"ins":29196},"empty_hyp":5,"utts_over_50pct":880}, + {"dataset":"fleurs","split":"test","language":"ne","language_hint":"ne","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":47.74,"ci95":[46.75,48.79],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":10447,"del":2686,"ins":2786},"empty_hyp":0,"utts_over_50pct":270}, + {"dataset":"fleurs","split":"test","language":"nl","language_hint":"nl","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":6.76,"ci95":[6.07,7.46],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":428,"del":70,"ins":70},"empty_hyp":0,"utts_over_50pct":0}, + {"dataset":"fleurs","split":"test","language":"nb","language_hint":"no","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":9.73,"ci95":[8.86,10.58],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":586,"del":75,"ins":80},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"fleurs","split":"test","language":"oc","language_hint":"oc","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":75.41,"ci95":[74.46,76.45],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":17232,"del":1280,"ins":1395},"empty_hyp":0,"utts_over_50pct":939}, + {"dataset":"fleurs","split":"test","language":"pa","language_hint":"pa","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":102.11,"ci95":[101.71,102.54],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":23505,"del":1937,"ins":601},"empty_hyp":0,"utts_over_50pct":574}, + {"dataset":"fleurs","split":"test","language":"pl","language_hint":"pl","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":5.87,"ci95":[5.27,6.59],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":685,"del":85,"ins":75},"empty_hyp":0,"utts_over_50pct":3}, + {"dataset":"fleurs","split":"test","language":"ps","language_hint":"ps","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":94.98,"ci95":[94.01,96.09],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":11028,"del":989,"ins":535},"empty_hyp":0,"utts_over_50pct":512}, + {"dataset":"fleurs","split":"test","language":"pt","language_hint":"pt","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":4.4,"ci95":[3.96,4.89],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":672,"del":100,"ins":172},"empty_hyp":0,"utts_over_50pct":3}, + {"dataset":"fleurs","split":"test","language":"ro","language_hint":"ro","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":17.1,"ci95":[16.07,18.11],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3016,"del":306,"ins":235},"empty_hyp":0,"utts_over_50pct":29}, + {"dataset":"fleurs","split":"test","language":"ru","language_hint":"ru","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":5.61,"ci95":[5.12,6.09],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":715,"del":77,"ins":55},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"fleurs","split":"test","language":"sd","language_hint":"sd","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":148.34,"ci95":[145.22,151.25],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":22192,"del":265,"ins":12608},"empty_hyp":0,"utts_over_50pct":980}, + {"dataset":"fleurs","split":"test","language":"sk","language_hint":"sk","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":12.49,"ci95":[11.55,13.36],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1517,"del":264,"ins":123},"empty_hyp":1,"utts_over_50pct":14}, + {"dataset":"fleurs","split":"test","language":"sl","language_hint":"sl","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":24.73,"ci95":[23.66,25.74],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3290,"del":340,"ins":454},"empty_hyp":0,"utts_over_50pct":59}, + {"dataset":"fleurs","split":"test","language":"sn","language_hint":"sn","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":127.84,"ci95":[125.11,130.72],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":13112,"del":682,"ins":6660},"empty_hyp":1,"utts_over_50pct":925}, + {"dataset":"fleurs","split":"test","language":"so","language_hint":"so","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":106.8,"ci95":[105.08,108.77],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":14461,"del":9700,"ins":1669},"empty_hyp":11,"utts_over_50pct":1019}, + {"dataset":"fleurs","split":"test","language":"sr","language_hint":"sr","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":38.06,"ci95":[34.91,41.17],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":4808,"del":314,"ins":150},"empty_hyp":0,"utts_over_50pct":186}, + {"dataset":"fleurs","split":"test","language":"sv","language_hint":"sv","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":9.25,"ci95":[8.6,9.89],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1074,"del":188,"ins":161},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"fleurs","split":"test","language":"sw","language_hint":"sw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":41.85,"ci95":[40.36,43.4],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3247,"del":315,"ins":622},"empty_hyp":0,"utts_over_50pct":138}, + {"dataset":"fleurs","split":"test","language":"ta","language_hint":"ta","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":20.12,"ci95":[18.65,21.85],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3548,"del":1451,"ins":1775},"empty_hyp":0,"utts_over_50pct":16}, + {"dataset":"fleurs","split":"test","language":"te","language_hint":"te","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":100.76,"ci95":[99.51,102.16],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":17295,"del":4791,"ins":693},"empty_hyp":3,"utts_over_50pct":464}, + {"dataset":"fleurs","split":"test","language":"tg","language_hint":"tg","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":89.42,"ci95":[87.58,91.43],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":10258,"del":205,"ins":1192},"empty_hyp":0,"utts_over_50pct":564}, + {"dataset":"fleurs","split":"test","language":"th","language_hint":"th","backend":"cuda","quant":"Q8_0","metric":"cer","err_pct":12.42,"ci95":[11.72,13.23],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":5816,"del":3146,"ins":3151},"empty_hyp":0,"utts_over_50pct":8}, + {"dataset":"fleurs","split":"test","language":"tr","language_hint":"tr","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":7.63,"ci95":[7.0,8.34],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":735,"del":137,"ins":142},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"fleurs","split":"test","language":"uk","language_hint":"uk","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":8.17,"ci95":[7.5,8.82],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":905,"del":140,"ins":129},"empty_hyp":0,"utts_over_50pct":4}, + {"dataset":"fleurs","split":"test","language":"ur","language_hint":"ur","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":23.96,"ci95":[22.62,25.48],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1624,"del":188,"ins":224},"empty_hyp":0,"utts_over_50pct":9}, + {"dataset":"fleurs","split":"test","language":"uz","language_hint":"uz","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":91.97,"ci95":[90.59,93.5],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":12363,"del":952,"ins":1371},"empty_hyp":1,"utts_over_50pct":854}, + {"dataset":"fleurs","split":"test","language":"vi","language_hint":"vi","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":11.25,"ci95":[10.49,12.04],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":2382,"del":331,"ins":214},"empty_hyp":0,"utts_over_50pct":8}, + {"dataset":"fleurs","split":"test","language":"yo","language_hint":"yo","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":96.0,"ci95":[95.28,96.79],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":16523,"del":4130,"ins":620},"empty_hyp":0,"utts_over_50pct":828}, + {"dataset":"fleurs","split":"test","language":"zh","language_hint":"zh","backend":"cuda","quant":"Q8_0","metric":"cer","err_pct":15.39,"ci95":[14.02,16.64],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3850,"del":1453,"ins":186},"empty_hyp":0,"utts_over_50pct":27}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":69.69,"ci95":[68.45,71.02],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"5d0c03e","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":8706,"del":806,"ins":738},"empty_hyp":0,"utts_over_50pct":638} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":8642.4,"xrt_compute":4.09,"load_ms":267.8,"mel_ms":4.9,"encode_ms":3036.0,"decode_ms":5601.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8642.9,"xrt_wall":4.09,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":4152.3,"xrt_compute":2.65,"load_ms":242.5,"mel_ms":4.3,"encode_ms":3084.4,"decode_ms":1063.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4152.8,"xrt_wall":2.65,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":8094.9,"xrt_compute":4.36,"load_ms":412.9,"mel_ms":4.9,"encode_ms":2756.9,"decode_ms":5333.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8095.4,"xrt_wall":4.36,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3729.5,"xrt_compute":2.95,"load_ms":360.2,"mel_ms":4.1,"encode_ms":2735.0,"decode_ms":990.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3730.0,"xrt_wall":2.95,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1543.4,"xrt_compute":22.89,"load_ms":279.2,"mel_ms":5.9,"encode_ms":306.5,"decode_ms":1231.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1543.6,"xrt_wall":22.89,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":590.0,"xrt_compute":18.65,"load_ms":260.6,"mel_ms":4.2,"encode_ms":307.8,"decode_ms":278.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":590.2,"xrt_wall":18.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1600.4,"xrt_compute":22.08,"load_ms":430.0,"mel_ms":6.6,"encode_ms":307.6,"decode_ms":1286.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1600.6,"xrt_wall":22.07,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":597.7,"xrt_compute":18.4,"load_ms":389.0,"mel_ms":4.2,"encode_ms":304.8,"decode_ms":288.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":597.9,"xrt_wall":18.4,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":29473.7,"xrt_compute":1.2,"load_ms":705.8,"mel_ms":33.8,"encode_ms":9565.7,"decode_ms":19874.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":29481.6,"xrt_wall":1.2,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":13692.2,"xrt_compute":0.8,"load_ms":1135.4,"mel_ms":29.6,"encode_ms":9619.4,"decode_ms":4043.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13699.8,"xrt_wall":0.8,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":31185.9,"xrt_compute":1.13,"load_ms":1154.5,"mel_ms":37.7,"encode_ms":9825.2,"decode_ms":21323.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":31193.9,"xrt_wall":1.13,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":13956.1,"xrt_compute":0.79,"load_ms":1944.3,"mel_ms":32.2,"encode_ms":9802.2,"decode_ms":4121.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13964.4,"xrt_wall":0.79,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":15047.8,"xrt_compute":2.35,"load_ms":496.0,"mel_ms":47.7,"encode_ms":4511.1,"decode_ms":10489.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15048.8,"xrt_wall":2.35,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":6514.8,"xrt_compute":1.69,"load_ms":964.4,"mel_ms":40.5,"encode_ms":4501.7,"decode_ms":1972.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6515.4,"xrt_wall":1.69,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":15244.5,"xrt_compute":2.32,"load_ms":638.9,"mel_ms":46.5,"encode_ms":4436.3,"decode_ms":10761.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15245.3,"xrt_wall":2.32,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":6480.0,"xrt_compute":1.7,"load_ms":1428.1,"mel_ms":45.0,"encode_ms":4417.4,"decode_ms":2017.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6480.7,"xrt_wall":1.7,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-large-v3-turbo.json b/catalog/whisper-large-v3-turbo.json new file mode 100644 index 00000000..87f52fe0 --- /dev/null +++ b/catalog/whisper-large-v3-turbo.json @@ -0,0 +1,166 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-large-v3-turbo", + "family": "whisper", + "display_name": "Whisper large-v3-turbo", + "params": 808904208, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-large-v3-turbo", + "upstream_commit": "41f01f3", + "published_repo": "handy-computer/whisper-large-v3-turbo-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "yue", "zh" + ], + "language_tag_form": "bare-bcp47", + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"],"note":"upstream exposes word-level via cross-attention alignment; this port ships segment"}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F16","filename":"whisper-large-v3-turbo-F16.gguf","size_bytes":1625935520}, + {"quant":"Q8_0","filename":"whisper-large-v3-turbo-Q8_0.gguf","size_bytes":886381760}, + {"quant":"Q6_K","filename":"whisper-large-v3-turbo-Q6_K.gguf","size_bytes":692536928}, + {"quant":"Q5_K_M","filename":"whisper-large-v3-turbo-Q5_K_M.gguf","size_bytes":619628128}, + {"quant":"Q4_K_M","filename":"whisper-large-v3-turbo-Q4_K_M.gguf","size_bytes":536069728} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":36.06,"ci95":[33.6,38.5],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1651,"del":326,"ins":133},"empty_hyp":0,"utts_over_50pct":50,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":146.29,"ci95":[142.41,150.14],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8041,"del":508,"ins":3979},"empty_hyp":0,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":15.48,"ci95":[14.1,16.99],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":923,"del":312,"ins":40},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":101.22,"ci95":[100.8,101.73],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":25156,"del":21677,"ins":733},"empty_hyp":5,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":23.15,"ci95":[22.1,24.28],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3443,"del":342,"ins":364},"empty_hyp":0,"utts_over_50pct":41,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":50.65,"ci95":[49.58,51.71],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9393,"del":571,"ins":585},"empty_hyp":0,"utts_over_50pct":464,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":13.58,"ci95":[12.78,14.38],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1565,"del":217,"ins":142},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":67.53,"ci95":[66.01,68.93],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":18201,"del":9171,"ins":2863},"empty_hyp":1,"utts_over_50pct":683,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":14.77,"ci95":[14.06,15.48],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2262,"del":304,"ins":214},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":5.42,"ci95":[5.01,5.89],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":958,"del":188,"ins":166},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":11.81,"ci95":[11.06,12.54],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1337,"del":131,"ins":149},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":36.42,"ci95":[35.37,37.52],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7517,"del":948,"ins":1078},"empty_hyp":0,"utts_over_50pct":169,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":13.6,"ci95":[12.93,14.3],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2121,"del":291,"ins":330},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":4.54,"ci95":[4.14,4.96],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":616,"del":100,"ins":134},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":13.26,"ci95":[12.35,14.13],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1763,"del":151,"ins":98},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.38,"ci95":[3.95,4.84],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":446,"del":107,"ins":84},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.12,"ci95":[2.8,3.48],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":529,"del":103,"ins":95},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":18.44,"ci95":[17.5,19.37],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2055,"del":247,"ins":442},"empty_hyp":0,"utts_over_50pct":29,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":30.56,"ci95":[29.52,31.66],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4263,"del":251,"ins":1628},"empty_hyp":0,"utts_over_50pct":83,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":8.29,"ci95":[7.65,8.93],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":961,"del":136,"ins":129},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":12.08,"ci95":[11.43,12.72],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2349,"del":474,"ins":368},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":5.51,"ci95":[5.01,6.06],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":757,"del":132,"ins":103},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":12.76,"ci95":[12.22,13.36],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2392,"del":250,"ins":189},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":78.95,"ci95":[77.6,80.23],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15141,"del":19233,"ins":1145},"empty_hyp":6,"utts_over_50pct":813,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":97.24,"ci95":[95.3,99.32],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10766,"del":3171,"ins":1557},"empty_hyp":0,"utts_over_50pct":618,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":29.71,"ci95":[28.05,31.36],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3185,"del":570,"ins":358},"empty_hyp":0,"utts_over_50pct":127,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":18.85,"ci95":[17.82,20.1],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2200,"del":923,"ins":413},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":12.54,"ci95":[11.83,13.26],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1791,"del":223,"ins":212},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":15.07,"ci95":[14.25,15.92],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2051,"del":231,"ins":291},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":45.62,"ci95":[44.42,46.72],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6640,"del":960,"ins":686},"empty_hyp":0,"utts_over_50pct":334,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":7.2,"ci95":[6.56,7.82],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":724,"del":115,"ins":137},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":21.39,"ci95":[17.91,25.44],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":179,"del":17,"ins":25},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":2.77,"ci95":[2.45,3.08],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":439,"del":69,"ins":81},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":4.82,"ci95":[4.4,5.3],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1109,"del":295,"ins":171},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":109.21,"ci95":[107.78,110.55],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11569,"del":5555,"ins":1594},"empty_hyp":3,"utts_over_50pct":979,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":21.27,"ci95":[20.15,22.31],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2480,"del":245,"ins":468},"empty_hyp":0,"utts_over_50pct":41,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":32.57,"ci95":[31.26,34.08],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5855,"del":5913,"ins":1954},"empty_hyp":0,"utts_over_50pct":101,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":5.24,"ci95":[4.3,6.25],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":350,"del":529,"ins":47},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":87.21,"ci95":[86.38,88.1],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15457,"del":2032,"ins":790},"empty_hyp":0,"utts_over_50pct":927,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":75.39,"ci95":[72.93,78.09],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5670,"del":599,"ins":1608},"empty_hyp":0,"utts_over_50pct":398,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":25.11,"ci95":[24.23,26.06],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3887,"del":211,"ins":204},"empty_hyp":0,"utts_over_50pct":62,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":19.53,"ci95":[18.64,20.43],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2588,"del":163,"ins":259},"empty_hyp":0,"utts_over_50pct":19,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","quant":"Q8_0","metric":"wer","err_pct":48.91,"ci95":[47.53,50.35],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11122,"del":1818,"ins":2234},"empty_hyp":0,"utts_over_50pct":409,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":17.85,"ci95":[17.07,18.66],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3048,"del":244,"ins":472},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ml","quant":"Q8_0","metric":"wer","err_pct":98.75,"ci95":[98.01,99.54],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":41848,"del":10353,"ins":1361},"empty_hyp":0,"utts_over_50pct":940,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mn","quant":"Q8_0","metric":"wer","err_pct":101.49,"ci95":[100.04,103.08],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15122,"del":2192,"ins":1495},"empty_hyp":0,"utts_over_50pct":949,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mr","quant":"Q8_0","metric":"wer","err_pct":36.12,"ci95":[35.26,36.99],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11530,"del":3929,"ins":3077},"empty_hyp":0,"utts_over_50pct":155,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":8.64,"ci95":[8.0,9.3],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1073,"del":97,"ins":105},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":70.92,"ci95":[69.87,71.96],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13004,"del":1412,"ins":1350},"empty_hyp":0,"utts_over_50pct":822,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nb","quant":"Q8_0","metric":"wer","err_pct":9.1,"ci95":[8.23,10.07],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":535,"del":81,"ins":77},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ne","quant":"Q8_0","metric":"wer","err_pct":43.15,"ci95":[42.1,44.17],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9161,"del":2433,"ins":2796},"empty_hyp":0,"utts_over_50pct":207,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":5.98,"ci95":[5.35,6.64],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":370,"del":61,"ins":71},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"oc","quant":"Q8_0","metric":"wer","err_pct":70.94,"ci95":[69.87,72.07],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15960,"del":1678,"ins":1089},"empty_hyp":0,"utts_over_50pct":898,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pa","quant":"Q8_0","metric":"wer","err_pct":99.53,"ci95":[98.36,100.52],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":22923,"del":1664,"ins":799},"empty_hyp":0,"utts_over_50pct":559,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":5.81,"ci95":[5.26,6.45],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":676,"del":84,"ins":76},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ps","quant":"Q8_0","metric":"wer","err_pct":91.81,"ci95":[90.6,93.25],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10325,"del":1018,"ins":790},"empty_hyp":0,"utts_over_50pct":512,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":4.17,"ci95":[3.73,4.69],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":620,"del":94,"ins":181},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":10.9,"ci95":[10.18,11.65],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1871,"del":238,"ins":159},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":5.93,"ci95":[4.94,7.56],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":677,"del":77,"ins":141},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sd","quant":"Q8_0","metric":"wer","err_pct":122.1,"ci95":[119.64,124.58],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":19571,"del":3445,"ins":5846},"empty_hyp":4,"utts_over_50pct":980,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":10.21,"ci95":[9.44,11.03],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1290,"del":141,"ins":125},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":20.56,"ci95":[19.59,21.52],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2703,"del":270,"ins":422},"empty_hyp":0,"utts_over_50pct":23,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sn","quant":"Q8_0","metric":"wer","err_pct":110.94,"ci95":[109.37,112.62],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12894,"del":310,"ins":4547},"empty_hyp":0,"utts_over_50pct":923,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"so","quant":"Q8_0","metric":"wer","err_pct":101.29,"ci95":[99.86,102.77],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":18032,"del":3950,"ins":2514},"empty_hyp":3,"utts_over_50pct":1019,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sr","quant":"Q8_0","metric":"wer","err_pct":32.36,"ci95":[29.53,35.38],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4167,"del":176,"ins":140},"empty_hyp":0,"utts_over_50pct":170,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":8.72,"ci95":[8.12,9.34],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1053,"del":167,"ins":121},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sw","quant":"Q8_0","metric":"wer","err_pct":33.96,"ci95":[32.53,35.33],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2595,"del":279,"ins":521},"empty_hyp":0,"utts_over_50pct":63,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ta","quant":"Q8_0","metric":"wer","err_pct":27.41,"ci95":[26.29,28.76],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5586,"del":1714,"ins":1929},"empty_hyp":0,"utts_over_50pct":39,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"te","quant":"Q8_0","metric":"wer","err_pct":63.03,"ci95":[60.4,65.94],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8435,"del":4727,"ins":1087},"empty_hyp":2,"utts_over_50pct":238,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tg","quant":"Q8_0","metric":"wer","err_pct":106.06,"ci95":[104.44,107.88],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11801,"del":798,"ins":1225},"empty_hyp":0,"utts_over_50pct":598,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":13.15,"ci95":[12.45,14.04],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6162,"del":3459,"ins":3205},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":6.97,"ci95":[6.33,7.64],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":655,"del":139,"ins":133},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":7.31,"ci95":[6.72,7.87],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":832,"del":108,"ins":110},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ur","quant":"Q8_0","metric":"wer","err_pct":23.19,"ci95":[21.92,24.57],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1569,"del":168,"ins":234},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uz","quant":"Q8_0","metric":"wer","err_pct":102.52,"ci95":[101.45,103.64],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13634,"del":1908,"ins":828},"empty_hyp":0,"utts_over_50pct":862,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":9.48,"ci95":[8.85,10.17],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1970,"del":282,"ins":215},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yo","quant":"Q8_0","metric":"wer","err_pct":99.38,"ci95":[98.13,100.68],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17716,"del":1333,"ins":2972},"empty_hyp":0,"utts_over_50pct":830,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yue","quant":"Q8_0","metric":"cer","err_pct":34.62,"ci95":[33.41,36.13],"n_utts":819,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9232,"del":303,"ins":504},"empty_hyp":0,"utts_over_50pct":69,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":8.5,"ci95":[7.65,9.4],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1379,"del":1471,"ins":182},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.01,"ci95":[1.83,2.21],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":765,"del":183,"ins":117},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.01,"ci95":[1.82,2.22],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":763,"del":185,"ins":117},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.01,"ci95":[1.82,2.2],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":769,"del":185,"ins":111},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.03,"ci95":[1.83,2.25],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":769,"del":174,"ins":131},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.04,"ci95":[1.85,2.26],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":784,"del":188,"ins":111},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":95.2,"ci95":[93.97,96.37],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":29731,"del":27154,"ins":3767},"empty_hyp":2,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":115.41,"ci95":[111.93,119.15],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":25892,"del":10285,"ins":7018},"empty_hyp":0,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"my","quant":"Q8_0","metric":"cer","err_pct":121.67,"ci95":[117.84,125.71],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":39288,"del":15150,"ins":16732},"empty_hyp":1,"utts_over_50pct":880,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":53.8,"ci95":[52.56,55.0],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":6586,"del":622,"ins":705},"empty_hyp":0,"utts_over_50pct":405} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":6302.9,"xrt_compute":5.61,"load_ms":160.7,"mel_ms":6.1,"encode_ms":2969.6,"decode_ms":3327.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6304.0,"xrt_wall":5.6,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3111.7,"xrt_compute":3.54,"load_ms":148.9,"mel_ms":5.2,"encode_ms":2961.9,"decode_ms":144.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3112.4,"xrt_wall":3.53,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":5803.3,"xrt_compute":6.09,"load_ms":210.3,"mel_ms":6.0,"encode_ms":2682.8,"decode_ms":3114.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5803.9,"xrt_wall":6.09,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2854.9,"xrt_compute":3.85,"load_ms":210.9,"mel_ms":5.2,"encode_ms":2713.4,"decode_ms":136.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2855.4,"xrt_wall":3.85,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":666.7,"xrt_compute":53.0,"load_ms":178.6,"mel_ms":7.1,"encode_ms":240.0,"decode_ms":419.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":666.9,"xrt_wall":52.98,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":291.1,"xrt_compute":37.79,"load_ms":152.3,"mel_ms":5.8,"encode_ms":239.3,"decode_ms":46.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":291.3,"xrt_wall":37.77,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":691.2,"xrt_compute":51.12,"load_ms":233.9,"mel_ms":7.0,"encode_ms":244.9,"decode_ms":439.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":691.4,"xrt_wall":51.1,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":303.8,"xrt_compute":36.21,"load_ms":216.3,"mel_ms":6.4,"encode_ms":248.0,"decode_ms":49.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":304.0,"xrt_wall":36.19,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":19717.3,"xrt_compute":1.79,"load_ms":683.3,"mel_ms":40.9,"encode_ms":8996.5,"decode_ms":10680.0,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":19724.4,"xrt_wall":1.79,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":9578.0,"xrt_compute":1.15,"load_ms":426.4,"mel_ms":36.7,"encode_ms":8968.1,"decode_ms":573.2,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":9585.0,"xrt_wall":1.15,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":20355.1,"xrt_compute":1.74,"load_ms":1121.0,"mel_ms":44.7,"encode_ms":9226.6,"decode_ms":11083.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":20362.7,"xrt_wall":1.74,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":9849.8,"xrt_compute":1.12,"load_ms":662.0,"mel_ms":36.0,"encode_ms":9201.2,"decode_ms":612.6,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":9857.2,"xrt_wall":1.12,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":9669.2,"xrt_compute":3.65,"load_ms":575.3,"mel_ms":40.5,"encode_ms":4162.7,"decode_ms":5465.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":9670.1,"xrt_wall":3.65,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":4447.5,"xrt_compute":2.47,"load_ms":290.4,"mel_ms":34.3,"encode_ms":3999.3,"decode_ms":413.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4448.1,"xrt_wall":2.47,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":9587.4,"xrt_compute":3.69,"load_ms":906.1,"mel_ms":41.4,"encode_ms":4065.9,"decode_ms":5480.1,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":9588.1,"xrt_wall":3.69,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":4401.2,"xrt_compute":2.5,"load_ms":473.9,"mel_ms":35.6,"encode_ms":3945.7,"decode_ms":419.9,"engine_sha":"cd0ea568","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4401.8,"xrt_wall":2.5,"publication_profile":"asr-publication-v2"} + ], + "language_aliases": { + "tl": "fil", + "no": "nb" + }, + "encoder_window_s": 30.0, + "max_output_tokens": 448 +} diff --git a/catalog/whisper-large-v3.json b/catalog/whisper-large-v3.json new file mode 100644 index 00000000..e5616049 --- /dev/null +++ b/catalog/whisper-large-v3.json @@ -0,0 +1,159 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-large-v3", + "family": "whisper", + "display_name": "whisper-large-v3", + "params": 1543516688, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-large-v3", + "upstream_commit": "06f233f", + "published_repo": "handy-computer/whisper-large-v3-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "yue", "zh" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F16","filename":"whisper-large-v3-F16.gguf","size_bytes":3107236640}, + {"quant":"Q8_0","filename":"whisper-large-v3-Q8_0.gguf","size_bytes":1668741440}, + {"quant":"Q6_K","filename":"whisper-large-v3-Q6_K.gguf","size_bytes":1297130208}, + {"quant":"Q5_K_M","filename":"whisper-large-v3-Q5_K_M.gguf","size_bytes":1161143008}, + {"quant":"Q4_K_M","filename":"whisper-large-v3-Q4_K_M.gguf","size_bytes":997303008} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":32.43,"ci95":[30.62,34.37],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1498,"del":265,"ins":135},"empty_hyp":0,"utts_over_50pct":29,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":132.7,"ci95":[128.33,137.71],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6805,"del":1749,"ins":2810},"empty_hyp":0,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":14.92,"ci95":[13.55,16.39],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":883,"del":299,"ins":47},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":104.77,"ci95":[104.31,105.27],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":43092,"del":3842,"ins":2300},"empty_hyp":1,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":21.1,"ci95":[20.13,22.06],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3148,"del":338,"ins":296},"empty_hyp":0,"utts_over_50pct":27,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":43.78,"ci95":[42.88,44.74],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7979,"del":547,"ins":591},"empty_hyp":0,"utts_over_50pct":271,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":12.82,"ci95":[12.0,13.63],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1466,"del":195,"ins":156},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":55.03,"ci95":[53.68,56.39],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12373,"del":9950,"ins":2315},"empty_hyp":1,"utts_over_50pct":438,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":13.5,"ci95":[12.8,14.17],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2047,"del":302,"ins":193},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":4.97,"ci95":[4.52,5.44],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":868,"del":168,"ins":168},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":10.5,"ci95":[9.79,11.22],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1170,"del":129,"ins":139},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":30.67,"ci95":[29.64,31.68],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6204,"del":929,"ins":904},"empty_hyp":0,"utts_over_50pct":90,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":12.48,"ci95":[11.78,13.16],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1937,"del":251,"ins":327},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":4.13,"ci95":[3.74,4.51],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":550,"del":104,"ins":118},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":11.53,"ci95":[10.7,12.34],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1528,"del":136,"ins":85},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.03,"ci95":[3.59,4.46],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":405,"del":110,"ins":72},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":2.7,"ci95":[2.41,3.01],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":459,"del":99,"ins":71},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":18.3,"ci95":[17.41,19.25],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2027,"del":266,"ins":430},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":30.11,"ci95":[29.13,31.18],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4173,"del":250,"ins":1630},"empty_hyp":0,"utts_over_50pct":80,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":7.73,"ci95":[7.12,8.32],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":904,"del":125,"ins":114},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":11.82,"ci95":[11.17,12.51],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2237,"del":507,"ins":379},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":5.39,"ci95":[4.88,5.94],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":732,"del":148,"ins":90},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":13.27,"ci95":[12.74,13.87],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2458,"del":248,"ins":238},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":66.38,"ci95":[64.99,67.88],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11626,"del":16597,"ins":1640},"empty_hyp":4,"utts_over_50pct":650,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":85.65,"ci95":[84.44,86.99],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10593,"del":1944,"ins":1111},"empty_hyp":0,"utts_over_50pct":610,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":26.73,"ci95":[25.1,28.27],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2795,"del":592,"ins":313},"empty_hyp":0,"utts_over_50pct":103,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":17.06,"ci95":[15.97,18.29],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1908,"del":924,"ins":368},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":10.94,"ci95":[10.34,11.55],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1586,"del":178,"ins":178},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":13.4,"ci95":[12.72,14.14],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1820,"del":190,"ins":279},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":43.64,"ci95":[42.4,44.86],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6202,"del":1153,"ins":570},"empty_hyp":0,"utts_over_50pct":284,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":6.08,"ci95":[5.48,6.67],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":620,"del":83,"ins":121},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":31.85,"ci95":[27.47,36.75],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":272,"del":23,"ins":34},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":2.54,"ci95":[2.16,2.98],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":374,"del":113,"ins":52},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":4.81,"ci95":[4.25,5.61],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1057,"del":360,"ins":155},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":93.97,"ci95":[93.03,94.94],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10347,"del":4748,"ins":1011},"empty_hyp":0,"utts_over_50pct":964,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":33.07,"ci95":[32.07,34.13],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4150,"del":391,"ins":424},"empty_hyp":0,"utts_over_50pct":95,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":31.99,"ci95":[30.69,33.47],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5727,"del":6119,"ins":1634},"empty_hyp":1,"utts_over_50pct":90,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":4.89,"ci95":[3.93,5.9],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":302,"del":526,"ins":37},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":85.67,"ci95":[84.45,86.95],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15504,"del":1429,"ins":1024},"empty_hyp":0,"utts_over_50pct":921,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":72.22,"ci95":[70.26,74.29],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5560,"del":429,"ins":1557},"empty_hyp":0,"utts_over_50pct":408,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":24.55,"ci95":[23.68,25.41],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3793,"del":201,"ins":212},"empty_hyp":0,"utts_over_50pct":61,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":19.21,"ci95":[18.35,20.07],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2560,"del":159,"ins":242},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","quant":"Q8_0","metric":"wer","err_pct":38.78,"ci95":[37.57,40.01],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8945,"del":1195,"ins":1889},"empty_hyp":0,"utts_over_50pct":255,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":15.09,"ci95":[14.29,15.89],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2541,"del":234,"ins":407},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ml","quant":"Q8_0","metric":"wer","err_pct":100.17,"ci95":[99.88,100.48],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":40925,"del":13002,"ins":404},"empty_hyp":0,"utts_over_50pct":957,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mn","quant":"Q8_0","metric":"wer","err_pct":85.6,"ci95":[84.78,86.45],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13800,"del":979,"ins":1085},"empty_hyp":0,"utts_over_50pct":946,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mr","quant":"Q8_0","metric":"wer","err_pct":34.3,"ci95":[33.52,35.13],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11097,"del":3989,"ins":2516},"empty_hyp":0,"utts_over_50pct":110,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":7.59,"ci95":[7.04,8.19],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":937,"del":96,"ins":86},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":68.79,"ci95":[67.66,70.06],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12676,"del":1117,"ins":1501},"empty_hyp":0,"utts_over_50pct":801,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nb","quant":"Q8_0","metric":"wer","err_pct":8.19,"ci95":[7.34,9.09],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":492,"del":67,"ins":65},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ne","quant":"Q8_0","metric":"wer","err_pct":40.44,"ci95":[39.64,41.28],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8605,"del":2155,"ins":2725},"empty_hyp":0,"utts_over_50pct":144,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":5.42,"ci95":[4.81,6.07],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":327,"del":63,"ins":65},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"oc","quant":"Q8_0","metric":"wer","err_pct":69.15,"ci95":[68.2,70.07],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15986,"del":1325,"ins":945},"empty_hyp":0,"utts_over_50pct":872,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pa","quant":"Q8_0","metric":"wer","err_pct":57.39,"ci95":[55.51,59.08],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7879,"del":6045,"ins":713},"empty_hyp":2,"utts_over_50pct":303,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":4.69,"ci95":[4.15,5.26],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":526,"del":95,"ins":54},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ps","quant":"Q8_0","metric":"wer","err_pct":89.14,"ci95":[88.29,90.04],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10194,"del":926,"ins":661},"empty_hyp":0,"utts_over_50pct":512,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":3.88,"ci95":[3.45,4.37],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":580,"del":99,"ins":154},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":9.2,"ci95":[8.56,9.85],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1532,"del":223,"ins":158},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":4.96,"ci95":[4.51,5.41],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":633,"del":72,"ins":44},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sd","quant":"Q8_0","metric":"wer","err_pct":184.2,"ci95":[182.18,186.34],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":23422,"del":132,"ins":19988},"empty_hyp":0,"utts_over_50pct":980,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":9.25,"ci95":[8.57,10.01],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1168,"del":146,"ins":96},"empty_hyp":1,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":19.15,"ci95":[18.25,20.02],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2549,"del":229,"ins":384},"empty_hyp":0,"utts_over_50pct":21,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sn","quant":"Q8_0","metric":"wer","err_pct":115.38,"ci95":[113.66,117.14],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13054,"del":159,"ins":5247},"empty_hyp":0,"utts_over_50pct":924,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"so","quant":"Q8_0","metric":"wer","err_pct":91.1,"ci95":[90.11,92.16],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17706,"del":1512,"ins":2815},"empty_hyp":1,"utts_over_50pct":1015,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sr","quant":"Q8_0","metric":"wer","err_pct":28.49,"ci95":[25.44,31.46],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3680,"del":149,"ins":118},"empty_hyp":0,"utts_over_50pct":127,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":7.8,"ci95":[7.22,8.37],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":930,"del":155,"ins":115},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sw","quant":"Q8_0","metric":"wer","err_pct":34.57,"ci95":[33.19,35.91],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2697,"del":301,"ins":458},"empty_hyp":0,"utts_over_50pct":72,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ta","quant":"Q8_0","metric":"wer","err_pct":20.04,"ci95":[18.55,21.85],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3722,"del":1327,"ins":1697},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"te","quant":"Q8_0","metric":"wer","err_pct":65.84,"ci95":[63.45,68.22],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7074,"del":6984,"ins":826},"empty_hyp":7,"utts_over_50pct":278,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tg","quant":"Q8_0","metric":"wer","err_pct":80.86,"ci95":[78.75,83.24],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8522,"del":336,"ins":1681},"empty_hyp":0,"utts_over_50pct":561,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":8.78,"ci95":[8.1,9.55],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3851,"del":2090,"ins":2618},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":6.51,"ci95":[5.9,7.13],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":616,"del":129,"ins":120},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":6.28,"ci95":[5.74,6.83],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":722,"del":95,"ins":85},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ur","quant":"Q8_0","metric":"wer","err_pct":21.75,"ci95":[20.42,23.08],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1446,"del":169,"ins":233},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uz","quant":"Q8_0","metric":"wer","err_pct":86.28,"ci95":[85.19,87.37],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12025,"del":972,"ins":780},"empty_hyp":0,"utts_over_50pct":847,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":8.74,"ci95":[8.12,9.46],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1771,"del":287,"ins":215},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yo","quant":"Q8_0","metric":"wer","err_pct":97.26,"ci95":[96.45,98.11],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17488,"del":2843,"ins":1221},"empty_hyp":0,"utts_over_50pct":827,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yue","quant":"Q8_0","metric":"cer","err_pct":22.06,"ci95":[20.16,24.13],"n_utts":819,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3416,"del":1614,"ins":1366},"empty_hyp":0,"utts_over_50pct":77,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":7.98,"ci95":[7.12,8.82],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1179,"del":1484,"ins":181},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":1.81,"ci95":[1.66,1.96],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":702,"del":144,"ins":116},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":1.82,"ci95":[1.67,1.98],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":704,"del":146,"ins":117},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":1.83,"ci95":[1.68,1.98],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":707,"del":145,"ins":117},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":1.84,"ci95":[1.7,2.0],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":709,"del":150,"ins":119},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":1.86,"ci95":[1.71,2.01],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":714,"del":157,"ins":117},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":101.09,"ci95":[99.27,102.93],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":39194,"del":14305,"ins":10904},"empty_hyp":0,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":100.28,"ci95":[99.24,101.49],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":33708,"del":2361,"ins":1462},"empty_hyp":0,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"my","quant":"Q8_0","metric":"cer","err_pct":143.39,"ci95":[138.96,148.17],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":44509,"del":13259,"ins":26105},"empty_hyp":15,"utts_over_50pct":880,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":64.72,"ci95":[63.4,65.92],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"5d0c03e","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":8055,"del":760,"ins":704},"empty_hyp":0,"utts_over_50pct":587} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":8715.1,"xrt_compute":4.05,"load_ms":268.5,"mel_ms":6.2,"encode_ms":3035.9,"decode_ms":5673.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8715.7,"xrt_wall":4.05,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":4139.1,"xrt_compute":2.66,"load_ms":245.6,"mel_ms":5.2,"encode_ms":3032.3,"decode_ms":1101.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4139.6,"xrt_wall":2.66,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":8073.9,"xrt_compute":4.38,"load_ms":410.7,"mel_ms":6.2,"encode_ms":2751.6,"decode_ms":5316.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8074.5,"xrt_wall":4.38,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3729.2,"xrt_compute":2.95,"load_ms":360.5,"mel_ms":5.2,"encode_ms":2723.8,"decode_ms":1000.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3729.9,"xrt_wall":2.95,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1576.6,"xrt_compute":22.41,"load_ms":289.0,"mel_ms":7.2,"encode_ms":303.5,"decode_ms":1266.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1576.9,"xrt_wall":22.41,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":616.5,"xrt_compute":17.84,"load_ms":248.8,"mel_ms":5.2,"encode_ms":308.9,"decode_ms":302.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":616.7,"xrt_wall":17.84,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1789.7,"xrt_compute":19.74,"load_ms":433.9,"mel_ms":7.5,"encode_ms":351.0,"decode_ms":1431.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1790.0,"xrt_wall":19.74,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":744.8,"xrt_compute":14.77,"load_ms":554.6,"mel_ms":34.3,"encode_ms":299.7,"decode_ms":410.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":745.2,"xrt_wall":14.76,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":28104.0,"xrt_compute":1.26,"load_ms":700.7,"mel_ms":33.9,"encode_ms":8933.8,"decode_ms":19136.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":28111.6,"xrt_wall":1.26,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":12952.0,"xrt_compute":0.85,"load_ms":1135.3,"mel_ms":28.0,"encode_ms":8943.7,"decode_ms":3980.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":12959.5,"xrt_wall":0.85,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":31241.4,"xrt_compute":1.13,"load_ms":1132.0,"mel_ms":36.0,"encode_ms":9778.0,"decode_ms":21427.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":31249.6,"xrt_wall":1.13,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":13597.0,"xrt_compute":0.81,"load_ms":1903.8,"mel_ms":29.5,"encode_ms":9345.9,"decode_ms":4221.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13604.9,"xrt_wall":0.81,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":15047.9,"xrt_compute":2.35,"load_ms":417.5,"mel_ms":46.7,"encode_ms":4504.4,"decode_ms":10496.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15049.0,"xrt_wall":2.35,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":6633.0,"xrt_compute":1.66,"load_ms":895.5,"mel_ms":40.8,"encode_ms":4492.1,"decode_ms":2100.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6633.8,"xrt_wall":1.66,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":15202.0,"xrt_compute":2.32,"load_ms":716.4,"mel_ms":47.7,"encode_ms":4431.1,"decode_ms":10723.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15203.3,"xrt_wall":2.32,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":6602.0,"xrt_compute":1.67,"load_ms":1495.8,"mel_ms":42.3,"encode_ms":4410.5,"decode_ms":2149.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6603.0,"xrt_wall":1.67,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-large.json b/catalog/whisper-large.json new file mode 100644 index 00000000..2b7da1bb --- /dev/null +++ b/catalog/whisper-large.json @@ -0,0 +1,160 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-large", + "family": "whisper", + "display_name": "whisper-large", + "params": 1543321440, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-large", + "upstream_commit": "4ef9b41", + "published_repo": "handy-computer/whisper-large-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "zh" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-large-F32.gguf","size_bytes":6175245376}, + {"quant":"F16","filename":"whisper-large-F16.gguf","size_bytes":3106458176}, + {"quant":"Q8_0","filename":"whisper-large-Q8_0.gguf","size_bytes":1667964192}, + {"quant":"Q6_K","filename":"whisper-large-Q6_K.gguf","size_bytes":1296353248}, + {"quant":"Q5_K_M","filename":"whisper-large-Q5_K_M.gguf","size_bytes":1160366048}, + {"quant":"Q4_K_M","filename":"whisper-large-Q4_K_M.gguf","size_bytes":996526048} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":44.92,"ci95":[42.61,47.28],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2113,"del":306,"ins":210},"empty_hyp":1,"utts_over_50pct":93,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":133.86,"ci95":[128.35,140.33],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6600,"del":1957,"ins":2907},"empty_hyp":5,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":19.47,"ci95":[18.01,21.12],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1198,"del":329,"ins":77},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":105.09,"ci95":[104.38,106.0],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":33038,"del":13910,"ins":2436},"empty_hyp":0,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":30.38,"ci95":[29.46,31.21],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4687,"del":436,"ins":322},"empty_hyp":0,"utts_over_50pct":72,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":58.26,"ci95":[57.26,59.33],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10751,"del":652,"ins":731},"empty_hyp":0,"utts_over_50pct":650,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":19.89,"ci95":[18.96,20.85],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2307,"del":286,"ins":226},"empty_hyp":0,"utts_over_50pct":17,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":105.08,"ci95":[104.55,105.69],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":39224,"del":5467,"ins":2355},"empty_hyp":0,"utts_over_50pct":920,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":22.2,"ci95":[21.32,23.09],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3408,"del":462,"ins":310},"empty_hyp":0,"utts_over_50pct":39,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":7.01,"ci95":[6.46,7.61],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1197,"del":286,"ins":216},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":18.77,"ci95":[17.92,19.75],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2172,"del":175,"ins":223},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":36.33,"ci95":[35.35,37.3],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7435,"del":801,"ins":1283},"empty_hyp":1,"utts_over_50pct":164,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":17.82,"ci95":[16.95,18.65],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2802,"del":353,"ins":436},"empty_hyp":0,"utts_over_50pct":30,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":5.04,"ci95":[4.63,5.47],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":702,"del":111,"ins":130},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":18.79,"ci95":[17.47,20.14],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2200,"del":490,"ins":161},"empty_hyp":0,"utts_over_50pct":31,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.46,"ci95":[3.9,5.11],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":416,"del":149,"ins":84},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.55,"ci95":[3.18,3.98],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":600,"del":133,"ins":96},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":27.31,"ci95":[26.31,28.35],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3118,"del":410,"ins":536},"empty_hyp":0,"utts_over_50pct":81,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":37.88,"ci95":[36.76,39.01],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5654,"del":428,"ins":1532},"empty_hyp":0,"utts_over_50pct":186,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":12.73,"ci95":[12.01,13.53],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1514,"del":167,"ins":202},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":15.44,"ci95":[14.78,16.15],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3038,"del":423,"ins":620},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":6.95,"ci95":[6.39,7.59],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":947,"del":169,"ins":134},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":20.1,"ci95":[19.39,20.93],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3582,"del":482,"ins":394},"empty_hyp":0,"utts_over_50pct":21,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":104.29,"ci95":[103.91,104.68],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":41887,"del":2986,"ins":2046},"empty_hyp":0,"utts_over_50pct":1000,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":90.12,"ci95":[88.77,91.57],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10808,"del":2550,"ins":1001},"empty_hyp":1,"utts_over_50pct":620,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":30.18,"ci95":[28.45,31.92],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3239,"del":587,"ins":351},"empty_hyp":0,"utts_over_50pct":128,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":29.1,"ci95":[27.16,31.25],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3293,"del":1570,"ins":595},"empty_hyp":0,"utts_over_50pct":36,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":18.32,"ci95":[17.52,19.12],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2654,"del":322,"ins":276},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":21.97,"ci95":[21.06,22.93],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2990,"del":351,"ins":410},"empty_hyp":0,"utts_over_50pct":37,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":54.73,"ci95":[53.53,55.85],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8043,"del":1408,"ins":489},"empty_hyp":0,"utts_over_50pct":524,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":8.93,"ci95":[8.27,9.68],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":972,"del":96,"ins":143},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":49.76,"ci95":[42.72,58.2],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":409,"del":31,"ins":74},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":3.67,"ci95":[3.26,4.11],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":570,"del":124,"ins":85},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":6.95,"ci95":[6.39,7.5],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1569,"del":439,"ins":260},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":119.65,"ci95":[116.17,122.89],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10777,"del":6338,"ins":3391},"empty_hyp":10,"utts_over_50pct":979,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":47.29,"ci95":[46.19,48.4],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5942,"del":514,"ins":644},"empty_hyp":0,"utts_over_50pct":324,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":74.18,"ci95":[72.35,76.2],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15951,"del":14036,"ins":1267},"empty_hyp":1,"utts_over_50pct":549,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":5.25,"ci95":[4.26,6.34],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":350,"del":534,"ins":45},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":91.22,"ci95":[89.48,93.29],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15211,"del":2637,"ins":1271},"empty_hyp":1,"utts_over_50pct":926,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":81.13,"ci95":[78.91,83.39],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6249,"del":417,"ins":1811},"empty_hyp":0,"utts_over_50pct":435,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":37.58,"ci95":[36.63,38.51],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5758,"del":319,"ins":362},"empty_hyp":0,"utts_over_50pct":194,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":30.18,"ci95":[29.26,31.23],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4018,"del":295,"ins":340},"empty_hyp":0,"utts_over_50pct":85,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":22.81,"ci95":[21.91,23.68],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3886,"del":364,"ins":560},"empty_hyp":0,"utts_over_50pct":41,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.72,"ci95":[2.42,3.06],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":912,"del":309,"ins":221},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.72,"ci95":[2.42,3.06],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":910,"del":311,"ins":221},"empty_hyp":1,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.71,"ci95":[2.4,3.03],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":908,"del":290,"ins":237},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.62,"ci95":[2.36,2.92],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":912,"del":299,"ins":181},"empty_hyp":0,"utts_over_50pct":13,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.84,"ci95":[2.48,3.31],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":916,"del":283,"ins":309},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.67,"ci95":[2.38,2.98],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":933,"del":292,"ins":191},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":118.15,"ci95":[114.41,122.27],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":29338,"del":32074,"ins":13862},"empty_hyp":128,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":102.35,"ci95":[101.26,103.98],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":35086,"del":1850,"ins":1371},"empty_hyp":0,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","language_hint":"mi","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":54.38,"ci95":[52.4,56.47],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":11313,"del":2533,"ins":3025},"empty_hyp":2,"utts_over_50pct":477}, + {"dataset":"fleurs","split":"test","language":"ml","language_hint":"ml","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":101.2,"ci95":[100.79,101.68],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":34921,"del":19187,"ins":780},"empty_hyp":15,"utts_over_50pct":958}, + {"dataset":"fleurs","split":"test","language":"mn","language_hint":"mn","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":115.92,"ci95":[112.19,119.7],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":12379,"del":6007,"ins":3097},"empty_hyp":9,"utts_over_50pct":949}, + {"dataset":"fleurs","split":"test","language":"mr","language_hint":"mr","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":48.22,"ci95":[47.0,49.6],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":14682,"del":7182,"ins":2879},"empty_hyp":0,"utts_over_50pct":351}, + {"dataset":"fleurs","split":"test","language":"ms","language_hint":"ms","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":11.28,"ci95":[10.54,12.03],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1422,"del":114,"ins":128},"empty_hyp":0,"utts_over_50pct":4}, + {"dataset":"fleurs","split":"test","language":"mt","language_hint":"mt","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":84.11,"ci95":[82.88,85.46],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":15296,"del":1580,"ins":1824},"empty_hyp":0,"utts_over_50pct":914}, + {"dataset":"fleurs","split":"test","language":"my","language_hint":"my","backend":"cuda","quant":"Q8_0","metric":"cer","err_pct":128.0,"ci95":[124.8,131.79],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":40608,"del":17542,"ins":16719},"empty_hyp":10,"utts_over_50pct":880}, + {"dataset":"fleurs","split":"test","language":"ne","language_hint":"ne","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":55.71,"ci95":[54.38,57.09],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":11265,"del":4142,"ins":3171},"empty_hyp":0,"utts_over_50pct":408}, + {"dataset":"fleurs","split":"test","language":"nl","language_hint":"nl","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":8.48,"ci95":[7.73,9.28],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":521,"del":93,"ins":98},"empty_hyp":0,"utts_over_50pct":0}, + {"dataset":"fleurs","split":"test","language":"nb","language_hint":"no","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":11.98,"ci95":[11.03,13.03],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":729,"del":94,"ins":90},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"fleurs","split":"test","language":"oc","language_hint":"oc","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":76.72,"ci95":[75.6,78.0],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":17277,"del":1470,"ins":1508},"empty_hyp":0,"utts_over_50pct":936}, + {"dataset":"fleurs","split":"test","language":"pa","language_hint":"pa","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":102.98,"ci95":[102.47,103.54],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":23755,"del":1683,"ins":828},"empty_hyp":0,"utts_over_50pct":573}, + {"dataset":"fleurs","split":"test","language":"pl","language_hint":"pl","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":7.44,"ci95":[6.82,8.18],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":849,"del":112,"ins":110},"empty_hyp":0,"utts_over_50pct":4}, + {"dataset":"fleurs","split":"test","language":"ps","language_hint":"ps","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":98.52,"ci95":[96.54,100.83],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":9149,"del":3103,"ins":769},"empty_hyp":4,"utts_over_50pct":512}, + {"dataset":"fleurs","split":"test","language":"pt","language_hint":"pt","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":4.49,"ci95":[4.04,4.99],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":686,"del":96,"ins":183},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"fleurs","split":"test","language":"ro","language_hint":"ro","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":17.52,"ci95":[16.52,18.48],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3028,"del":315,"ins":301},"empty_hyp":0,"utts_over_50pct":30}, + {"dataset":"fleurs","split":"test","language":"ru","language_hint":"ru","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":6.49,"ci95":[5.95,7.04],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":826,"del":89,"ins":65},"empty_hyp":0,"utts_over_50pct":1}, + {"dataset":"fleurs","split":"test","language":"sd","language_hint":"sd","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":176.25,"ci95":[173.82,178.51],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":22764,"del":820,"ins":18077},"empty_hyp":0,"utts_over_50pct":980}, + {"dataset":"fleurs","split":"test","language":"sk","language_hint":"sk","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":17.17,"ci95":[16.27,18.23],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":2177,"del":290,"ins":151},"empty_hyp":3,"utts_over_50pct":26}, + {"dataset":"fleurs","split":"test","language":"sl","language_hint":"sl","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":29.94,"ci95":[28.95,31.0],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3996,"del":411,"ins":538},"empty_hyp":0,"utts_over_50pct":75}, + {"dataset":"fleurs","split":"test","language":"sn","language_hint":"sn","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":142.46,"ci95":[136.47,148.75],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":10442,"del":5350,"ins":7001},"empty_hyp":41,"utts_over_50pct":925}, + {"dataset":"fleurs","split":"test","language":"so","language_hint":"so","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":105.45,"ci95":[103.87,107.13],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":13797,"del":10353,"ins":1354},"empty_hyp":10,"utts_over_50pct":1019}, + {"dataset":"fleurs","split":"test","language":"sr","language_hint":"sr","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":35.57,"ci95":[32.62,38.51],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":4529,"del":201,"ins":197},"empty_hyp":0,"utts_over_50pct":142}, + {"dataset":"fleurs","split":"test","language":"sv","language_hint":"sv","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":11.5,"ci95":[10.75,12.26],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1333,"del":241,"ins":195},"empty_hyp":0,"utts_over_50pct":4}, + {"dataset":"fleurs","split":"test","language":"sw","language_hint":"sw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":52.38,"ci95":[50.83,54.02],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3975,"del":751,"ins":510},"empty_hyp":0,"utts_over_50pct":246}, + {"dataset":"fleurs","split":"test","language":"ta","language_hint":"ta","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":21.7,"ci95":[20.49,23.11],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":3924,"del":1687,"ins":1695},"empty_hyp":0,"utts_over_50pct":22}, + {"dataset":"fleurs","split":"test","language":"te","language_hint":"te","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":99.27,"ci95":[98.16,100.29],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":14112,"del":7648,"ins":682},"empty_hyp":4,"utts_over_50pct":464}, + {"dataset":"fleurs","split":"test","language":"tg","language_hint":"tg","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":79.55,"ci95":[77.71,81.6],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":8649,"del":572,"ins":1147},"empty_hyp":1,"utts_over_50pct":573}, + {"dataset":"fleurs","split":"test","language":"th","language_hint":"th","backend":"cuda","quant":"Q8_0","metric":"cer","err_pct":13.88,"ci95":[13.16,14.69],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":6538,"del":3569,"ins":3425},"empty_hyp":0,"utts_over_50pct":15}, + {"dataset":"fleurs","split":"test","language":"tr","language_hint":"tr","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":8.73,"ci95":[7.99,9.46],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":855,"del":154,"ins":152},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"fleurs","split":"test","language":"uk","language_hint":"uk","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":9.87,"ci95":[9.18,10.58],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1156,"del":131,"ins":131},"empty_hyp":0,"utts_over_50pct":2}, + {"dataset":"fleurs","split":"test","language":"ur","language_hint":"ur","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":26.28,"ci95":[24.87,27.75],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":1798,"del":217,"ins":218},"empty_hyp":0,"utts_over_50pct":16}, + {"dataset":"fleurs","split":"test","language":"uz","language_hint":"uz","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":96.56,"ci95":[95.43,97.96],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":13128,"del":1131,"ins":1159},"empty_hyp":0,"utts_over_50pct":861}, + {"dataset":"fleurs","split":"test","language":"vi","language_hint":"vi","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":11.51,"ci95":[10.82,12.15],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":2474,"del":301,"ins":220},"empty_hyp":0,"utts_over_50pct":8}, + {"dataset":"fleurs","split":"test","language":"yo","language_hint":"yo","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":113.72,"ci95":[111.14,116.64],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":13837,"del":7939,"ins":3424},"empty_hyp":6,"utts_over_50pct":831}, + {"dataset":"fleurs","split":"test","language":"zh","language_hint":"zh","backend":"cuda","quant":"Q8_0","metric":"cer","err_pct":19.32,"ci95":[18.03,20.64],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":"0a26478","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":5198,"del":1508,"ins":184},"empty_hyp":0,"utts_over_50pct":43}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":86.31,"ci95":[83.02,89.82],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"5d0c03e","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":8176,"del":2424,"ins":2094},"empty_hyp":25,"utts_over_50pct":640} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":8896.8,"xrt_compute":3.97,"load_ms":280.2,"mel_ms":4.9,"encode_ms":3079.9,"decode_ms":5812.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8897.4,"xrt_wall":3.97,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":4065.1,"xrt_compute":2.71,"load_ms":246.7,"mel_ms":4.3,"encode_ms":3003.4,"decode_ms":1057.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4065.6,"xrt_wall":2.71,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":8050.8,"xrt_compute":4.39,"load_ms":416.3,"mel_ms":4.8,"encode_ms":2730.9,"decode_ms":5315.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":8051.4,"xrt_wall":4.39,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3700.4,"xrt_compute":2.97,"load_ms":373.0,"mel_ms":4.2,"encode_ms":2739.6,"decode_ms":956.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3700.9,"xrt_wall":2.97,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1418.6,"xrt_compute":24.91,"load_ms":294.2,"mel_ms":4.8,"encode_ms":256.4,"decode_ms":1157.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1418.8,"xrt_wall":24.9,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":562.3,"xrt_compute":19.56,"load_ms":254.3,"mel_ms":4.1,"encode_ms":292.4,"decode_ms":265.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":562.5,"xrt_wall":19.55,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1633.4,"xrt_compute":21.63,"load_ms":438.3,"mel_ms":5.6,"encode_ms":300.6,"decode_ms":1327.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1633.7,"xrt_wall":21.63,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":596.0,"xrt_compute":18.46,"load_ms":386.5,"mel_ms":4.3,"encode_ms":304.8,"decode_ms":286.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":596.1,"xrt_wall":18.45,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":29554.5,"xrt_compute":1.2,"load_ms":696.8,"mel_ms":36.9,"encode_ms":9537.7,"decode_ms":19979.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":29562.2,"xrt_wall":1.2,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":13568.6,"xrt_compute":0.81,"load_ms":1119.3,"mel_ms":26.4,"encode_ms":9549.7,"decode_ms":3992.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":13576.5,"xrt_wall":0.81,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":31331.9,"xrt_compute":1.13,"load_ms":1143.1,"mel_ms":31.3,"encode_ms":9839.8,"decode_ms":21460.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":31340.0,"xrt_wall":1.13,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":13998.7,"xrt_compute":0.79,"load_ms":1912.5,"mel_ms":34.6,"encode_ms":9845.5,"decode_ms":4118.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14006.5,"xrt_wall":0.79,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":15105.2,"xrt_compute":2.34,"load_ms":476.1,"mel_ms":42.7,"encode_ms":4507.4,"decode_ms":10555.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15106.1,"xrt_wall":2.34,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":6513.2,"xrt_compute":1.69,"load_ms":966.8,"mel_ms":40.0,"encode_ms":4499.7,"decode_ms":1973.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6514.0,"xrt_wall":1.69,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":15306.3,"xrt_compute":2.31,"load_ms":650.0,"mel_ms":44.1,"encode_ms":4425.3,"decode_ms":10836.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15307.3,"xrt_wall":2.31,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":6476.3,"xrt_compute":1.7,"load_ms":1426.4,"mel_ms":44.0,"encode_ms":4414.2,"decode_ms":2018.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6477.0,"xrt_wall":1.7,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-medium.en.json b/catalog/whisper-medium.en.json new file mode 100644 index 00000000..f4665d03 --- /dev/null +++ b/catalog/whisper-medium.en.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-medium.en", + "family": "whisper", + "display_name": "whisper-medium.en", + "params": 763873376, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-medium.en", + "upstream_commit": "2e98eb6", + "published_repo": "handy-computer/whisper-medium.en-gguf", + "docs_page": "whisper.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["segment"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-medium.en-F32.gguf","size_bytes":3057356864}, + {"quant":"F16","filename":"whisper-medium.en-F16.gguf","size_bytes":1541853248}, + {"quant":"Q8_0","filename":"whisper-medium.en-Q8_0.gguf","size_bytes":831460928}, + {"quant":"Q6_K","filename":"whisper-medium.en-Q6_K.gguf","size_bytes":647942912}, + {"quant":"Q5_K_M","filename":"whisper-medium.en-Q5_K_M.gguf","size_bytes":582669056}, + {"quant":"Q4_K_M","filename":"whisper-medium.en-Q4_K_M.gguf","size_bytes":504025856} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.88,"ci95":[4.32,5.56],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":457,"del":158,"ins":96},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.74,"ci95":[2.52,2.99],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":960,"del":337,"ins":155},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.73,"ci95":[2.51,2.98],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":959,"del":335,"ins":155},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.72,"ci95":[2.51,2.97],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":956,"del":337,"ins":151},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.82,"ci95":[2.54,3.15],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":956,"del":331,"ins":211},"empty_hyp":0,"utts_over_50pct":7,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.75,"ci95":[2.53,2.99],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":963,"del":325,"ins":168},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.91,"ci95":[2.55,3.43],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":976,"del":303,"ins":266},"empty_hyp":0,"utts_over_50pct":8,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4152.5,"xrt_compute":8.51,"load_ms":160.4,"mel_ms":5.2,"encode_ms":1561.3,"decode_ms":2586.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4152.9,"xrt_wall":8.51,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1886.6,"xrt_compute":5.83,"load_ms":145.2,"mel_ms":4.5,"encode_ms":1554.3,"decode_ms":327.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1887.0,"xrt_wall":5.83,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":3885.7,"xrt_compute":9.09,"load_ms":228.8,"mel_ms":5.2,"encode_ms":1406.1,"decode_ms":2474.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3886.2,"xrt_wall":9.09,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1748.5,"xrt_compute":6.29,"load_ms":207.1,"mel_ms":4.4,"encode_ms":1430.8,"decode_ms":313.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1749.0,"xrt_wall":6.29,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1266.5,"xrt_compute":27.9,"load_ms":288.7,"mel_ms":22.7,"encode_ms":157.2,"decode_ms":1086.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1266.8,"xrt_wall":27.89,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":270.2,"xrt_compute":40.72,"load_ms":173.3,"mel_ms":4.3,"encode_ms":141.2,"decode_ms":124.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":270.3,"xrt_wall":40.7,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":912.8,"xrt_compute":38.71,"load_ms":244.2,"mel_ms":4.8,"encode_ms":150.6,"decode_ms":757.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":913.0,"xrt_wall":38.7,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":291.6,"xrt_compute":37.73,"load_ms":216.8,"mel_ms":4.1,"encode_ms":150.1,"decode_ms":137.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":291.7,"xrt_wall":37.71,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":14006.0,"xrt_compute":2.52,"load_ms":379.8,"mel_ms":29.5,"encode_ms":4652.3,"decode_ms":9324.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14011.9,"xrt_wall":2.52,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":5936.1,"xrt_compute":1.85,"load_ms":602.6,"mel_ms":24.4,"encode_ms":4586.6,"decode_ms":1325.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5942.0,"xrt_wall":1.85,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":14993.8,"xrt_compute":2.36,"load_ms":597.4,"mel_ms":35.5,"encode_ms":4815.1,"decode_ms":10143.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14999.9,"xrt_wall":2.36,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":6204.4,"xrt_compute":1.77,"load_ms":962.0,"mel_ms":24.7,"encode_ms":4750.4,"decode_ms":1429.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6210.7,"xrt_wall":1.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":7058.2,"xrt_compute":5.01,"load_ms":261.4,"mel_ms":41.5,"encode_ms":2096.7,"decode_ms":4920.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7058.8,"xrt_wall":5.01,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2647.3,"xrt_compute":4.16,"load_ms":458.5,"mel_ms":35.0,"encode_ms":1949.5,"decode_ms":662.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2647.6,"xrt_wall":4.15,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7461.7,"xrt_compute":4.74,"load_ms":378.5,"mel_ms":42.7,"encode_ms":2088.5,"decode_ms":5330.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7462.2,"xrt_wall":4.73,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2705.7,"xrt_compute":4.07,"load_ms":772.5,"mel_ms":36.3,"encode_ms":1923.9,"decode_ms":745.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2706.1,"xrt_wall":4.06,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-medium.json b/catalog/whisper-medium.json new file mode 100644 index 00000000..ae528b4e --- /dev/null +++ b/catalog/whisper-medium.json @@ -0,0 +1,160 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-medium", + "family": "whisper", + "display_name": "whisper-medium", + "params": 763874400, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-medium", + "upstream_commit": "abdf7c3", + "published_repo": "handy-computer/whisper-medium-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "zh" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-medium-F32.gguf","size_bytes":3057437088}, + {"quant":"F16","filename":"whisper-medium-F16.gguf","size_bytes":1541931424}, + {"quant":"Q8_0","filename":"whisper-medium-Q8_0.gguf","size_bytes":831538144}, + {"quant":"Q6_K","filename":"whisper-medium-Q6_K.gguf","size_bytes":648019904}, + {"quant":"Q5_K_M","filename":"whisper-medium-Q5_K_M.gguf","size_bytes":582746048}, + {"quant":"Q4_K_M","filename":"whisper-medium-Q4_K_M.gguf","size_bytes":504102848} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":47.33,"ci95":[45.26,49.54],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2217,"del":331,"ins":222},"empty_hyp":1,"utts_over_50pct":109,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":116.27,"ci95":[112.69,120.65],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4276,"del":4283,"ins":1398},"empty_hyp":13,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":21.9,"ci95":[20.4,23.52],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1384,"del":328,"ins":92},"empty_hyp":1,"utts_over_50pct":24,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":101.63,"ci95":[100.95,102.58],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":18442,"del":28506,"ins":812},"empty_hyp":32,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":34.57,"ci95":[33.63,35.47],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5326,"del":471,"ins":399},"empty_hyp":0,"utts_over_50pct":123,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":61.87,"ci95":[61.01,62.73],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11460,"del":744,"ins":681},"empty_hyp":0,"utts_over_50pct":761,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":23.0,"ci95":[22.0,24.03],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2645,"del":358,"ins":257},"empty_hyp":0,"utts_over_50pct":27,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":102.52,"ci95":[101.97,103.14],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":22763,"del":21923,"ins":1214},"empty_hyp":17,"utts_over_50pct":920,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":26.69,"ci95":[25.6,27.94],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4160,"del":544,"ins":321},"empty_hyp":0,"utts_over_50pct":69,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":8.02,"ci95":[7.47,8.58],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1422,"del":284,"ins":237},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":22.85,"ci95":[21.93,23.89],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2670,"del":209,"ins":250},"empty_hyp":0,"utts_over_50pct":31,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":40.08,"ci95":[39.13,41.07],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8213,"del":907,"ins":1381},"empty_hyp":2,"utts_over_50pct":222,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":21.17,"ci95":[20.35,22.01],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3320,"del":385,"ins":562},"empty_hyp":0,"utts_over_50pct":35,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":6.23,"ci95":[5.74,6.71],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":853,"del":129,"ins":184},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":20.06,"ci95":[19.06,21.13],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2638,"del":242,"ins":164},"empty_hyp":0,"utts_over_50pct":29,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":4.64,"ci95":[4.2,5.15],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":464,"del":107,"ins":105},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":3.8,"ci95":[3.42,4.2],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":658,"del":131,"ins":97},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":31.52,"ci95":[30.39,32.63],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3643,"del":367,"ins":680},"empty_hyp":0,"utts_over_50pct":111,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":42.57,"ci95":[41.43,43.78],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6452,"del":549,"ins":1556},"empty_hyp":1,"utts_over_50pct":241,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":14.67,"ci95":[13.94,15.57],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1746,"del":178,"ins":247},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":18.36,"ci95":[17.6,19.2],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3631,"del":666,"ins":556},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":8.07,"ci95":[7.43,8.8],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1101,"del":201,"ins":151},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":22.24,"ci95":[21.48,23.08],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4030,"del":433,"ins":471},"empty_hyp":0,"utts_over_50pct":27,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":104.11,"ci95":[103.71,104.53],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":41921,"del":2955,"ins":1960},"empty_hyp":1,"utts_over_50pct":1000,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":95.12,"ci95":[93.21,97.17],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9066,"del":4896,"ins":1194},"empty_hyp":9,"utts_over_50pct":620,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":33.69,"ci95":[32.08,35.46],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3633,"del":677,"ins":353},"empty_hyp":3,"utts_over_50pct":140,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":26.09,"ci95":[24.77,27.63],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3102,"del":1185,"ins":606},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":21.19,"ci95":[20.29,22.07],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3039,"del":425,"ins":298},"empty_hyp":0,"utts_over_50pct":33,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":26.07,"ci95":[25.08,27.04],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3554,"del":389,"ins":509},"empty_hyp":0,"utts_over_50pct":61,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":58.42,"ci95":[57.36,59.46],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8851,"del":1166,"ins":594},"empty_hyp":0,"utts_over_50pct":613,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":10.79,"ci95":[10.01,11.69],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1148,"del":131,"ins":184},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":51.6,"ci95":[47.14,56.58],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":442,"del":36,"ins":55},"empty_hyp":0,"utts_over_50pct":25,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":4.17,"ci95":[3.75,4.65],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":670,"del":121,"ins":96},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":7.35,"ci95":[6.79,7.91],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1733,"del":353,"ins":313},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":128.01,"ci95":[123.78,132.1],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10194,"del":6941,"ins":4804},"empty_hyp":6,"utts_over_50pct":979,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":53.09,"ci95":[51.93,54.34],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6733,"del":624,"ins":615},"empty_hyp":0,"utts_over_50pct":446,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":87.3,"ci95":[85.89,88.77],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13965,"del":21068,"ins":1752},"empty_hyp":89,"utts_over_50pct":767,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":5.46,"ci95":[4.53,6.45],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":405,"del":520,"ins":40},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":98.33,"ci95":[96.12,101.1],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15744,"del":2687,"ins":2180},"empty_hyp":0,"utts_over_50pct":932,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":92.09,"ci95":[89.41,94.9],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6091,"del":1684,"ins":1848},"empty_hyp":4,"utts_over_50pct":457,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":43.34,"ci95":[42.26,44.4],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6604,"del":372,"ins":450},"empty_hyp":0,"utts_over_50pct":309,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":33.58,"ci95":[32.56,34.62],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4501,"del":267,"ins":409},"empty_hyp":0,"utts_over_50pct":106,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","quant":"Q8_0","metric":"wer","err_pct":95.79,"ci95":[92.89,98.86],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13857,"del":11921,"ins":3939},"empty_hyp":33,"utts_over_50pct":867,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":24.75,"ci95":[23.74,25.86],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4134,"del":550,"ins":534},"empty_hyp":0,"utts_over_50pct":61,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ml","quant":"Q8_0","metric":"wer","err_pct":101.0,"ci95":[100.56,101.54],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15995,"del":38186,"ins":597},"empty_hyp":68,"utts_over_50pct":958,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mn","quant":"Q8_0","metric":"wer","err_pct":110.55,"ci95":[108.56,112.6],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13110,"del":5312,"ins":2065},"empty_hyp":4,"utts_over_50pct":949,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mr","quant":"Q8_0","metric":"wer","err_pct":58.43,"ci95":[56.53,60.21],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15064,"del":12384,"ins":2533},"empty_hyp":5,"utts_over_50pct":474,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":13.23,"ci95":[12.44,14.03],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1674,"del":129,"ins":149},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":85.06,"ci95":[84.12,86.01],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15307,"del":2503,"ins":1100},"empty_hyp":1,"utts_over_50pct":918,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nb","quant":"Q8_0","metric":"wer","err_pct":13.66,"ci95":[12.66,14.67],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":846,"del":97,"ins":98},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ne","quant":"Q8_0","metric":"wer","err_pct":54.08,"ci95":[52.85,55.3],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11663,"del":3261,"ins":3109},"empty_hyp":5,"utts_over_50pct":396,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":10.4,"ci95":[9.48,11.33],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":652,"del":94,"ins":127},"empty_hyp":0,"utts_over_50pct":0,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"oc","quant":"Q8_0","metric":"wer","err_pct":81.55,"ci95":[80.44,82.77],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17947,"del":1612,"ins":1971},"empty_hyp":0,"utts_over_50pct":958,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pa","quant":"Q8_0","metric":"wer","err_pct":103.19,"ci95":[102.42,104.2],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":21364,"del":4096,"ins":859},"empty_hyp":2,"utts_over_50pct":574,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":8.59,"ci95":[7.97,9.27],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1004,"del":103,"ins":129},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ps","quant":"Q8_0","metric":"wer","err_pct":105.77,"ci95":[103.72,108.13],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6456,"del":6602,"ins":920},"empty_hyp":4,"utts_over_50pct":512,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":5.07,"ci95":[4.61,5.57],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":763,"del":125,"ins":200},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":24.17,"ci95":[23.1,25.29],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4326,"del":362,"ins":339},"empty_hyp":0,"utts_over_50pct":63,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":7.3,"ci95":[6.71,7.85],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":922,"del":88,"ins":92},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sd","quant":"Q8_0","metric":"wer","err_pct":132.27,"ci95":[129.31,135.25],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15137,"del":8469,"ins":7659},"empty_hyp":7,"utts_over_50pct":980,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":18.73,"ci95":[17.83,19.63],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2411,"del":238,"ins":207},"empty_hyp":3,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":33.98,"ci95":[32.85,34.99],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4555,"del":457,"ins":600},"empty_hyp":0,"utts_over_50pct":129,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sn","quant":"Q8_0","metric":"wer","err_pct":134.28,"ci95":[128.93,140.03],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9511,"del":6457,"ins":5516},"empty_hyp":20,"utts_over_50pct":925,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"so","quant":"Q8_0","metric":"wer","err_pct":102.63,"ci95":[101.53,104.07],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12572,"del":11593,"ins":657},"empty_hyp":36,"utts_over_50pct":1019,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sr","quant":"Q8_0","metric":"wer","err_pct":55.14,"ci95":[51.94,58.16],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7014,"del":399,"ins":225},"empty_hyp":0,"utts_over_50pct":315,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":12.47,"ci95":[11.7,13.21],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1487,"del":235,"ins":196},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sw","quant":"Q8_0","metric":"wer","err_pct":57.09,"ci95":[55.23,58.92],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4195,"del":943,"ins":569},"empty_hyp":0,"utts_over_50pct":299,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ta","quant":"Q8_0","metric":"wer","err_pct":23.72,"ci95":[22.33,25.5],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4480,"del":1412,"ins":2094},"empty_hyp":0,"utts_over_50pct":23,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"te","quant":"Q8_0","metric":"wer","err_pct":102.69,"ci95":[101.92,103.57],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11459,"del":10956,"ins":800},"empty_hyp":56,"utts_over_50pct":472,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tg","quant":"Q8_0","metric":"wer","err_pct":76.32,"ci95":[75.16,77.38],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8545,"del":669,"ins":734},"empty_hyp":0,"utts_over_50pct":578,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":16.08,"ci95":[15.23,16.99],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7429,"del":4562,"ins":3687},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":9.35,"ci95":[8.65,10.19],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":937,"del":147,"ins":159},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":11.59,"ci95":[10.77,12.39],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1333,"del":186,"ins":146},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ur","quant":"Q8_0","metric":"wer","err_pct":28.67,"ci95":[27.22,30.21],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1988,"del":188,"ins":260},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uz","quant":"Q8_0","metric":"wer","err_pct":115.02,"ci95":[111.22,119.5],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11393,"del":3591,"ins":3382},"empty_hyp":9,"utts_over_50pct":861,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":13.74,"ci95":[12.94,14.63],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2896,"del":391,"ins":289},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yo","quant":"Q8_0","metric":"wer","err_pct":109.0,"ci95":[106.23,111.93],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13499,"del":8066,"ins":2588},"empty_hyp":2,"utts_over_50pct":831,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":13.13,"ci95":[11.97,14.25],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2999,"del":1485,"ins":197},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":2.64,"ci95":[2.41,2.91],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1001,"del":198,"ins":200},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.63,"ci95":[2.4,2.9],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1000,"del":197,"ins":200},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":2.64,"ci95":[2.41,2.91],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1003,"del":200,"ins":199},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.59,"ci95":[2.39,2.79],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":998,"del":213,"ins":160},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":2.62,"ci95":[2.42,2.83],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1013,"del":218,"ins":159},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":2.59,"ci95":[2.39,2.8],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1013,"del":207,"ins":153},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":108.92,"ci95":[106.01,112.36],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":20297,"del":42057,"ins":7036},"empty_hyp":232,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":101.16,"ci95":[100.74,101.59],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":34910,"del":2014,"ins":936},"empty_hyp":0,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"my","quant":"Q8_0","metric":"cer","err_pct":117.51,"ci95":[114.11,121.31],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":27200,"del":30958,"ins":10578},"empty_hyp":9,"utts_over_50pct":880,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":73.01,"ci95":[71.38,74.72],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"5d0c03e","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":8442,"del":1353,"ins":943},"empty_hyp":5,"utts_over_50pct":638} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4404.3,"xrt_compute":8.02,"load_ms":162.6,"mel_ms":5.2,"encode_ms":1571.2,"decode_ms":2828.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4404.9,"xrt_wall":8.02,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2083.2,"xrt_compute":5.28,"load_ms":144.5,"mel_ms":4.4,"encode_ms":1545.7,"decode_ms":533.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2083.7,"xrt_wall":5.28,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":4081.9,"xrt_compute":8.66,"load_ms":232.2,"mel_ms":5.1,"encode_ms":1420.8,"decode_ms":2656.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4082.4,"xrt_wall":8.65,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1883.5,"xrt_compute":5.84,"load_ms":204.1,"mel_ms":4.2,"encode_ms":1402.4,"decode_ms":476.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1884.0,"xrt_wall":5.84,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":891.0,"xrt_compute":39.65,"load_ms":163.2,"mel_ms":4.9,"encode_ms":148.6,"decode_ms":737.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":891.2,"xrt_wall":39.64,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":325.9,"xrt_compute":33.76,"load_ms":146.2,"mel_ms":4.2,"encode_ms":155.0,"decode_ms":166.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":326.0,"xrt_wall":33.74,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1020.7,"xrt_compute":34.62,"load_ms":235.9,"mel_ms":6.0,"encode_ms":161.5,"decode_ms":853.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1020.9,"xrt_wall":34.61,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":343.4,"xrt_compute":32.03,"load_ms":403.4,"mel_ms":7.4,"encode_ms":146.2,"decode_ms":189.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":343.7,"xrt_wall":32.01,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":14669.3,"xrt_compute":2.41,"load_ms":383.2,"mel_ms":29.9,"encode_ms":4652.3,"decode_ms":9987.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":14675.4,"xrt_wall":2.41,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":6648.9,"xrt_compute":1.65,"load_ms":599.9,"mel_ms":27.1,"encode_ms":4571.1,"decode_ms":2050.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6654.7,"xrt_wall":1.65,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":15600.5,"xrt_compute":2.26,"load_ms":603.3,"mel_ms":37.1,"encode_ms":4817.8,"decode_ms":10745.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":15606.6,"xrt_wall":2.26,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":6934.1,"xrt_compute":1.59,"load_ms":968.9,"mel_ms":25.3,"encode_ms":4773.7,"decode_ms":2135.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":6940.4,"xrt_wall":1.58,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":7403.3,"xrt_compute":4.77,"load_ms":228.3,"mel_ms":36.5,"encode_ms":2135.7,"decode_ms":5231.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7404.0,"xrt_wall":4.77,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":3011.1,"xrt_compute":3.65,"load_ms":458.9,"mel_ms":37.2,"encode_ms":1973.8,"decode_ms":1000.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3011.8,"xrt_wall":3.65,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":7767.5,"xrt_compute":4.55,"load_ms":345.3,"mel_ms":37.4,"encode_ms":2099.6,"decode_ms":5630.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":7768.3,"xrt_wall":4.55,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":3041.6,"xrt_compute":3.62,"load_ms":740.4,"mel_ms":39.2,"encode_ms":1944.9,"decode_ms":1057.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":3042.2,"xrt_wall":3.62,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-small.en.json b/catalog/whisper-small.en.json new file mode 100644 index 00000000..79094e41 --- /dev/null +++ b/catalog/whisper-small.en.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-small.en", + "family": "whisper", + "display_name": "whisper-small.en", + "params": 241750624, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-small.en", + "upstream_commit": "e872752", + "published_repo": "handy-computer/whisper-small.en-gguf", + "docs_page": "whisper.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["segment"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-small.en-F32.gguf","size_bytes":968835616}, + {"quant":"F16","filename":"whisper-small.en-F16.gguf","size_bytes":492810784}, + {"quant":"Q8_0","filename":"whisper-small.en-Q8_0.gguf","size_bytes":269674144}, + {"quant":"Q6_K","filename":"whisper-small.en-Q6_K.gguf","size_bytes":212030528}, + {"quant":"Q5_K_M","filename":"whisper-small.en-Q5_K_M.gguf","size_bytes":193672256}, + {"quant":"Q4_K_M","filename":"whisper-small.en-Q4_K_M.gguf","size_bytes":171553856} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":6.14,"ci95":[5.47,6.87],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":576,"del":176,"ins":142},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":3.11,"ci95":[2.83,3.46],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1170,"del":229,"ins":248},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":2.97,"ci95":[2.77,3.2],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1168,"del":230,"ins":176},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.09,"ci95":[2.81,3.45],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1168,"del":227,"ins":242},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":2.97,"ci95":[2.77,3.21],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1174,"del":226,"ins":174},"empty_hyp":0,"utts_over_50pct":9,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":3.11,"ci95":[2.82,3.49],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1173,"del":234,"ins":244},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":3.09,"ci95":[2.86,3.34],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1194,"del":243,"ins":199},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1598.8,"xrt_compute":22.1,"load_ms":81.5,"mel_ms":5.5,"encode_ms":554.4,"decode_ms":1038.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1599.2,"xrt_wall":22.09,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":953.9,"xrt_compute":11.53,"load_ms":86.3,"mel_ms":5.1,"encode_ms":573.1,"decode_ms":375.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":954.3,"xrt_wall":11.53,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1391.4,"xrt_compute":25.39,"load_ms":113.7,"mel_ms":5.2,"encode_ms":480.9,"decode_ms":905.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1391.7,"xrt_wall":25.39,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":597.0,"xrt_compute":18.42,"load_ms":96.9,"mel_ms":4.3,"encode_ms":476.5,"decode_ms":116.2,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":597.4,"xrt_wall":18.41,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":387.5,"xrt_compute":91.18,"load_ms":90.4,"mel_ms":5.2,"encode_ms":47.5,"decode_ms":334.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":387.6,"xrt_wall":91.15,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":114.0,"xrt_compute":96.47,"load_ms":76.8,"mel_ms":4.1,"encode_ms":47.8,"decode_ms":62.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":114.1,"xrt_wall":96.38,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":384.3,"xrt_compute":91.93,"load_ms":108.2,"mel_ms":4.8,"encode_ms":46.2,"decode_ms":333.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":384.5,"xrt_wall":91.9,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":119.2,"xrt_compute":92.31,"load_ms":102.1,"mel_ms":4.1,"encode_ms":46.4,"decode_ms":68.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":119.3,"xrt_wall":92.22,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":4964.0,"xrt_compute":7.12,"load_ms":186.7,"mel_ms":38.1,"encode_ms":1477.1,"decode_ms":3448.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":4964.1,"xrt_wall":7.12,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1900.8,"xrt_compute":5.79,"load_ms":264.0,"mel_ms":37.6,"encode_ms":1389.6,"decode_ms":473.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1901.0,"xrt_wall":5.79,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":5085.7,"xrt_compute":6.95,"load_ms":248.1,"mel_ms":36.8,"encode_ms":1475.7,"decode_ms":3573.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5085.9,"xrt_wall":6.95,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1976.0,"xrt_compute":5.57,"load_ms":372.7,"mel_ms":31.4,"encode_ms":1414.5,"decode_ms":530.1,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1976.1,"xrt_wall":5.57,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2556.3,"xrt_compute":13.82,"load_ms":135.0,"mel_ms":36.0,"encode_ms":634.1,"decode_ms":1886.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2556.9,"xrt_wall":13.82,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":925.5,"xrt_compute":11.89,"load_ms":1648.8,"mel_ms":31.4,"encode_ms":603.1,"decode_ms":291.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":925.8,"xrt_wall":11.88,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2616.0,"xrt_compute":13.51,"load_ms":166.1,"mel_ms":36.9,"encode_ms":622.9,"decode_ms":1956.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2616.5,"xrt_wall":13.5,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":961.3,"xrt_compute":11.44,"load_ms":293.3,"mel_ms":32.5,"encode_ms":588.6,"decode_ms":340.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":961.6,"xrt_wall":11.44,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-small.json b/catalog/whisper-small.json new file mode 100644 index 00000000..314785e7 --- /dev/null +++ b/catalog/whisper-small.json @@ -0,0 +1,160 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-small", + "family": "whisper", + "display_name": "whisper-small", + "params": 241751392, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-small", + "upstream_commit": "973afd2", + "published_repo": "handy-computer/whisper-small-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "zh" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-small-F32.gguf","size_bytes":968914848}, + {"quant":"F16","filename":"whisper-small-F16.gguf","size_bytes":492888480}, + {"quant":"Q8_0","filename":"whisper-small-Q8_0.gguf","size_bytes":269751136}, + {"quant":"Q6_K","filename":"whisper-small-Q6_K.gguf","size_bytes":212107328}, + {"quant":"Q5_K_M","filename":"whisper-small-Q5_K_M.gguf","size_bytes":193749056}, + {"quant":"Q4_K_M","filename":"whisper-small-Q4_K_M.gguf","size_bytes":171630656} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":62.2,"ci95":[59.75,64.6],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2942,"del":371,"ins":327},"empty_hyp":1,"utts_over_50pct":184,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":128.3,"ci95":[123.47,133.78],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7256,"del":1300,"ins":2432},"empty_hyp":5,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":32.15,"ci95":[30.52,33.87],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2186,"del":338,"ins":124},"empty_hyp":0,"utts_over_50pct":62,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":104.15,"ci95":[103.27,105.12],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":27225,"del":19755,"ins":1962},"empty_hyp":11,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":51.64,"ci95":[50.65,52.57],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7924,"del":703,"ins":629},"empty_hyp":0,"utts_over_50pct":461,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":76.85,"ci95":[75.97,77.8],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13938,"del":972,"ins":1095},"empty_hyp":0,"utts_over_50pct":940,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":40.39,"ci95":[39.26,41.61],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4549,"del":565,"ins":610},"empty_hyp":0,"utts_over_50pct":184,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":103.04,"ci95":[102.14,104.02],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":20511,"del":24224,"ins":1398},"empty_hyp":14,"utts_over_50pct":920,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":42.35,"ci95":[41.23,43.44],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6420,"del":792,"ins":761},"empty_hyp":0,"utts_over_50pct":284,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":14.19,"ci95":[13.47,14.81],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2599,"del":377,"ins":463},"empty_hyp":0,"utts_over_50pct":10,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":40.57,"ci95":[39.32,41.83],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4568,"del":491,"ins":496},"empty_hyp":0,"utts_over_50pct":180,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":64.47,"ci95":[63.31,65.55],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12549,"del":1076,"ins":3267},"empty_hyp":2,"utts_over_50pct":795,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":35.52,"ci95":[34.52,36.61],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5540,"del":637,"ins":983},"empty_hyp":1,"utts_over_50pct":179,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":9.86,"ci95":[9.23,10.49],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1401,"del":228,"ins":217},"empty_hyp":0,"utts_over_50pct":1,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":33.98,"ci95":[32.74,35.34],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4373,"del":466,"ins":317},"empty_hyp":0,"utts_over_50pct":98,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":6.51,"ci95":[5.88,7.22],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":626,"del":158,"ins":164},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":5.92,"ci95":[5.49,6.37],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1020,"del":192,"ins":169},"empty_hyp":0,"utts_over_50pct":3,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":54.79,"ci95":[53.58,55.93],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6485,"del":733,"ins":935},"empty_hyp":0,"utts_over_50pct":527,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":58.44,"ci95":[57.1,59.92],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9373,"del":771,"ins":1602},"empty_hyp":0,"utts_over_50pct":590,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":26.48,"ci95":[25.4,27.59],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3107,"del":358,"ins":452},"empty_hyp":0,"utts_over_50pct":75,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":28.52,"ci95":[27.52,29.53],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5541,"del":943,"ins":1053},"empty_hyp":0,"utts_over_50pct":81,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":13.3,"ci95":[12.47,14.19],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1820,"del":267,"ins":306},"empty_hyp":0,"utts_over_50pct":11,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":32.72,"ci95":[31.82,33.69],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5810,"del":732,"ins":717},"empty_hyp":0,"utts_over_50pct":108,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":104.02,"ci95":[103.44,104.59],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":37575,"del":7389,"ins":1833},"empty_hyp":3,"utts_over_50pct":1000,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":94.21,"ci95":[92.9,95.9],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10866,"del":3080,"ins":1066},"empty_hyp":5,"utts_over_50pct":620,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":46.06,"ci95":[44.34,47.8],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5157,"del":766,"ins":453},"empty_hyp":1,"utts_over_50pct":289,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":42.05,"ci95":[39.97,44.32],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4697,"del":2288,"ins":901},"empty_hyp":0,"utts_over_50pct":81,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":36.05,"ci95":[35.0,37.15],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5195,"del":606,"ins":599},"empty_hyp":0,"utts_over_50pct":163,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":42.39,"ci95":[41.32,43.49],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5727,"del":650,"ins":861},"empty_hyp":0,"utts_over_50pct":269,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":87.84,"ci95":[86.83,88.93],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12647,"del":2219,"ins":1087},"empty_hyp":0,"utts_over_50pct":925,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":18.02,"ci95":[17.02,19.15],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1938,"del":197,"ins":309},"empty_hyp":0,"utts_over_50pct":33,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":74.54,"ci95":[69.87,80.21],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":623,"del":42,"ins":105},"empty_hyp":0,"utts_over_50pct":43,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":7.97,"ci95":[7.4,8.52],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1284,"del":213,"ins":196},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":12.81,"ci95":[12.05,13.52],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3160,"del":507,"ins":514},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":130.64,"ci95":[125.94,135.97],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9677,"del":7454,"ins":5259},"empty_hyp":17,"utts_over_50pct":979,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":73.54,"ci95":[72.52,74.64],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9347,"del":918,"ins":777},"empty_hyp":0,"utts_over_50pct":794,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":99.65,"ci95":[98.84,100.59],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15683,"del":24930,"ins":1374},"empty_hyp":59,"utts_over_50pct":838,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":7.7,"ci95":[6.65,8.76],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":751,"del":548,"ins":62},"empty_hyp":0,"utts_over_50pct":2,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":110.02,"ci95":[107.69,112.73],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14841,"del":5194,"ins":3025},"empty_hyp":3,"utts_over_50pct":932,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":98.65,"ci95":[95.84,101.91],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7129,"del":1357,"ins":1822},"empty_hyp":1,"utts_over_50pct":472,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":70.21,"ci95":[68.92,71.71],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10337,"del":639,"ins":1053},"empty_hyp":0,"utts_over_50pct":856,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":57.42,"ci95":[56.2,58.69],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7448,"del":609,"ins":795},"empty_hyp":0,"utts_over_50pct":533,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","quant":"Q8_0","metric":"wer","err_pct":62.82,"ci95":[61.29,64.46],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14048,"del":2879,"ins":2561},"empty_hyp":0,"utts_over_50pct":714,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":41.53,"ci95":[40.36,42.67],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6896,"del":760,"ins":1099},"empty_hyp":0,"utts_over_50pct":283,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ml","quant":"Q8_0","metric":"wer","err_pct":100.35,"ci95":[100.05,100.79],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12515,"del":41676,"ins":237},"empty_hyp":30,"utts_over_50pct":958,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mn","quant":"Q8_0","metric":"wer","err_pct":142.37,"ci95":[136.73,148.04],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11557,"del":6971,"ins":7856},"empty_hyp":5,"utts_over_50pct":949,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mr","quant":"Q8_0","metric":"wer","err_pct":63.66,"ci95":[62.16,65.15],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":18096,"del":11034,"ins":3537},"empty_hyp":1,"utts_over_50pct":686,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":21.48,"ci95":[20.38,22.72],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2669,"del":192,"ins":308},"empty_hyp":0,"utts_over_50pct":34,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":97.32,"ci95":[95.64,99.2],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14303,"del":5979,"ins":1355},"empty_hyp":1,"utts_over_50pct":926,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nb","quant":"Q8_0","metric":"wer","err_pct":25.53,"ci95":[24.2,26.95],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1546,"del":193,"ins":206},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ne","quant":"Q8_0","metric":"wer","err_pct":70.48,"ci95":[69.13,71.94],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14229,"del":5862,"ins":3409},"empty_hyp":4,"utts_over_50pct":653,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":18.48,"ci95":[17.29,19.57],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1183,"del":138,"ins":231},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"oc","quant":"Q8_0","metric":"wer","err_pct":90.43,"ci95":[87.82,93.27],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":16820,"del":3823,"ins":3231},"empty_hyp":3,"utts_over_50pct":973,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pa","quant":"Q8_0","metric":"wer","err_pct":101.31,"ci95":[100.72,102.18],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15183,"del":10299,"ins":358},"empty_hyp":4,"utts_over_50pct":574,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":16.82,"ci95":[15.87,17.8],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1938,"del":216,"ins":266},"empty_hyp":0,"utts_over_50pct":18,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ps","quant":"Q8_0","metric":"wer","err_pct":93.61,"ci95":[92.96,94.25],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9931,"del":2216,"ins":224},"empty_hyp":1,"utts_over_50pct":512,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":7.65,"ci95":[6.96,8.31],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1180,"del":185,"ins":278},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":33.88,"ci95":[32.65,35.07],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5666,"del":674,"ins":707},"empty_hyp":0,"utts_over_50pct":160,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":11.9,"ci95":[11.19,12.62],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1488,"del":163,"ins":146},"empty_hyp":0,"utts_over_50pct":4,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sd","quant":"Q8_0","metric":"wer","err_pct":112.24,"ci95":[110.08,114.51],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17543,"del":6065,"ins":2923},"empty_hyp":6,"utts_over_50pct":980,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":36.05,"ci95":[34.98,37.21],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4592,"del":442,"ins":462},"empty_hyp":2,"utts_over_50pct":136,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":52.7,"ci95":[51.43,53.95],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6902,"del":697,"ins":1104},"empty_hyp":0,"utts_over_50pct":441,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sn","quant":"Q8_0","metric":"wer","err_pct":132.9,"ci95":[128.31,138.18],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8931,"del":6863,"ins":5470},"empty_hyp":11,"utts_over_50pct":925,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"so","quant":"Q8_0","metric":"wer","err_pct":103.39,"ci95":[102.69,104.15],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":19764,"del":4416,"ins":824},"empty_hyp":3,"utts_over_50pct":1019,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sr","quant":"Q8_0","metric":"wer","err_pct":44.91,"ci95":[42.81,47.03],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5295,"del":447,"ins":479},"empty_hyp":0,"utts_over_50pct":191,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":23.1,"ci95":[21.94,24.32],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2667,"del":409,"ins":478},"empty_hyp":0,"utts_over_50pct":51,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sw","quant":"Q8_0","metric":"wer","err_pct":76.22,"ci95":[74.6,77.92],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5696,"del":914,"ins":1010},"empty_hyp":2,"utts_over_50pct":447,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ta","quant":"Q8_0","metric":"wer","err_pct":35.35,"ci95":[33.42,37.78],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6572,"del":1889,"ins":3438},"empty_hyp":0,"utts_over_50pct":67,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"te","quant":"Q8_0","metric":"wer","err_pct":102.23,"ci95":[101.21,103.53],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7500,"del":15074,"ins":537},"empty_hyp":55,"utts_over_50pct":472,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tg","quant":"Q8_0","metric":"wer","err_pct":86.78,"ci95":[85.6,87.89],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9590,"del":681,"ins":1040},"empty_hyp":0,"utts_over_50pct":593,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":22.55,"ci95":[21.71,23.49],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11005,"del":6214,"ins":4771},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":15.95,"ci95":[14.93,17.0],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1651,"del":202,"ins":268},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":20.42,"ci95":[19.43,21.41],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2378,"del":272,"ins":285},"empty_hyp":0,"utts_over_50pct":29,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ur","quant":"Q8_0","metric":"wer","err_pct":39.75,"ci95":[38.23,41.19],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2715,"del":339,"ins":324},"empty_hyp":0,"utts_over_50pct":49,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uz","quant":"Q8_0","metric":"wer","err_pct":114.92,"ci95":[111.78,118.41],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12794,"del":2162,"ins":3394},"empty_hyp":15,"utts_over_50pct":862,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":22.47,"ci95":[21.57,23.47],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5010,"del":382,"ins":455},"empty_hyp":0,"utts_over_50pct":41,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yo","quant":"Q8_0","metric":"wer","err_pct":118.5,"ci95":[114.83,122.44],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13451,"del":8340,"ins":4468},"empty_hyp":9,"utts_over_50pct":831,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":23.06,"ci95":[21.79,24.35],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6447,"del":1536,"ins":239},"empty_hyp":0,"utts_over_50pct":60,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":3.34,"ci95":[3.11,3.58],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1325,"del":245,"ins":199},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":3.33,"ci95":[3.11,3.57],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1324,"del":245,"ins":198},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":3.33,"ci95":[3.11,3.56],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1327,"del":243,"ins":195},"empty_hyp":0,"utts_over_50pct":12,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":3.33,"ci95":[3.11,3.56],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1331,"del":238,"ins":196},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":3.37,"ci95":[3.15,3.6],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1328,"del":240,"ins":219},"empty_hyp":0,"utts_over_50pct":14,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":3.4,"ci95":[3.18,3.64],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1367,"del":235,"ins":200},"empty_hyp":0,"utts_over_50pct":15,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":116.96,"ci95":[113.31,120.85],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":28044,"del":34512,"ins":11959},"empty_hyp":29,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":101.52,"ci95":[100.96,102.18],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":35194,"del":1900,"ins":900},"empty_hyp":0,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"my","quant":"Q8_0","metric":"cer","err_pct":132.91,"ci95":[127.85,138.7],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":33204,"del":24938,"ins":19602},"empty_hyp":10,"utts_over_50pct":880,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":94.42,"ci95":[91.78,97.26],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"5d0c03e","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":9612,"del":2113,"ins":2162},"empty_hyp":4,"utts_over_50pct":714} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":1605.5,"xrt_compute":22.01,"load_ms":83.5,"mel_ms":5.6,"encode_ms":543.6,"decode_ms":1056.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1605.9,"xrt_wall":22.0,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":718.6,"xrt_compute":15.31,"load_ms":81.0,"mel_ms":4.8,"encode_ms":525.8,"decode_ms":188.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":718.9,"xrt_wall":15.3,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":1455.0,"xrt_compute":24.28,"load_ms":104.4,"mel_ms":5.2,"encode_ms":479.2,"decode_ms":970.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1455.3,"xrt_wall":24.28,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":647.9,"xrt_compute":16.98,"load_ms":98.9,"mel_ms":4.3,"encode_ms":479.2,"decode_ms":164.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":648.2,"xrt_wall":16.97,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":394.0,"xrt_compute":89.68,"load_ms":93.4,"mel_ms":4.8,"encode_ms":47.8,"decode_ms":341.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":394.2,"xrt_wall":89.63,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":122.8,"xrt_compute":89.56,"load_ms":89.5,"mel_ms":4.1,"encode_ms":46.8,"decode_ms":71.9,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":123.0,"xrt_wall":89.44,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":400.8,"xrt_compute":88.15,"load_ms":113.5,"mel_ms":4.8,"encode_ms":45.8,"decode_ms":350.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":401.0,"xrt_wall":88.11,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":123.1,"xrt_compute":89.39,"load_ms":97.3,"mel_ms":4.2,"encode_ms":45.5,"decode_ms":73.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":123.2,"xrt_wall":89.29,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":5163.7,"xrt_compute":6.84,"load_ms":181.3,"mel_ms":39.0,"encode_ms":1459.5,"decode_ms":3665.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5163.8,"xrt_wall":6.84,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":2106.5,"xrt_compute":5.22,"load_ms":255.6,"mel_ms":30.7,"encode_ms":1388.6,"decode_ms":687.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2106.7,"xrt_wall":5.22,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":5311.9,"xrt_compute":6.65,"load_ms":242.1,"mel_ms":36.6,"encode_ms":1468.7,"decode_ms":3806.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":5312.2,"xrt_wall":6.65,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":2169.3,"xrt_compute":5.07,"load_ms":375.3,"mel_ms":34.7,"encode_ms":1415.4,"decode_ms":719.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2169.4,"xrt_wall":5.07,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":2736.8,"xrt_compute":12.91,"load_ms":134.5,"mel_ms":37.4,"encode_ms":634.1,"decode_ms":2065.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2737.3,"xrt_wall":12.91,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":1022.3,"xrt_compute":10.76,"load_ms":214.7,"mel_ms":32.8,"encode_ms":604.0,"decode_ms":385.5,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1022.6,"xrt_wall":10.76,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":2811.3,"xrt_compute":12.57,"load_ms":164.8,"mel_ms":37.3,"encode_ms":619.4,"decode_ms":2154.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":2811.8,"xrt_wall":12.57,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":1017.0,"xrt_compute":10.82,"load_ms":292.5,"mel_ms":33.4,"encode_ms":588.2,"decode_ms":395.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":1017.3,"xrt_wall":10.81,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-tiny.en.json b/catalog/whisper-tiny.en.json new file mode 100644 index 00000000..1754775d --- /dev/null +++ b/catalog/whisper-tiny.en.json @@ -0,0 +1,69 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-tiny.en", + "family": "whisper", + "display_name": "whisper-tiny.en", + "params": 37776736, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-tiny.en", + "upstream_commit": "87c7102", + "published_repo": "handy-computer/whisper-tiny.en-gguf", + "docs_page": "whisper.md", + "languages": ["en"], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":false}, + "lang_detect": {"supported":false}, + "timestamps": {"supported":true,"granularities":["segment"],"verified":false}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-tiny.en-F32.gguf","size_bytes":152920160}, + {"quant":"F16","filename":"whisper-tiny.en-F16.gguf","size_bytes":80058464}, + {"quant":"Q8_0","filename":"whisper-tiny.en-Q8_0.gguf","size_bytes":45904544}, + {"quant":"Q6_K","filename":"whisper-tiny.en-Q6_K.gguf","size_bytes":44761760}, + {"quant":"Q5_K_M","filename":"whisper-tiny.en-Q5_K_M.gguf","size_bytes":44135072}, + {"quant":"Q4_K_M","filename":"whisper-tiny.en-Q4_K_M.gguf","size_bytes":43545248} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":10.72,"ci95":[9.89,11.58],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1097,"del":172,"ins":292},"empty_hyp":0,"utts_over_50pct":6,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":5.77,"ci95":[5.46,6.09],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2290,"del":343,"ins":429},"empty_hyp":0,"utts_over_50pct":26,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":5.78,"ci95":[5.46,6.09],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2284,"del":352,"ins":429},"empty_hyp":0,"utts_over_50pct":25,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":5.72,"ci95":[5.41,6.03],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2275,"del":327,"ins":432},"empty_hyp":0,"utts_over_50pct":25,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":5.83,"ci95":[5.51,6.16],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2296,"del":345,"ins":449},"empty_hyp":0,"utts_over_50pct":26,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":5.91,"ci95":[5.58,6.23],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2324,"del":336,"ins":475},"empty_hyp":0,"utts_over_50pct":27,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":5.96,"ci95":[5.64,6.3],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2318,"del":345,"ins":499},"empty_hyp":0,"utts_over_50pct":29,"measurement_provenance":"legacy-published"} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":249.8,"xrt_compute":141.46,"load_ms":52.2,"mel_ms":4.8,"encode_ms":66.1,"decode_ms":178.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":250.0,"xrt_wall":141.35,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":97.5,"xrt_compute":112.83,"load_ms":49.7,"mel_ms":4.1,"encode_ms":67.8,"decode_ms":25.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":97.7,"xrt_wall":112.62,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":246.8,"xrt_compute":143.18,"load_ms":51.2,"mel_ms":4.8,"encode_ms":65.5,"decode_ms":176.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":247.0,"xrt_wall":143.06,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":93.2,"xrt_compute":118.02,"load_ms":49.2,"mel_ms":4.1,"encode_ms":63.4,"decode_ms":25.7,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":93.4,"xrt_wall":117.78,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":136.1,"xrt_compute":259.57,"load_ms":55.7,"mel_ms":4.8,"encode_ms":8.7,"decode_ms":122.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":136.2,"xrt_wall":259.33,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":37.8,"xrt_compute":291.22,"load_ms":50.5,"mel_ms":4.1,"encode_ms":8.6,"decode_ms":25.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":37.8,"xrt_wall":290.63,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":142.8,"xrt_compute":247.49,"load_ms":55.7,"mel_ms":4.8,"encode_ms":8.6,"decode_ms":129.3,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":142.9,"xrt_wall":247.25,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":38.1,"xrt_compute":288.65,"load_ms":51.1,"mel_ms":4.1,"encode_ms":8.5,"decode_ms":25.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":38.2,"xrt_wall":287.92,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":790.4,"xrt_compute":44.7,"load_ms":95.0,"mel_ms":28.7,"encode_ms":169.2,"decode_ms":592.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":790.5,"xrt_wall":44.7,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":282.9,"xrt_compute":38.88,"load_ms":112.8,"mel_ms":29.8,"encode_ms":161.3,"decode_ms":91.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":283.1,"xrt_wall":38.85,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":799.3,"xrt_compute":44.2,"load_ms":98.6,"mel_ms":39.4,"encode_ms":163.0,"decode_ms":596.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":799.4,"xrt_wall":44.2,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":287.7,"xrt_compute":38.24,"load_ms":115.5,"mel_ms":30.1,"encode_ms":161.5,"decode_ms":96.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":287.9,"xrt_wall":38.21,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":665.6,"xrt_compute":53.09,"load_ms":84.5,"mel_ms":36.6,"encode_ms":89.0,"decode_ms":539.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":665.9,"xrt_wall":53.06,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":251.0,"xrt_compute":43.82,"load_ms":99.3,"mel_ms":32.7,"encode_ms":90.0,"decode_ms":128.3,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":251.2,"xrt_wall":43.79,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":686.2,"xrt_compute":51.49,"load_ms":85.1,"mel_ms":38.0,"encode_ms":89.5,"decode_ms":558.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":686.5,"xrt_wall":51.46,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":253.0,"xrt_compute":43.47,"load_ms":106.2,"mel_ms":33.0,"encode_ms":88.2,"decode_ms":131.8,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":253.2,"xrt_wall":43.44,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/catalog/whisper-tiny.json b/catalog/whisper-tiny.json new file mode 100644 index 00000000..48308456 --- /dev/null +++ b/catalog/whisper-tiny.json @@ -0,0 +1,160 @@ +{ + "schema": "transcribe-catalog-v1", + "variant": "whisper-tiny", + "family": "whisper", + "display_name": "whisper-tiny", + "params": 37777120, + "license": { + "spdx": "apache-2.0", + "display": "Apache-2.0" + }, + "upstream_repo": "openai/whisper-tiny", + "upstream_commit": "169d4a4", + "published_repo": "handy-computer/whisper-tiny-gguf", + "docs_page": "whisper.md", + "languages": [ + "af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "bo", "br", "bs", + "ca", "cs", "cy", "da", "de", "el", "en", "es", "et", "eu", "fa", "fi", + "fo", "fr", "gl", "gu", "haw", "ha", "he", "hi", "hr", "ht", "hu", "hy", + "id", "is", "it", "ja", "jw", "ka", "kk", "km", "kn", "ko", "la", "lb", + "ln", "lo", "lt", "lv", "mg", "mi", "mk", "ml", "mn", "mr", "ms", "mt", + "my", "ne", "nl", "nn", "no", "oc", "pa", "pl", "ps", "pt", "ro", "ru", + "sa", "sd", "si", "sk", "sl", "sn", "so", "sq", "sr", "su", "sv", "sw", + "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "uk", "ur", "uz", "vi", + "yi", "yo", "zh" + ], + "long_form_strategy": "chunked-unbounded", + "capabilities": { + "transcribe": {"supported":true,"verified":false}, + "translate": {"supported":true,"verified":false,"targets":["en"],"pairs":null}, + "lang_detect": {"supported":true,"verified":false}, + "timestamps": {"supported":true,"verified":false,"granularities":["segment"]}, + "streaming": {"supported":false}, + "diarize": {"supported":false}, + "batching": {"supported":true,"verified":false} + }, + "downloads": [ + {"quant":"F32","filename":"whisper-tiny-F32.gguf","size_bytes":152997824}, + {"quant":"F16","filename":"whisper-tiny-F16.gguf","size_bytes":80135360}, + {"quant":"Q8_0","filename":"whisper-tiny-Q8_0.gguf","size_bytes":45981088}, + {"quant":"Q6_K","filename":"whisper-tiny-Q6_K.gguf","size_bytes":44838304}, + {"quant":"Q5_K_M","filename":"whisper-tiny-Q5_K_M.gguf","size_bytes":44211616}, + {"quant":"Q4_K_M","filename":"whisper-tiny-Q4_K_M.gguf","size_bytes":43621792} + ], + "accuracy_benchmarks": [ + {"dataset":"fleurs","split":"test","language":"af","quant":"Q8_0","metric":"wer","err_pct":95.3,"ci95":[92.75,97.9],"n_utts":264,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3884,"del":1188,"ins":505},"empty_hyp":0,"utts_over_50pct":261,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"am","quant":"Q8_0","metric":"wer","err_pct":122.14,"ci95":[118.55,125.97],"n_utts":516,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5399,"del":3160,"ins":1901},"empty_hyp":39,"utts_over_50pct":516,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ar","quant":"Q8_0","metric":"wer","err_pct":67.48,"ci95":[65.37,69.5],"n_utts":428,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4632,"del":603,"ins":323},"empty_hyp":0,"utts_over_50pct":341,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"as","quant":"Q8_0","metric":"wer","err_pct":100.56,"ci95":[100.15,101.06],"n_utts":984,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10468,"del":36519,"ins":270},"empty_hyp":28,"utts_over_50pct":984,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"az","quant":"Q8_0","metric":"wer","err_pct":94.73,"ci95":[93.35,96.23],"n_utts":923,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13375,"del":2022,"ins":1583},"empty_hyp":6,"utts_over_50pct":922,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"be","quant":"Q8_0","metric":"wer","err_pct":96.03,"ci95":[95.33,96.92],"n_utts":967,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17196,"del":1334,"ins":1469},"empty_hyp":1,"utts_over_50pct":966,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bg","quant":"Q8_0","metric":"wer","err_pct":84.87,"ci95":[83.52,86.32],"n_utts":658,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9159,"del":1504,"ins":1364},"empty_hyp":0,"utts_over_50pct":650,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bn","quant":"Q8_0","metric":"wer","err_pct":100.37,"ci95":[100.05,100.83],"n_utts":920,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12551,"del":32205,"ins":180},"empty_hyp":13,"utts_over_50pct":920,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"bs","quant":"Q8_0","metric":"wer","err_pct":87.22,"ci95":[85.97,88.45],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12685,"del":1895,"ins":1842},"empty_hyp":8,"utts_over_50pct":916,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ca","quant":"Q8_0","metric":"wer","err_pct":46.33,"ci95":[45.13,47.46],"n_utts":940,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8391,"del":1048,"ins":1786},"empty_hyp":0,"utts_over_50pct":358,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cs","quant":"Q8_0","metric":"wer","err_pct":87.17,"ci95":[85.89,88.48],"n_utts":723,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9464,"del":1164,"ins":1308},"empty_hyp":0,"utts_over_50pct":711,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"cy","quant":"Q8_0","metric":"wer","err_pct":116.68,"ci95":[114.19,119.56],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":18637,"del":4958,"ins":6979},"empty_hyp":3,"utts_over_50pct":1021,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"da","quant":"Q8_0","metric":"wer","err_pct":88.42,"ci95":[86.83,90.31],"n_utts":930,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12334,"del":2877,"ins":2611},"empty_hyp":2,"utts_over_50pct":890,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"de","quant":"Q8_0","metric":"wer","err_pct":31.56,"ci95":[30.53,32.7],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4414,"del":540,"ins":952},"empty_hyp":0,"utts_over_50pct":119,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"el","quant":"Q8_0","metric":"wer","err_pct":77.76,"ci95":[75.95,79.8],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9028,"del":1488,"ins":1283},"empty_hyp":0,"utts_over_50pct":610,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"en","quant":"Q8_0","metric":"wer","err_pct":13.84,"ci95":[12.85,14.94],"n_utts":647,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":1377,"del":216,"ins":421},"empty_hyp":0,"utts_over_50pct":16,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"es","quant":"Q8_0","metric":"wer","err_pct":18.95,"ci95":[18.18,19.77],"n_utts":908,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3282,"del":439,"ins":697},"empty_hyp":0,"utts_over_50pct":20,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"et","quant":"Q8_0","metric":"wer","err_pct":101.33,"ci95":[99.81,102.93],"n_utts":893,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11464,"del":946,"ins":2668},"empty_hyp":2,"utts_over_50pct":891,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fa","quant":"Q8_0","metric":"wer","err_pct":100.18,"ci95":[97.96,102.5],"n_utts":871,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15164,"del":2249,"ins":2723},"empty_hyp":1,"utts_over_50pct":871,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fi","quant":"Q8_0","metric":"wer","err_pct":68.58,"ci95":[67.17,70.11],"n_utts":918,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7738,"del":675,"ins":1732},"empty_hyp":1,"utts_over_50pct":720,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fil","quant":"Q8_0","metric":"wer","err_pct":70.63,"ci95":[69.17,72.18],"n_utts":964,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13748,"del":1873,"ins":3045},"empty_hyp":0,"utts_over_50pct":805,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"fr","quant":"Q8_0","metric":"wer","err_pct":44.19,"ci95":[42.58,45.78],"n_utts":676,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5719,"del":578,"ins":1655},"empty_hyp":0,"utts_over_50pct":227,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gl","quant":"Q8_0","metric":"wer","err_pct":59.44,"ci95":[58.41,60.62],"n_utts":927,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10333,"del":1492,"ins":1359},"empty_hyp":0,"utts_over_50pct":630,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"gu","quant":"Q8_0","metric":"wer","err_pct":100.26,"ci95":[100.02,100.57],"n_utts":1000,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17845,"del":27120,"ins":140},"empty_hyp":3,"utts_over_50pct":1000,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ha","quant":"Q8_0","metric":"wer","err_pct":105.17,"ci95":[102.75,108.2],"n_utts":621,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8174,"del":7090,"ins":1494},"empty_hyp":17,"utts_over_50pct":621,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"he","quant":"Q8_0","metric":"wer","err_pct":77.35,"ci95":[75.88,78.82],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8820,"del":927,"ins":960},"empty_hyp":1,"utts_over_50pct":698,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hi","quant":"Q8_0","metric":"wer","err_pct":101.52,"ci95":[100.33,103.06],"n_utts":418,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7832,"del":10913,"ins":296},"empty_hyp":3,"utts_over_50pct":418,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hr","quant":"Q8_0","metric":"wer","err_pct":84.36,"ci95":[83.29,85.63],"n_utts":914,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11902,"del":1309,"ins":1763},"empty_hyp":0,"utts_over_50pct":891,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hu","quant":"Q8_0","metric":"wer","err_pct":90.54,"ci95":[88.94,92.19],"n_utts":905,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10744,"del":2052,"ins":2665},"empty_hyp":10,"utts_over_50pct":883,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"hy","quant":"Q8_0","metric":"wer","err_pct":110.5,"ci95":[108.71,112.3],"n_utts":932,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11422,"del":6734,"ins":1913},"empty_hyp":8,"utts_over_50pct":932,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"id","quant":"Q8_0","metric":"wer","err_pct":60.49,"ci95":[58.53,62.76],"n_utts":687,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6055,"del":1143,"ins":1005},"empty_hyp":1,"utts_over_50pct":406,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"is","quant":"Q8_0","metric":"wer","err_pct":115.88,"ci95":[108.2,125.7],"n_utts":46,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":773,"del":211,"ins":213},"empty_hyp":0,"utts_over_50pct":46,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"it","quant":"Q8_0","metric":"wer","err_pct":31.24,"ci95":[30.23,32.38],"n_utts":865,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4968,"del":699,"ins":971},"empty_hyp":0,"utts_over_50pct":114,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ja","quant":"Q8_0","metric":"cer","err_pct":39.38,"ci95":[37.82,40.98],"n_utts":650,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9377,"del":1337,"ins":2142},"empty_hyp":0,"utts_over_50pct":127,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ka","quant":"Q8_0","metric":"wer","err_pct":110.34,"ci95":[108.68,112.16],"n_utts":979,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7520,"del":9618,"ins":1774},"empty_hyp":52,"utts_over_50pct":979,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kk","quant":"Q8_0","metric":"wer","err_pct":136.56,"ci95":[131.42,141.63],"n_utts":856,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11205,"del":3587,"ins":5713},"empty_hyp":20,"utts_over_50pct":856,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"kn","quant":"Q8_0","metric":"wer","err_pct":100.32,"ci95":[100.1,100.62],"n_utts":838,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14520,"del":27608,"ins":143},"empty_hyp":18,"utts_over_50pct":838,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ko","quant":"Q8_0","metric":"cer","err_pct":19.07,"ci95":[17.69,20.43],"n_utts":382,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2539,"del":686,"ins":147},"empty_hyp":0,"utts_over_50pct":5,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lb","quant":"Q8_0","metric":"wer","err_pct":99.69,"ci95":[98.93,100.53],"n_utts":934,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11165,"del":9208,"ins":522},"empty_hyp":10,"utts_over_50pct":934,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ln","quant":"Q8_0","metric":"wer","err_pct":103.55,"ci95":[101.39,106.01],"n_utts":478,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5722,"del":4118,"ins":980},"empty_hyp":13,"utts_over_50pct":478,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lt","quant":"Q8_0","metric":"wer","err_pct":105.0,"ci95":[103.27,106.96],"n_utts":986,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13218,"del":2585,"ins":2187},"empty_hyp":11,"utts_over_50pct":986,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lv","quant":"Q8_0","metric":"wer","err_pct":95.98,"ci95":[94.86,97.2],"n_utts":851,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12048,"del":1009,"ins":1739},"empty_hyp":0,"utts_over_50pct":849,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mi","quant":"Q8_0","metric":"wer","err_pct":96.27,"ci95":[95.25,97.41],"n_utts":1008,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13868,"del":14682,"ins":1315},"empty_hyp":18,"utts_over_50pct":1006,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mk","quant":"Q8_0","metric":"wer","err_pct":78.61,"ci95":[77.42,79.85],"n_utts":973,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12387,"del":2356,"ins":1831},"empty_hyp":6,"utts_over_50pct":933,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ml","quant":"Q8_0","metric":"wer","err_pct":100.03,"ci95":[99.99,100.1],"n_utts":958,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12520,"del":41715,"ins":19},"empty_hyp":51,"utts_over_50pct":958,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mn","quant":"Q8_0","metric":"wer","err_pct":110.21,"ci95":[107.98,112.39],"n_utts":949,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9447,"del":9084,"ins":1893},"empty_hyp":12,"utts_over_50pct":949,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mr","quant":"Q8_0","metric":"wer","err_pct":100.82,"ci95":[100.19,101.75],"n_utts":1015,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":17619,"del":33662,"ins":455},"empty_hyp":38,"utts_over_50pct":1015,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ms","quant":"Q8_0","metric":"wer","err_pct":62.17,"ci95":[60.25,64.27],"n_utts":749,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6685,"del":1333,"ins":1153},"empty_hyp":8,"utts_over_50pct":465,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"mt","quant":"Q8_0","metric":"wer","err_pct":99.85,"ci95":[99.1,100.67],"n_utts":926,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10746,"del":10997,"ins":456},"empty_hyp":39,"utts_over_50pct":926,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nb","quant":"Q8_0","metric":"wer","err_pct":67.36,"ci95":[65.41,69.32],"n_utts":357,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3845,"del":511,"ins":776},"empty_hyp":1,"utts_over_50pct":299,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ne","quant":"Q8_0","metric":"wer","err_pct":101.3,"ci95":[100.35,102.76],"n_utts":726,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11005,"del":22325,"ins":447},"empty_hyp":17,"utts_over_50pct":726,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"nl","quant":"Q8_0","metric":"wer","err_pct":54.86,"ci95":[53.09,56.75],"n_utts":364,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3348,"del":336,"ins":923},"empty_hyp":0,"utts_over_50pct":206,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"oc","quant":"Q8_0","metric":"wer","err_pct":96.44,"ci95":[94.89,98.25],"n_utts":998,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":14923,"del":8583,"ins":1953},"empty_hyp":2,"utts_over_50pct":995,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pa","quant":"Q8_0","metric":"wer","err_pct":100.54,"ci95":[100.06,101.23],"n_utts":574,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7117,"del":18386,"ins":141},"empty_hyp":53,"utts_over_50pct":574,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pl","quant":"Q8_0","metric":"wer","err_pct":54.22,"ci95":[52.62,55.68],"n_utts":758,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5939,"del":575,"ins":1287},"empty_hyp":1,"utts_over_50pct":420,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ps","quant":"Q8_0","metric":"wer","err_pct":101.04,"ci95":[100.5,101.76],"n_utts":512,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5378,"del":7820,"ins":155},"empty_hyp":12,"utts_over_50pct":512,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"pt","quant":"Q8_0","metric":"wer","err_pct":24.07,"ci95":[23.02,25.16],"n_utts":919,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3734,"del":478,"ins":956},"empty_hyp":0,"utts_over_50pct":63,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ro","quant":"Q8_0","metric":"wer","err_pct":82.43,"ci95":[81.13,83.73],"n_utts":883,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12506,"del":2196,"ins":2443},"empty_hyp":1,"utts_over_50pct":835,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ru","quant":"Q8_0","metric":"wer","err_pct":35.64,"ci95":[34.38,36.82],"n_utts":775,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4164,"del":528,"ins":691},"empty_hyp":0,"utts_over_50pct":145,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sd","quant":"Q8_0","metric":"wer","err_pct":105.42,"ci95":[103.11,108.5],"n_utts":980,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7382,"del":16251,"ins":1286},"empty_hyp":48,"utts_over_50pct":980,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sk","quant":"Q8_0","metric":"wer","err_pct":82.2,"ci95":[81.01,83.37],"n_utts":792,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9724,"del":1396,"ins":1412},"empty_hyp":24,"utts_over_50pct":768,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sl","quant":"Q8_0","metric":"wer","err_pct":91.29,"ci95":[90.06,92.5],"n_utts":834,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":11428,"del":2007,"ins":1641},"empty_hyp":1,"utts_over_50pct":831,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sn","quant":"Q8_0","metric":"wer","err_pct":118.33,"ci95":[115.19,121.59],"n_utts":925,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":7804,"del":8054,"ins":3075},"empty_hyp":29,"utts_over_50pct":925,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"so","quant":"Q8_0","metric":"wer","err_pct":105.31,"ci95":[103.9,106.84],"n_utts":1019,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9314,"del":14793,"ins":1363},"empty_hyp":54,"utts_over_50pct":1019,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sr","quant":"Q8_0","metric":"wer","err_pct":89.2,"ci95":[87.49,90.79],"n_utts":700,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":9565,"del":1404,"ins":1387},"empty_hyp":2,"utts_over_50pct":688,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sv","quant":"Q8_0","metric":"wer","err_pct":59.36,"ci95":[57.64,61.07],"n_utts":759,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6557,"del":743,"ins":1833},"empty_hyp":0,"utts_over_50pct":482,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"sw","quant":"Q8_0","metric":"wer","err_pct":99.96,"ci95":[98.66,101.47],"n_utts":487,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5865,"del":3501,"ins":627},"empty_hyp":26,"utts_over_50pct":487,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ta","quant":"Q8_0","metric":"wer","err_pct":87.46,"ci95":[84.77,90.18],"n_utts":591,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":15264,"del":4859,"ins":9319},"empty_hyp":7,"utts_over_50pct":541,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"te","quant":"Q8_0","metric":"wer","err_pct":101.18,"ci95":[100.3,102.48],"n_utts":472,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":5218,"del":17381,"ins":275},"empty_hyp":40,"utts_over_50pct":472,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tg","quant":"Q8_0","metric":"wer","err_pct":102.39,"ci95":[100.82,104.35],"n_utts":600,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":10290,"del":1778,"ins":1278},"empty_hyp":4,"utts_over_50pct":600,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"th","quant":"Q8_0","metric":"cer","err_pct":54.82,"ci95":[53.08,56.73],"n_utts":1021,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":28128,"del":15641,"ins":9688},"empty_hyp":12,"utts_over_50pct":377,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"tr","quant":"Q8_0","metric":"wer","err_pct":47.98,"ci95":[46.56,49.5],"n_utts":743,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4900,"del":561,"ins":919},"empty_hyp":1,"utts_over_50pct":312,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uk","quant":"Q8_0","metric":"wer","err_pct":57.49,"ci95":[56.26,58.8],"n_utts":750,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6515,"del":705,"ins":1042},"empty_hyp":0,"utts_over_50pct":479,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"ur","quant":"Q8_0","metric":"wer","err_pct":71.42,"ci95":[68.68,74.47],"n_utts":299,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":4547,"del":931,"ins":591},"empty_hyp":0,"utts_over_50pct":264,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"uz","quant":"Q8_0","metric":"wer","err_pct":103.19,"ci95":[102.2,104.3],"n_utts":862,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":6596,"del":9262,"ins":619},"empty_hyp":42,"utts_over_50pct":862,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"vi","quant":"Q8_0","metric":"wer","err_pct":64.49,"ci95":[62.89,65.97],"n_utts":857,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13099,"del":2224,"ins":1456},"empty_hyp":3,"utts_over_50pct":600,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"yo","quant":"Q8_0","metric":"wer","err_pct":102.29,"ci95":[101.01,103.99],"n_utts":831,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":8727,"del":13191,"ins":748},"empty_hyp":9,"utts_over_50pct":831,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"zh","quant":"Q8_0","metric":"cer","err_pct":41.91,"ci95":[40.85,42.93],"n_utts":945,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":12681,"del":1869,"ins":393},"empty_hyp":0,"utts_over_50pct":265,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F32","metric":"wer","err_pct":7.49,"ci95":[7.16,7.87],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2956,"del":382,"ins":633},"empty_hyp":0,"utts_over_50pct":37,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"F16","metric":"wer","err_pct":7.48,"ci95":[7.15,7.85],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2955,"del":382,"ins":632},"empty_hyp":0,"utts_over_50pct":35,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q8_0","metric":"wer","err_pct":7.52,"ci95":[7.18,7.88],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2964,"del":383,"ins":642},"empty_hyp":0,"utts_over_50pct":38,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q6_K","metric":"wer","err_pct":7.54,"ci95":[7.21,7.9],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2966,"del":384,"ins":649},"empty_hyp":0,"utts_over_50pct":38,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q5_K_M","metric":"wer","err_pct":7.82,"ci95":[7.37,8.37],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":2996,"del":401,"ins":749},"empty_hyp":0,"utts_over_50pct":47,"measurement_provenance":"legacy-published"}, + {"dataset":"librispeech","split":"test-clean","language":"en","quant":"Q4_K_M","metric":"wer","err_pct":7.78,"ci95":[7.44,8.15],"n_utts":2620,"batch_size":1,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":3033,"del":353,"ins":738},"empty_hyp":0,"utts_over_50pct":44,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"km","quant":"Q8_0","metric":"cer","err_pct":111.78,"ci95":[108.85,114.96],"n_utts":771,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":27592,"del":34973,"ins":8650},"empty_hyp":132,"utts_over_50pct":771,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"lo","quant":"Q8_0","metric":"cer","err_pct":105.48,"ci95":[102.73,108.71],"n_utts":405,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":13913,"del":22957,"ins":2606},"empty_hyp":21,"utts_over_50pct":405,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"my","quant":"Q8_0","metric":"cer","err_pct":108.53,"ci95":[106.54,110.87],"n_utts":880,"batch_size":8,"timestamps":"none","engine_sha":null,"measured_on":null,"errors":{"sub":16331,"del":41931,"ins":5222},"empty_hyp":251,"utts_over_50pct":880,"measurement_provenance":"legacy-published"}, + {"dataset":"fleurs","split":"test","language":"jv","language_hint":"jw","backend":"cuda","quant":"Q8_0","metric":"wer","err_pct":106.07,"ci95":[104.37,107.97],"n_utts":728,"batch_size":8,"timestamps":"none","engine_sha":"5d0c03e","publication_profile":"asr-publication-v2","measured_on":null,"errors":{"sub":7957,"del":6376,"ins":1267},"empty_hyp":15,"utts_over_50pct":728} + ], + "headline_benchmark": { + "dataset": "librispeech", + "split": "test-clean", + "language": "en", + "metric": "wer", + "batch_size": 1, + "timestamps": "none" + }, + "speed_benchmarks": [ + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":253.0,"xrt_compute":139.65,"load_ms":54.1,"mel_ms":4.8,"encode_ms":66.4,"decode_ms":181.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":253.2,"xrt_wall":139.54,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":99.7,"xrt_compute":110.34,"load_ms":53.2,"mel_ms":4.1,"encode_ms":65.5,"decode_ms":30.1,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":99.9,"xrt_wall":110.12,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":250.5,"xrt_compute":141.07,"load_ms":54.8,"mel_ms":4.8,"encode_ms":64.3,"decode_ms":181.4,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":250.7,"xrt_wall":140.94,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":99.9,"xrt_compute":110.09,"load_ms":50.6,"mel_ms":4.1,"encode_ms":65.2,"decode_ms":30.6,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":100.1,"xrt_wall":109.89,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":141.4,"xrt_compute":249.8,"load_ms":54.3,"mel_ms":4.9,"encode_ms":8.6,"decode_ms":128.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":141.6,"xrt_wall":249.58,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":39.5,"xrt_compute":278.34,"load_ms":50.9,"mel_ms":4.1,"encode_ms":8.6,"decode_ms":26.8,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":39.6,"xrt_wall":277.53,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":139.4,"xrt_compute":253.55,"load_ms":56.4,"mel_ms":4.9,"encode_ms":8.4,"decode_ms":126.0,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":139.5,"xrt_wall":253.28,"publication_profile":"asr-publication-v2"}, + {"machine":"m4-max","backend":"metal","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":39.0,"xrt_compute":282.04,"load_ms":51.0,"mel_ms":4.1,"encode_ms":8.4,"decode_ms":26.5,"engine_sha":"77b0c93","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":39.1,"xrt_wall":281.4,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":820.1,"xrt_compute":43.08,"load_ms":97.1,"mel_ms":32.7,"encode_ms":162.8,"decode_ms":624.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":820.3,"xrt_wall":43.07,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":310.8,"xrt_compute":35.39,"load_ms":113.1,"mel_ms":34.2,"encode_ms":161.5,"decode_ms":115.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":311.1,"xrt_wall":35.36,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":811.3,"xrt_compute":43.55,"load_ms":96.5,"mel_ms":41.3,"encode_ms":161.4,"decode_ms":608.7,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":811.5,"xrt_wall":43.54,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"cpu","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":305.9,"xrt_compute":35.96,"load_ms":114.3,"mel_ms":31.3,"encode_ms":161.5,"decode_ms":113.2,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":306.1,"xrt_wall":35.94,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"dots","sample_duration_s":35.332,"total_ms":825.1,"xrt_compute":42.82,"load_ms":85.1,"mel_ms":35.7,"encode_ms":88.4,"decode_ms":700.9,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":825.4,"xrt_wall":42.81,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q4_K_M","sample":"jfk","sample_duration_s":11.0,"total_ms":252.6,"xrt_compute":43.55,"load_ms":100.2,"mel_ms":37.1,"encode_ms":87.9,"decode_ms":127.6,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":253.0,"xrt_wall":43.48,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"dots","sample_duration_s":35.332,"total_ms":822.4,"xrt_compute":42.96,"load_ms":86.1,"mel_ms":36.6,"encode_ms":90.3,"decode_ms":695.4,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":822.7,"xrt_wall":42.95,"publication_profile":"asr-publication-v2"}, + {"machine":"ryzen-4750u","backend":"vulkan","quant":"Q8_0","sample":"jfk","sample_duration_s":11.0,"total_ms":254.5,"xrt_compute":43.22,"load_ms":103.4,"mel_ms":33.8,"encode_ms":90.7,"decode_ms":130.0,"engine_sha":"218aeae3","measured_on":"2026-09-14","thermal_gated":null,"wall_ms":254.7,"xrt_wall":43.18,"publication_profile":"asr-publication-v2"} + ] +} diff --git a/docs/_templates/model-card.md.j2 b/docs/_templates/model-card.md.j2 deleted file mode 100644 index 2b5dc72f..00000000 --- a/docs/_templates/model-card.md.j2 +++ /dev/null @@ -1,166 +0,0 @@ -{# Jinja2 template for a user-facing model card under docs/models/.md. - Rendered by porting-ship from the family's artifacts (intake, tolerances, - WER reports, bench reports). Authored prose sections stay in the variant's - rendered file after first authoring; this template is the initial - scaffold, not a live re-renderer. - - Required context: - display_name e.g. "Parakeet TDT 0.6B v2" - hf_repo upstream HF repo id (org/name) - one_liner one-sentence architecture summary - capabilities_prose one or two sentences on scope (English-only? - streaming? translation? timestamps?) - license_display human-facing license label - upstream_commit_short 7-char upstream SHA - upstream_commit_url HF commit link - pin_date ISO date when upstream_commit was pinned - - target_hf_repo our HF repo (e.g. handy-computer/-gguf) - presets list of {name, filename, size, wer_pct} dicts, - in order F32/BF16/F16/Q8_0/Q6_K/Q5_K_M/Q4_K_M - where each exists - dataset_display e.g. "LibriSpeech test-clean" - dataset_n number of utterances scored - upstream_wer_pct publisher's reported WER, percent (or null) - - validation_reference e.g. "NeMo" - validation_commit short sha of this repo when last validated - validation_tensor_rows list of {name, max_abs, mean_abs, note} - drift_source_prose one or two sentences paraphrasing the - tolerances _comment drift story - - reproduce_convert_cmd the exact uv run convert-* invocation - reproduce_validate_cmd uv run scripts/validate.py all --family --variant - - perf_machines optional list of {slug, os, transcribe_sha, - rows: [{backend, sample, quants: {q: latency_str}}]} -#} -# {{ display_name }} - -{{ hf_repo | replace("'", "\"") }}'s [`{{ hf_repo }}`](https://huggingface.co/{{ hf_repo }}) -ported to transcribe.cpp. {{ one_liner }} - -## What it's for - -{{ capabilities_prose }} - -See the upstream [model card](https://huggingface.co/{{ hf_repo }}) for training -data, intended use, and upstream evaluation methodology. - -Licensed {{ license_display }}. Ported from upstream commit -[`{{ upstream_commit_short }}`]({{ upstream_commit_url }}), pinned {{ pin_date }}. - -## Input limits - -{% if input_limit_prose is defined and input_limit_prose -%} -{{ input_limit_prose }} -{%- else -%} -{%- if max_audio_seconds is defined and max_audio_seconds -%} -This model accepts up to about **{{ max_audio_seconds }} s** of 16 kHz mono audio -per call (`transcribe_capabilities.max_audio_ms`). Longer input is rejected with -`TRANSCRIBE_ERR_INPUT_TOO_LONG` before transcription, or — for soft-window -families — accepted with a `WARN` and possibly reduced accuracy. Split longer -audio into segments. -{%- else -%} -This model has no practical per-call length limit -(`transcribe_capabilities.max_audio_ms == 0`): long audio is windowed -internally. Query `max_audio_ms` at runtime to confirm. -{%- endif -%} -{%- endif %} - -A transcript that reaches the model's context or generation budget before the -end of speech is returned with the hard status `TRANSCRIBE_ERR_OUTPUT_TRUNCATED` -(the partial transcript stays readable, and `transcribe_was_truncated()` is -also set, with a `WARN`); it is never silently cut or reported as `OK`. See -[docs/input-limits.md](../input-limits.md) for the -full contract and the per-family bucket table. - -## Download - -| Quantization | Download | Size | WER ({{ dataset_display }}) | -| --- | --- | ---: | ---: | -{% for p in presets -%} -| {{ p.name }} | [{{ p.filename }}](https://huggingface.co/{{ target_hf_repo }}/resolve/main/{{ p.filename }}) | {{ p.size }} | {{ "%.2f"|format(p.wer_pct) }}% | -{% endfor %} - -WER measured on the full {{ dataset_display }} split ({{ dataset_n }} utterances). -{%- if upstream_wer_pct is not none %} -Publisher's self-reported number on the same split is {{ "%.2f"|format(upstream_wer_pct) }}%. -{%- endif %} - -## Quick Start - -```bash -cmake -B build -cmake --build build --target transcribe-cli - -build/bin/transcribe-cli \ - models/{{ presets[0].filename.rsplit('-', 1)[0] }}/{{ presets[0].filename }} \ - samples/jfk.wav -``` - -If your audio is not already 16 kHz mono WAV, convert it first: - -```bash -ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav -``` - -## Numerical Validation - -transcribe.cpp is validated tensor-by-tensor against {{ validation_reference }} -on `samples/jfk.wav`. All contract tensors fall within family tolerance. Last -validated at commit [`{{ validation_commit }}`](https://github.com/handy-computer/transcribe.cpp/tree/{{ validation_commit }}). - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -{% for t in validation_tensor_rows -%} -| `{{ t.name }}` | `{{ t.max_abs }}` | `{{ t.mean_abs }}` | {{ t.note }} | -{% endfor %} - -{{ drift_source_prose }} - -## Performance - -{% if perf_machines %} -Wall-clock latency, mean over iterations after warmup. Speedup over realtime in parentheses. - -{% for m in perf_machines -%} -### {{ m.slug }} - -| Backend | Sample | {% for q in m.quants %}{{ q }}{% if not loop.last %} | {% endif %}{% endfor %} | -| --- | --- | {% for q in m.quants %}---:{% if not loop.last %} | {% endif %}{% endfor %} | -{% for row in m.rows -%} -| {{ row.backend }} | {{ row.sample }} | {% for q in m.quants %}{{ row.latency[q] }}{% if not loop.last %} | {% endif %}{% endfor %} | -{% endfor %} - -{{ m.os }}, transcribe.cpp `{{ m.transcribe_sha }}`. - -{% endfor -%} - -Benchmark reproduction: - -```bash -uv run scripts/bench/run.py \ - --models {{ variant }} \ - --quants q8_0,q4_k_m \ - --samples jfk \ - --iters 3 --warmup 1 \ - --name {{ variant }}-publication -``` -{% else %} -Performance numbers are pending. Run `uv run scripts/bench/run.py --models {{ variant }}` to generate them. -{% endif %} - -## Reproduction - -### Convert - -```bash -{{ reproduce_convert_cmd }} -``` - -### Validate - -```bash -{{ reproduce_validate_cmd }} -``` diff --git a/docs/models/canary-180m-flash.md b/docs/models/canary-180m-flash.md index c0799ca5..1bc1888d 100644 --- a/docs/models/canary-180m-flash.md +++ b/docs/models/canary-180m-flash.md @@ -1,8 +1,15 @@ # Canary 180M Flash -NVIDIA's [`nvidia/canary-180m-flash`](https://huggingface.co/nvidia/canary-180m-flash) -ported to transcribe.cpp. A 182M-parameter multitask AED with a 17-layer -FastConformer encoder and a 4-layer Transformer decoder. + +Upstream: [`nvidia/canary-180m-flash`](https://huggingface.co/nvidia/canary-180m-flash) at [`b12ab41`](https://huggingface.co/nvidia/canary-180m-flash/commit/b12ab41). + +Offline multilingual speech-to-text and translation. A multitask AED +with a 17-layer FastConformer encoder and a 4-layer Transformer decoder. +Supports automatic speech recognition in English, German, Spanish, and +French, and bidirectional EN↔{DE, ES, FR} translation. Takes a 16 kHz +mono WAV and produces a transcript. Not a streaming model; word/segment +timestamps are upstream-experimental and not exposed in the v1 port. + ## What it's for @@ -21,27 +28,44 @@ and not exposed in the v1 port (deferred — would require porting the See NVIDIA's [model card](https://huggingface.co/nvidia/canary-180m-flash) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`b12ab41`](https://huggingface.co/nvidia/canary-180m-flash/commit/b12ab418510d093e83890178fd0e8b0d0f7918a6), -pinned 2026-05-08. + +Licensed CC-BY-4.0. Ported from upstream commit [`b12ab41`](https://huggingface.co/nvidia/canary-180m-flash/commit/b12ab41), pinned 2026-05-08. Validated against the NeMo reference at transcribe.cpp commit [`db53eda`](https://github.com/handy-computer/transcribe.cpp/tree/db53eda) on 2026-05-08. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [canary-180m-flash-F32.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-F32.gguf) | 721 MB | 1.94% | -| F16 | [canary-180m-flash-F16.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-F16.gguf) | 364 MB | 1.94% | -| Q8_0 | [canary-180m-flash-Q8_0.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q8_0.gguf) | 208 MB | 1.93% | -| Q6_K | [canary-180m-flash-Q6_K.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q6_K.gguf) | 168 MB | 1.93% | -| Q5_K_M | [canary-180m-flash-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q5_K_M.gguf) | 151 MB | 1.90% | -| Q4_K_M | [canary-180m-flash-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q4_K_M.gguf) | 133 MB | 1.93% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding and no external LM. F32 reference baseline: 1.94%. -On the same wavs, NeMo's reference run produces 1.93% — one substitution -difference out of ~27k reference words — so the F32 port matches the -reference framework at the noise floor. NVIDIA's self-reported number on -the upstream model card is 1.87% +| F32 | [canary-180m-flash-F32.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-F32.gguf) | 756 MB | 1.94% | +| F16 | [canary-180m-flash-F16.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-F16.gguf) | 382 MB | 1.94% | +| Q8_0 | [canary-180m-flash-Q8_0.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q8_0.gguf) | 218 MB | 1.93% | +| Q6_K | [canary-180m-flash-Q6_K.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q6_K.gguf) | 176 MB | 1.93% | +| Q5_K_M | [canary-180m-flash-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q5_K_M.gguf) | 159 MB | 1.90% | +| Q4_K_M | [canary-180m-flash-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-180m-flash-gguf/resolve/main/canary-180m-flash-Q4_K_M.gguf) | 139 MB | 1.93% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 1.94%. On the same wavs, +NeMo's reference run produces 1.93% (one substitution difference out of ~27k +reference words), so the F32 port matches the reference framework at the noise +floor. NVIDIA's self-reported number on the upstream model card is 1.87%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 7.33% | +| en | WER | 5.98% | +| es | WER | 6.54% | +| fr | WER | 8.53% | + ## Quick Start @@ -78,53 +102,49 @@ CLI flags specific to canary: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ----------------: | ----------------: | -| Metal | jfk (11.0s) | 72.3 ms (152.1×) | 68.0 ms (161.8×) | -| Metal | dots (35.3s) | 278.3 ms (127.0×) | 255.0 ms (138.6×) | -| CPU | jfk (11.0s) | 137.1 ms (80.2×) | 124.0 ms (88.7×) | -| CPU | dots (35.3s) | 521.6 ms (67.7×) | 482.1 ms (73.3×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `19b3b87`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -----------------: | -----------------: | +| Metal | jfk (11.0s) | 64.0 ms (171.97×) | 59.2 ms (185.69×) | +| Metal | dots (35.3s) | 261.8 ms (134.95×) | 240.4 ms (146.98×) | +| CPU | jfk (11.0s) | 127.8 ms (86.10×) | 129.5 ms (84.94×) | +| CPU | dots (35.3s) | 492.7 ms (71.72×) | 490.1 ms (72.09×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Vulkan | jfk (11.0s) | 315.6 ms (34.9×) | 296.0 ms (37.2×) | -| Vulkan | dots (35.3s) | 1.22 s (28.9×) | 1.10 s (32.1×) | -| CPU | jfk (11.0s) | 454.3 ms (24.2×) | 370.5 ms (29.7×) | -| CPU | dots (35.3s) | 1.92 s (18.4×) | 1.63 s (21.7×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Vulkan | jfk (11.0s) | 307.1 ms (35.82×) | 283.6 ms (38.78×) | +| Vulkan | dots (35.3s) | 1.16 s (30.35×) | 1.07 s (33.14×) | +| CPU | jfk (11.0s) | 437.8 ms (25.13×) | 420.9 ms (26.13×) | +| CPU | dots (35.3s) | 1.93 s (18.33×) | 1.79 s (19.69×) | -Fedora Linux 43, transcribe.cpp `4d44530`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models canary-180m-flash \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name canary-180m-flash-publication +uv run scripts/bench/run.py --profile --models canary-180m-flash ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against NeMo on `samples/jfk.wav`. All 17 checkpointed tensors fall within family -tolerance, and the F32 transcript matches the NeMo reference at the -noise floor (one substitution out of ~27k reference words across full -test-clean). Last validated at commit -[`db53eda`](https://github.com/handy-computer/transcribe.cpp/tree/db53eda). +tolerance, and the F32 transcript matches the NeMo reference at the noise +floor (one substitution out of ~27k reference words across full +test-clean). | Field | Value | | --- | --- | diff --git a/docs/models/canary-1b-flash.md b/docs/models/canary-1b-flash.md index 0148f79c..d45ad98a 100644 --- a/docs/models/canary-1b-flash.md +++ b/docs/models/canary-1b-flash.md @@ -1,8 +1,15 @@ # Canary 1B Flash -NVIDIA's [`nvidia/canary-1b-flash`](https://huggingface.co/nvidia/canary-1b-flash) -ported to transcribe.cpp. An 883M-parameter multitask AED with a 32-layer -FastConformer encoder and a 4-layer Transformer decoder. + +Upstream: [`nvidia/canary-1b-flash`](https://huggingface.co/nvidia/canary-1b-flash) at [`a9a55e0`](https://huggingface.co/nvidia/canary-1b-flash/commit/a9a55e0). + +Offline multilingual speech-to-text and translation. A multitask AED +with a 32-layer FastConformer encoder and a 4-layer Transformer decoder. +Supports automatic speech recognition in English, German, Spanish, and +French, and bidirectional EN↔{DE, ES, FR} translation. Takes a 16 kHz +mono WAV and produces a transcript. Not a streaming model; word/segment +timestamps are upstream-experimental and not exposed in the v1 port. + ## What it's for @@ -17,24 +24,42 @@ Offline multilingual speech-to-text and translation. The model takes a See NVIDIA's [model card](https://huggingface.co/nvidia/canary-1b-flash) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`a9a55e0`](https://huggingface.co/nvidia/canary-1b-flash/commit/a9a55e0295e7dd50d0c8c2a19491900a0daf24f3), -pinned 2026-05-08. + +Licensed CC-BY-4.0. Ported from upstream commit [`a9a55e0`](https://huggingface.co/nvidia/canary-1b-flash/commit/a9a55e0), pinned 2026-05-08. Validated against the NeMo reference at transcribe.cpp commit [`db53eda`](https://github.com/handy-computer/transcribe.cpp/tree/db53eda) on 2026-05-08. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [canary-1b-flash-F32.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-F32.gguf) | 3.3 GB | 1.62% | -| F16 | [canary-1b-flash-F16.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-F16.gguf) | 1.7 GB | 1.62% | -| Q8_0 | [canary-1b-flash-Q8_0.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q8_0.gguf) | 1.0 GB | 1.62% | -| Q6_K | [canary-1b-flash-Q6_K.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q6_K.gguf) | 818 MB | 1.65% | -| Q5_K_M | [canary-1b-flash-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q5_K_M.gguf) | 734 MB | 1.64% | -| Q4_K_M | [canary-1b-flash-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q4_K_M.gguf) | 646 MB | 1.59% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding and no external LM. F32 reference baseline: 1.62%. -NVIDIA's self-reported number on the upstream model card is 1.48% +| F32 | [canary-1b-flash-F32.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-F32.gguf) | 3.56 GB | 1.62% | +| F16 | [canary-1b-flash-F16.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-F16.gguf) | 1.79 GB | 1.62% | +| Q8_0 | [canary-1b-flash-Q8_0.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q8_0.gguf) | 1.05 GB | 1.62% | +| Q6_K | [canary-1b-flash-Q6_K.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q6_K.gguf) | 858 MB | 1.65% | +| Q5_K_M | [canary-1b-flash-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q5_K_M.gguf) | 770 MB | 1.64% | +| Q4_K_M | [canary-1b-flash-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-flash-gguf/resolve/main/canary-1b-flash-Q4_K_M.gguf) | 677 MB | 1.59% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 1.62%. NVIDIA's +self-reported number on the upstream model card is 1.48%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 6.13% | +| en | WER | 4.75% | +| es | WER | 6.73% | +| fr | WER | 7.22% | + ## Quick Start @@ -71,43 +96,40 @@ CLI flags specific to canary: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 106.6 ms (103.2×) | 111.3 ms (98.8×) | -| Metal | dots (35.3s) | 368.6 ms (95.9×) | 349.2 ms (101.2×) | -| CPU | jfk (11.0s) | 518.3 ms (21.2×) | 430.1 ms (25.6×) | -| CPU | dots (35.3s) | 1.79 s (19.7×) | 1.51 s (23.4×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `19b3b87`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -----------------: | -----------------: | +| Metal | jfk (11.0s) | 93.5 ms (117.69×) | 91.8 ms (119.77×) | +| Metal | dots (35.3s) | 333.9 ms (105.82×) | 315.5 ms (112.00×) | +| CPU | jfk (11.0s) | 416.7 ms (26.40×) | 429.5 ms (25.61×) | +| CPU | dots (35.3s) | 1.44 s (24.47×) | 1.49 s (23.78×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Vulkan | jfk (11.0s) | 763.5 ms (14.4×) | 704.2 ms (15.6×) | -| Vulkan | dots (35.3s) | 2.46 s (14.4×) | 2.31 s (15.3×) | -| CPU | jfk (11.0s) | 1.49 s (7.4×) | 1.13 s (9.7×) | -| CPU | dots (35.3s) | 5.48 s (6.4×) | 4.51 s (7.8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Vulkan | jfk (11.0s) | 691.8 ms (15.90×) | 693.5 ms (15.86×) | +| Vulkan | dots (35.3s) | 2.35 s (15.06×) | 2.24 s (15.78×) | +| CPU | jfk (11.0s) | 1.09 s (10.07×) | 1.10 s (9.96×) | +| CPU | dots (35.3s) | 4.63 s (7.63×) | 4.54 s (7.79×) | -Fedora Linux 43, transcribe.cpp `4d44530`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models canary-1b-flash \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name canary-1b-flash-publication +uv run scripts/bench/run.py --profile --models canary-1b-flash ``` ## Numerical Validation diff --git a/docs/models/canary-1b-v2.md b/docs/models/canary-1b-v2.md index 6358f791..6d1c9f70 100644 --- a/docs/models/canary-1b-v2.md +++ b/docs/models/canary-1b-v2.md @@ -1,9 +1,16 @@ # Canary 1B v2 -NVIDIA's [`nvidia/canary-1b-v2`](https://huggingface.co/nvidia/canary-1b-v2) -ported to transcribe.cpp. A 978M-parameter multitask AED with a 32-layer -FastConformer encoder and an 8-layer Transformer decoder, covering 25 -European languages. + +Upstream: [`nvidia/canary-1b-v2`](https://huggingface.co/nvidia/canary-1b-v2) at [`87bc526`](https://huggingface.co/nvidia/canary-1b-v2/commit/87bc526). + +Offline multilingual speech-to-text and translation across 25 European +languages. A multitask AED with a 32-layer FastConformer encoder and an +8-layer Transformer decoder. Supports automatic speech recognition for +any of the 25 supported languages, plus translation between supported +language pairs (per the upstream model card). Takes a 16 kHz mono WAV and +produces a transcript. Not a streaming model; word and segment timestamps +from the upstream model are not exposed in the v1 port. + ## What it's for @@ -29,26 +36,65 @@ model are not exposed in the v1 port. See NVIDIA's [model card](https://huggingface.co/nvidia/canary-1b-v2) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`87bc526`](https://huggingface.co/nvidia/canary-1b-v2/commit/87bc52657add533cd0156b3fc1aef027280754bf), -pinned 2026-05-08. + +Licensed CC-BY-4.0. Ported from upstream commit [`87bc526`](https://huggingface.co/nvidia/canary-1b-v2/commit/87bc526), pinned 2026-05-08. Validated against the NeMo reference at transcribe.cpp commit [`db53eda`](https://github.com/handy-computer/transcribe.cpp/tree/db53eda) on 2026-05-08. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [canary-1b-v2-F32.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-F32.gguf) | 3.7 GB | 1.92% | -| F16 | [canary-1b-v2-F16.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-F16.gguf) | 1.8 GB | 1.92% | -| Q8_0 | [canary-1b-v2-Q8_0.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q8_0.gguf) | 1.1 GB | 1.91% | -| Q6_K | [canary-1b-v2-Q6_K.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q6_K.gguf) | 889 MB | 1.94% | -| Q5_K_M | [canary-1b-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q5_K_M.gguf) | 798 MB | 1.93% | -| Q4_K_M | [canary-1b-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q4_K_M.gguf) | 701 MB | 1.91% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding and no external LM. F32 reference baseline: 1.92%. -NVIDIA's self-reported number on the upstream model card is 2.18%; our -F32 port comes in slightly under upstream (Δ −0.26pp) and is likely -down to scoring differences. +| F32 | [canary-1b-v2-F32.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-F32.gguf) | 3.92 GB | 1.92% | +| F16 | [canary-1b-v2-F16.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-F16.gguf) | 1.97 GB | 1.92% | +| Q8_0 | [canary-1b-v2-Q8_0.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q8_0.gguf) | 1.14 GB | 1.91% | +| Q6_K | [canary-1b-v2-Q6_K.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q6_K.gguf) | 932 MB | 1.94% | +| Q5_K_M | [canary-1b-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q5_K_M.gguf) | 837 MB | 1.93% | +| Q4_K_M | [canary-1b-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-v2-gguf/resolve/main/canary-1b-v2-Q4_K_M.gguf) | 735 MB | 1.91% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 1.92%. NVIDIA's +self-reported number on the upstream model card is 2.18%; our F32 port comes in +slightly under the upstream-reported number (Δ −0.26pp) and is likely down to +scoring differences. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| bg | WER | 9.22% | +| cs | WER | 8.56% | +| da | WER | 11.34% | +| de | WER | 4.46% | +| el | WER | 26.02% | +| en | WER | 4.47% | +| es | WER | 3.10% | +| et | WER | 12.72% | +| fi | WER | 8.86% | +| fr | WER | 5.09% | +| hr | WER | 8.40% | +| hu | WER | 13.06% | +| it | WER | 3.10% | +| lt | WER | 13.45% | +| lv | WER | 10.41% | +| mt | WER | 19.75% | +| nl | WER | 6.28% | +| pl | WER | 6.88% | +| pt | WER | 4.50% | +| ro | WER | 6.87% | +| ru | WER | 7.83% | +| sk | WER | 6.84% | +| sl | WER | 12.76% | +| sv | WER | 9.74% | +| uk | WER | 10.58% | + ## Quick Start @@ -94,43 +140,40 @@ CLI flags specific to canary: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 124.3 ms (88.5×) | 121.6 ms (90.5×) | -| Metal | dots (35.3s) | 430.7 ms (82.0×) | 406.1 ms (87.0×) | -| CPU | jfk (11.0s) | 555.0 ms (19.8×) | 453.5 ms (24.3×) | -| CPU | dots (35.3s) | 1.96 s (18.0×) | 1.66 s (21.3×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -----------------: | -----------------: | +| Metal | jfk (11.0s) | 105.1 ms (104.66×) | 104.1 ms (105.66×) | +| Metal | dots (35.3s) | 384.7 ms (91.84×) | 362.1 ms (97.57×) | +| CPU | jfk (11.0s) | 415.1 ms (26.50×) | 442.4 ms (24.86×) | +| CPU | dots (35.3s) | 1.49 s (23.66×) | 1.56 s (22.71×) | -macOS 26.4.1, transcribe.cpp `19b3b87`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Vulkan | jfk (11.0s) | 829.9 ms (13.3×) | 748.8 ms (14.7×) | -| Vulkan | dots (35.3s) | 2.70 s (13.1×) | 2.46 s (14.4×) | -| CPU | jfk (11.0s) | 1.55 s (7.1×) | 1.16 s (9.5×) | -| CPU | dots (35.3s) | 5.74 s (6.2×) | 4.70 s (7.5×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Vulkan | jfk (11.0s) | 758.9 ms (14.49×) | 732.3 ms (15.02×) | +| Vulkan | dots (35.3s) | 2.57 s (13.77×) | 2.41 s (14.64×) | +| CPU | jfk (11.0s) | 1.15 s (9.54×) | 1.15 s (9.54×) | +| CPU | dots (35.3s) | 4.97 s (7.11×) | 4.77 s (7.40×) | -Fedora Linux 43, transcribe.cpp `4d44530`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models canary-1b-v2 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name canary-1b-v2-publication +uv run scripts/bench/run.py --profile --models canary-1b-v2 ``` ## Numerical Validation diff --git a/docs/models/canary-1b.md b/docs/models/canary-1b.md index 6b893ff7..71d576e9 100644 --- a/docs/models/canary-1b.md +++ b/docs/models/canary-1b.md @@ -1,14 +1,17 @@ # Canary 1B -NVIDIA's [`nvidia/canary-1b`](https://huggingface.co/nvidia/canary-1b) -ported to transcribe.cpp. A 1B-parameter multitask AED with a 24-layer -FastConformer encoder and a 24-layer Transformer decoder — the original -canary release. - -> **License: CC-BY-NC-4.0 (non-commercial only).** This is the only -> canary variant under a non-commercial license. Every shipped GGUF -> carries `general.license: CC-BY-NC-4.0` in its KV metadata so -> downstream tooling can detect this without re-reading the model card. + +Upstream: [`nvidia/canary-1b`](https://huggingface.co/nvidia/canary-1b) at [`1698acf`](https://huggingface.co/nvidia/canary-1b/commit/1698acf). + +Offline multilingual speech-to-text and translation. A multitask AED +with a 24-layer FastConformer encoder and a 24-layer Transformer +decoder — the original canary release. Supports automatic speech +recognition in English, German, Spanish, and French, and translation +between supported pairs. Takes a 16 kHz mono WAV and produces a +transcript. Not a streaming model. **License: CC-BY-NC-4.0 +(non-commercial only)** — the only canary variant under a +non-commercial license. + ## What it's for @@ -21,26 +24,44 @@ Offline multilingual speech-to-text and translation. The model takes a See NVIDIA's [model card](https://huggingface.co/nvidia/canary-1b) for training data, intended use, and upstream evaluation methodology. -Ported from upstream commit -[`1698acf`](https://huggingface.co/nvidia/canary-1b/commit/1698acf1700ed316ffce1cb42d79437c7e360cfa), -pinned 2026-05-08. + +Licensed CC-BY-NC-4.0. Ported from upstream commit [`1698acf`](https://huggingface.co/nvidia/canary-1b/commit/1698acf), pinned 2026-05-08. Validated against the NeMo reference at transcribe.cpp commit [`db53eda`](https://github.com/handy-computer/transcribe.cpp/tree/db53eda) on 2026-05-08. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [canary-1b-F32.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-F32.gguf) | 3.8 GB | 1.55% | -| F16 | [canary-1b-F16.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-F16.gguf) | 1.9 GB | 1.55% | -| Q8_0 | [canary-1b-Q8_0.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q8_0.gguf) | 1.1 GB | 1.55% | -| Q6_K | [canary-1b-Q6_K.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q6_K.gguf) | 891 MB | 1.57% | -| Q5_K_M | [canary-1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q5_K_M.gguf) | 799 MB | 1.57% | -| Q4_K_M | [canary-1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q4_K_M.gguf) | 696 MB | 1.55% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding and no external LM. F32 reference baseline: 1.55%. -NVIDIA's self-reported number on the upstream model card is 1.48%. -Likely this is due to differences in how we score WER based on the results -we have from Canary 180m. +| F32 | [canary-1b-F32.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-F32.gguf) | 4.09 GB | 1.55% | +| F16 | [canary-1b-F16.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-F16.gguf) | 2.05 GB | 1.55% | +| Q8_0 | [canary-1b-Q8_0.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q8_0.gguf) | 1.16 GB | 1.55% | +| Q6_K | [canary-1b-Q6_K.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q6_K.gguf) | 934 MB | 1.57% | +| Q5_K_M | [canary-1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q5_K_M.gguf) | 838 MB | 1.57% | +| Q4_K_M | [canary-1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-1b-gguf/resolve/main/canary-1b-Q4_K_M.gguf) | 730 MB | 1.55% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 1.55%. NVIDIA's +self-reported number on the upstream model card is 1.48%; likely this is due to +differences in how we score WER, based on the results we have from +canary-180m-flash. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 6.45% | +| en | WER | 4.44% | +| es | WER | 6.06% | +| fr | WER | 7.44% | + ## Quick Start @@ -78,10 +99,6 @@ CLI flags specific to canary: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). - The 24-layer decoder makes this the slowest canary variant for decode-bound workloads — roughly 1.5× the wall time of canary-1b-flash on the same backend, and the GPU win over CPU is smaller here than on the *flash @@ -91,37 +108,38 @@ overhead. ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 216.0 ms (50.9×) | 198.1 ms (55.5×) | -| Metal | dots (35.3s) | 1.03 s (34.2×) | 937.2 ms (37.7×) | -| CPU | jfk (11.0s) | 511.1 ms (21.5×) | 426.2 ms (25.8×) | -| CPU | dots (35.3s) | 2.05 s (17.3×) | 1.72 s (20.6×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `19b3b87`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 207.3 ms (53.07×) | 187.5 ms (58.67×) | +| Metal | dots (35.3s) | 1.01 s (34.96×) | 930.9 ms (37.95×) | +| CPU | jfk (11.0s) | 425.7 ms (25.84×) | 430.2 ms (25.57×) | +| CPU | dots (35.3s) | 1.79 s (19.74×) | 1.73 s (20.41×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | -------------: | ---------------: | -| Vulkan | jfk (11.0s) | 1.07 s (10.2×) | 923.4 ms (11.9×) | -| Vulkan | dots (35.3s) | 4.63 s (7.6×) | 3.87 s (9.1×) | -| CPU | jfk (11.0s) | 1.71 s (6.4×) | 1.24 s (8.9×) | -| CPU | dots (35.3s) | 7.59 s (4.7×) | 5.97 s (5.9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Vulkan | jfk (11.0s) | 962.2 ms (11.43×) | 879.4 ms (12.51×) | +| Vulkan | dots (35.3s) | 4.36 s (8.10×) | 3.81 s (9.28×) | +| CPU | jfk (11.0s) | 1.40 s (7.88×) | 1.26 s (8.73×) | +| CPU | dots (35.3s) | 6.96 s (5.08×) | 6.26 s (5.65×) | -Fedora Linux 43, transcribe.cpp `4d44530`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models canary-1b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name canary-1b-publication +uv run scripts/bench/run.py --profile --models canary-1b ``` ## Numerical Validation diff --git a/docs/models/canary-qwen-2.5b.md b/docs/models/canary-qwen-2.5b.md index 3aef2ecb..ed09ae47 100644 --- a/docs/models/canary-qwen-2.5b.md +++ b/docs/models/canary-qwen-2.5b.md @@ -1,11 +1,15 @@ # Canary-Qwen 2.5B -NVIDIA's [`nvidia/canary-qwen-2.5b`](https://huggingface.co/nvidia/canary-qwen-2.5b) -ported to transcribe.cpp. A NeMo SALM (Speech-Augmented Language Model): -a 32-layer FastConformer audio encoder (`d_model=1024`, 16 heads) feeds + +Upstream: [`nvidia/canary-qwen-2.5b`](https://huggingface.co/nvidia/canary-qwen-2.5b) at [`b1469e1`](https://huggingface.co/nvidia/canary-qwen-2.5b/commit/b1469e1). + +Offline English speech-to-text. NeMo SALM (Speech-Augmented Language +Model): a FastConformer audio encoder (32 layers, `d_model=1024`) feeds audio embeddings into a Qwen3-1.7B causal LM (28 layers, -`hidden_size=2048`, `intermediate_size=6144`) via audio-token injection -at a sentinel position in the prompt. +`hidden_size=2048`) via audio-token injection at a sentinel position in +the prompt. English only. Takes a 16 kHz mono WAV and produces a +transcript via greedy decoding. + ## What it's for @@ -17,9 +21,9 @@ implicitly when the audio supports it). See NVIDIA's [model card](https://huggingface.co/nvidia/canary-qwen-2.5b) for training data, intended use, and upstream evaluation. -Licensed CC-BY-4.0. Ported from upstream commit -[`b1469e1`](https://huggingface.co/nvidia/canary-qwen-2.5b/commit/b1469e1bba1cfe140205529c79c434ca47180960), -pinned 2026-05-15. + +Licensed CC-BY-4.0. Ported from upstream commit [`b1469e1`](https://huggingface.co/nvidia/canary-qwen-2.5b/commit/b1469e1), pinned 2026-05-15. Validated against the NeMo SALM 2.7.3 reference at transcribe.cpp commit [`6f6c699`](https://github.com/handy-computer/transcribe.cpp/tree/6f6c699) on 2026-05-16. + ## Input limits @@ -32,25 +36,40 @@ exact per-session value. See the [input-length contract](../input-limits.md). ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [canary-qwen-2.5b-BF16.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-BF16.gguf) | 4.73 GB | 1.63% | -| F16 | [canary-qwen-2.5b-F16.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-F16.gguf) | 4.73 GB | 1.63% | -| Q8_0 | [canary-qwen-2.5b-Q8_0.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q8_0.gguf) | 2.61 GB | 1.63% | -| Q6_K | [canary-qwen-2.5b-Q6_K.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q6_K.gguf) | 2.06 GB | 1.63% | -| Q5_K_M | [canary-qwen-2.5b-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q5_K_M.gguf) | 1.85 GB | 1.63% | -| Q4_K_M | [canary-qwen-2.5b-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q4_K_M.gguf) | 1.62 GB | 1.63% | - -WER measured on the full LibriSpeech `test-clean` split (2620 utterances) -with the Whisper-style English text normalizer and jiwer 3.x. The -same-machine NeMo SALM reference run (CPU torch, dither=0.0, greedy -`model.generate`) lands at **1.61%** with 95% bootstrap CI [1.47%, -1.75%]; NVIDIA's published number is 1.60% (within the same CI). All six -GGUF presets land at exactly 1.63% (`+0.02pp` over our same-machine REF -run). The remaining `+0.02pp` C++ vs REF gap is BF16 weight-precision -cascade noise: of 2620 utterances, only 21 (0.8%) differ post-normalizer, -all classic small-margin token flips (homophones, word-boundary flips, -function-word substitutions). +| BF16 | [canary-qwen-2.5b-BF16.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-BF16.gguf) | 5.08 GB | 1.63% | +| F16 | [canary-qwen-2.5b-F16.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-F16.gguf) | 5.08 GB | 1.63% | +| Q8_0 | [canary-qwen-2.5b-Q8_0.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q8_0.gguf) | 2.80 GB | 1.63% | +| Q6_K | [canary-qwen-2.5b-Q6_K.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q6_K.gguf) | 2.21 GB | 1.63% | +| Q5_K_M | [canary-qwen-2.5b-Q5_K_M.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q5_K_M.gguf) | 1.98 GB | 1.63% | +| Q4_K_M | [canary-qwen-2.5b-Q4_K_M.gguf](https://huggingface.co/handy-computer/canary-qwen-2.5b-gguf/resolve/main/canary-qwen-2.5b-Q4_K_M.gguf) | 1.74 GB | 1.63% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Scored with the Whisper-style English text normalizer and jiwer 3.x on an Apple M4. +The same-machine NeMo SALM reference run (CPU torch, dither=0.0, greedy +`model.generate`) lands at **1.61%** with 95% bootstrap CI [1.47%, 1.75%]: `0.01` +above NVIDIA's published 1.60% but well within statistical noise. All six GGUF +presets land at exactly 1.63% (`+0.02` over our reference run, same CI band). +Investigation of the worst per-utterance differences shows scattered token-level +noise consistent with BF16 weight precision (homophones, word-boundary flips, +function-word substitutions). Reproduce with `scripts/wer/run.py` + +`scripts/wer/score.py`. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 3.58% | + ## Quick Start @@ -78,59 +97,51 @@ CLI flags: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 243.9 ms (45.1×) | 232.4 ms (47.3×) | -| Metal | dots (35.3s) | 1.02 s (34.5×) | 911.7 ms (38.8×) | -| CPU | jfk (11.0s) | 1.42 s (7.7×) | 1.06 s (10.4×) | -| CPU | dots (35.3s) | 5.14 s (6.9×) | 4.00 s (8.8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 228.6 ms (48.11×) | 204.7 ms (53.73×) | +| Metal | dots (35.3s) | 961.4 ms (36.75×) | 830.8 ms (42.53×) | +| CPU | jfk (11.0s) | 1.02 s (10.83×) | 1.04 s (10.55×) | +| CPU | dots (35.3s) | 3.83 s (9.22×) | 3.81 s (9.27×) | -macOS 26.4.1, transcribe.cpp `1b19945`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Vulkan | jfk (11.0s) | 2.41 s (4.6×) | 2.11 s (5.2×) | -| Vulkan | dots (35.3s) | 9.72 s (3.6×) | 8.48 s (4.2×) | -| CPU | jfk (11.0s) | 4.73 s (2.3×) | 3.43 s (3.2×) | -| CPU | dots (35.3s) | 18.42 s (1.9×) | 13.51 s (2.6×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -Fedora Linux 43, transcribe.cpp `51db32d`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 2.47 s (4.45×) | 2.15 s (5.12×) | +| Vulkan | dots (35.3s) | 10.09 s (3.50×) | 8.71 s (4.06×) | +| CPU | jfk (11.0s) | 3.87 s (2.84×) | 3.46 s (3.18×) | +| CPU | dots (35.3s) | 16.24 s (2.18×) | 13.87 s (2.55×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models canary-qwen-2.5b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name canary-qwen-2.5b-publication +uv run scripts/bench/run.py --profile --models canary-qwen-2.5b ``` ## Numerical Validation -transcribe.cpp is validated tensor-by-tensor against NeMo SALM -(`nemo.collections.speechlm2.SALM` 2.7.3) on `samples/jfk.wav` with the -strict CPU backend, BF16 weights promoted to F32 at load time. All 16 -checkpointed tensors fall within family tolerance, and the BF16 -transcript matches the reference verbatim -(`And so my fellow Americans ask not what your country can do for you ask what you can do for your country`). -Tolerances are pinned in `tests/tolerances/canary_qwen.json` with a -detailed `_comment` block naming the precision regime, the two -implementation gotchas (BF16 mel filterbank in NeMo's preprocessor, -forced F32 promotion of F16 depthwise conv kernels on CPU), and the -mechanism behind every widened entry. Last validated at commit -[`6f6c699`](https://github.com/handy-computer/transcribe.cpp/tree/6f6c699). +transcribe.cpp is validated tensor-by-tensor against NeMo SALM (`nemo.collections.speechlm2.SALM` 2.7.3) on +`samples/jfk.wav` with the strict CPU backend, BF16 weights promoted to F32 at load time. All 16 checkpointed +tensors fall within family tolerance, and the BF16 transcript matches the reference verbatim (`And so my +fellow Americans ask not what your country can do for you ask what you can do for your country`). Tolerances +are pinned in `tests/tolerances/canary_qwen.json` with a detailed `_comment` block naming the precision +regime, the two implementation gotchas (BF16 mel filterbank in NeMo's preprocessor, forced F32 promotion of +F16 depthwise conv kernels on CPU), and the mechanism behind every widened entry. | Field | Value | | --- | --- | @@ -140,27 +151,6 @@ mechanism behind every widened entry. Last validated at commit | Tolerances | `tests/tolerances/canary_qwen.json` | | Command | `uv run scripts/validate.py all --family canary_qwen --variant canary-qwen-2.5b` | -Selected tensors (observed on CPU, strict backend; see tolerance file -for budgets): - -| Tensor | Shape | Max abs diff | Mean abs diff | Notes | -| --- | --- | ---: | ---: | --- | -| `enc.mel.in` | `[128,1101]` | `6.724e-01` | `5.242e-05` | NeMo preprocessor's BF16 fb/window matrices vs C++ F32 STFT | -| `enc.pre_encode.out` | `[138,1024]` | `2.610e+02` | `5.795e-01` | Output of the conv subsampler; large extreme-bin spikes from the mel difference are absorbed here, then drained by the next LayerNorm | -| `enc.block.0.out` | `[138,1024]` | `1.653e+01` | `2.448e-02` | First FastConformer block | -| `enc.block.16.out` | `[138,1024]` | `2.528e+01` | `7.975e-02` | Mid-encoder | -| `enc.block.31.out` | `[138,1024]` | `7.921e-01` | `1.875e-02` | Final FastConformer block | -| `enc.final` | `[1024,138]` | `7.921e-01` | `1.875e-02` | Encoder output (transposed) | -| `perception.proj.out`| `[138,2048]` | `3.520e+00` | `4.314e-02` | Audio→LM width projection | -| `dec.token_emb` | `[15,2048]` | `0.000e+00` | `0.000e+00` | Pure embedding lookup | -| `dec.audio_injected` | `[152,2048]` | `3.520e+00` | `3.917e-02` | Audio-tokens scattered into the prompt sequence | -| `dec.block.0.out` | `[152,2048]` | `3.355e+00` | `4.365e-02` | First Qwen3 LM block | -| `dec.block.14.out` | `[152,2048]` | `1.506e+01` | `1.190e-01` | Mid-LM | -| `dec.block.27.out` | `[152,2048]` | `1.112e+02` | `9.333e-01` | Final LM block (accumulated) | -| `dec.out_before_head`| `[152,2048]` | `1.566e+01` | `4.152e-02` | Pre-head hidden state | -| `dec.logits_raw.gen0`| `[151936]` | `5.226e-01` | `6.571e-02` | Greedy step 0 logits | -| `dec.logits_raw.gen8`| `[151936]` | `1.514e+00` | `2.116e-01` | Greedy step 8 logits (mid-generation, exercises KV cache write/read) | - For the full porting writeup including the SALM trace, the audio-injection scatter contract, and the BF16-vs-F32 weight precision investigation, see diff --git a/docs/models/canary.md b/docs/models/canary.md index 693cf822..19718271 100644 --- a/docs/models/canary.md +++ b/docs/models/canary.md @@ -19,11 +19,10 @@ see the family doc at - **Best 4-language accuracy.** `canary-1b` (the original release; CC-BY-NC-4.0) uses a 24-layer decoder, trading decode speed for accuracy on en/de/es/fr. -- **Faster decode, same 4 languages.** `canary-1b-flash` (883M, 4-layer - decoder) — the speed-tuned sibling at ~1.6% WER on LibriSpeech - test-clean. -- **Smallest footprint.** `canary-180m-flash` (182M, 208 MB at Q8_0) — - the ultralight variant; same 4-language coverage as the flash 1B. +- **Faster decode, same 4 languages.** `canary-1b-flash` uses a 4-layer + decoder, the speed-tuned sibling of `canary-1b`. +- **Smallest footprint.** `canary-180m-flash` is the ultralight variant, + with the same 4-language coverage as the flash 1B. ## All variants @@ -31,12 +30,14 @@ WER is on LibriSpeech test-clean for the **Q8_0** preset, measured by transcribe.cpp's WER pipeline. See each per-variant doc for the full quant matrix and per-language WER/BLEU tables. -| Variant | Decoder depth | Params | Q8_0 size | WER (Q8_0) | Languages | Doc | -| --- | ---: | ---: | ---: | ---: | --- | --- | -| `canary-1b` | 24 | 1.0B | 1.1 GB | 1.55% | en, de, es, fr | [canary-1b.md](canary-1b.md) | -| `canary-1b-v2` | 8 | 978M | 1.1 GB | 1.91% | 25 European | [canary-1b-v2.md](canary-1b-v2.md) | -| `canary-1b-flash` | 4 | 883M | 1.0 GB | 1.62% | en, de, es, fr | [canary-1b-flash.md](canary-1b-flash.md) | -| `canary-180m-flash`| 4 | 182M | 208 MB | 1.93% | en, de, es, fr | [canary-180m-flash.md](canary-180m-flash.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `canary-1b` | 1B | en, de, es, fr | 1.16 GB | LibriSpeech test-clean (WER) | 1.55% | translate | [canary-1b.md](canary-1b.md) | +| `canary-1b-v2` | 980M | 25 languages | 1.14 GB | LibriSpeech test-clean (WER) | 1.91% | translate | [canary-1b-v2.md](canary-1b-v2.md) | +| `canary-1b-flash` | 890M | en, de, es, fr | 1.05 GB | LibriSpeech test-clean (WER) | 1.62% | translate | [canary-1b-flash.md](canary-1b-flash.md) | +| `canary-180m-flash` | 189M | en, de, es, fr | 218 MB | LibriSpeech test-clean (WER) | 1.93% | translate | [canary-180m-flash.md](canary-180m-flash.md) | + Pre-built GGUFs for every variant and quant are hosted under [`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); diff --git a/docs/models/cohere-transcribe-03-2026.md b/docs/models/cohere-transcribe-03-2026.md index f605605b..ed622c75 100644 --- a/docs/models/cohere-transcribe-03-2026.md +++ b/docs/models/cohere-transcribe-03-2026.md @@ -1,8 +1,14 @@ # Cohere Transcribe 03-2026 -Cohere's [`CohereLabs/cohere-transcribe-03-2026`](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026) -ported to transcribe.cpp. A Conformer encoder with a Transformer encoder-decoder -head (cross-attention, tied token embedding). + +Upstream: [`CohereLabs/cohere-transcribe-03-2026`](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026) at [`76b8b23`](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026/commit/76b8b23). + +Offline multilingual speech-to-text covering 14 languages (English, French, +German, Spanish, Italian, Portuguese, Dutch, Polish, Greek, Arabic, Japanese, +Chinese, Vietnamese, Korean). A Conformer encoder with a Transformer +encoder-decoder head (cross-attention, tied token embedding). Takes a 16 kHz +mono WAV and produces a transcript. Decoding is autoregressive. + ## What it's for @@ -15,9 +21,9 @@ autoregressive. See Cohere's [model card](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026) for training data, intended use, and upstream evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`76b8b23`](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026/commit/76b8b23e8607f35f0265a23d481b338fb0e26aea), -pinned 2026-04-16. + +Licensed Apache-2.0. Ported from upstream commit [`76b8b23`](https://huggingface.co/CohereLabs/cohere-transcribe-03-2026/commit/76b8b23), pinned 2026-04-16. Validated against the Transformers reference at transcribe.cpp commit [`bf0d0b7`](https://github.com/handy-computer/transcribe.cpp/tree/bf0d0b7) on 2026-04-18. + ## Input limits @@ -28,20 +34,49 @@ it into shorter segments. See the [input-length contract](../input-limits.md). ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [cohere-transcribe-03-2026-BF16.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-BF16.gguf) | 4.10 GB | 1.26% | -| F16 | [cohere-transcribe-03-2026-F16.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-F16.gguf) | 4.11 GB | 1.26% | -| Q8_0 | [cohere-transcribe-03-2026-Q8_0.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q8_0.gguf) | 2.41 GB | 1.27% | -| Q6_K | [cohere-transcribe-03-2026-Q6_K.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q6_K.gguf) | 1.97 GB | 1.27% | -| Q5_K_M | [cohere-transcribe-03-2026-Q5_K_M.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q5_K_M.gguf) | 1.76 GB | 1.25% | -| Q4_K_M | [cohere-transcribe-03-2026-Q4_K_M.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q4_K_M.gguf) | 1.55 GB | 1.25% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding and no external LM. BF16 reference baseline: 1.26%. -Cohere's self-reported number on the same split is 1.25% (Open ASR Leaderboard, -as of 2026-03-26). Text normalizer: Whisper `EnglishTextNormalizer` — the same -normalizer the Open ASR Leaderboard uses, so the comparison is apples-to-apples. +| BF16 | [cohere-transcribe-03-2026-BF16.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-BF16.gguf) | 4.11 GB | 1.26% | +| F16 | [cohere-transcribe-03-2026-F16.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-F16.gguf) | 4.11 GB | 1.26% | +| Q8_0 | [cohere-transcribe-03-2026-Q8_0.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q8_0.gguf) | 2.41 GB | 1.27% | +| Q6_K | [cohere-transcribe-03-2026-Q6_K.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q6_K.gguf) | 1.97 GB | 1.27% | +| Q5_K_M | [cohere-transcribe-03-2026-Q5_K_M.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q5_K_M.gguf) | 1.77 GB | 1.25% | +| Q4_K_M | [cohere-transcribe-03-2026-Q4_K_M.gguf](https://huggingface.co/handy-computer/cohere-transcribe-03-2026-gguf/resolve/main/cohere-transcribe-03-2026-Q4_K_M.gguf) | 1.56 GB | 1.24% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch sizes 1 and 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. BF16 reference baseline: 1.26%. Cohere's +self-reported number on the same split is 1.25% (Open ASR Leaderboard, as of +2026-03-26). Both ours and Cohere's numbers use the Whisper EnglishTextNormalizer, +so the comparison is apples-to-apples and our port matches the upstream reference +within rounding. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| ar | WER | 13.60% | +| de | WER | 5.06% | +| el | WER | 8.96% | +| en | WER | 5.08% | +| es | WER | 3.97% | +| fr | WER | 5.23% | +| it | WER | 3.24% | +| ja | CER | 5.13% | +| ko | CER | 6.57% | +| nl | WER | 7.16% | +| pl | WER | 6.15% | +| pt | WER | 5.18% | +| vi | WER | 7.39% | +| zh | CER | 11.18% | + ## Quick Start @@ -62,43 +97,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 150 ms (74×) | 154 ms (71×) | -| Metal | dots (35.3s) | 491 ms (72×) | 465 ms (76×) | -| CPU | jfk (11.0s) | 1.21 s (9×) | 1.05 s (11×) | -| CPU | dots (35.3s) | 4.13 s (9×) | 3.49 s (10×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 144 ms (76.51×) | 144 ms (76.32×) | +| Metal | dots (35.3s) | 470 ms (75.14×) | 492 ms (71.76×) | +| CPU | jfk (11.0s) | 926 ms (11.87×) | 1.00 s (10.95×) | +| CPU | dots (35.3s) | 3.25 s (10.86×) | 3.54 s (9.99×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 1.43 s (8×) | 1.33 s (8×) | -| Vulkan | dots (35.3s) | 4.25 s (8×) | 4.25 s (8×) | -| CPU | jfk (11.0s) | 3.57 s (3×) | 2.90 s (4×) | -| CPU | dots (35.3s) | 12.40 s (3×) | 10.08 s (4×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -------------: | -------------: | +| Vulkan | jfk (11.0s) | 1.43 s (7.67×) | 1.41 s (7.81×) | +| Vulkan | dots (35.3s) | 4.15 s (8.52×) | 4.00 s (8.83×) | +| CPU | jfk (11.0s) | 2.40 s (4.59×) | 2.47 s (4.46×) | +| CPU | dots (35.3s) | 8.76 s (4.03×) | 8.93 s (3.96×) | -Fedora 43, transcribe.cpp `2ab01b8`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models cohere-transcribe-03-2026 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name cohere-transcribe-03-2026-publication +uv run scripts/bench/run.py --profile --models cohere-transcribe-03-2026 ``` ## Numerical Validation @@ -106,8 +138,7 @@ uv run scripts/bench/run.py \ transcribe.cpp is validated tensor-by-tensor against the Transformers reference implementation on `samples/jfk.wav`. All 22 checkpointed tensors fall within family tolerance, and the final transcript matches the reference -verbatim. Last validated at commit -[`bf0d0b7`](https://github.com/handy-computer/transcribe.cpp/tree/bf0d0b7). +verbatim. | Field | Value | | --- | --- | @@ -116,26 +147,6 @@ verbatim. Last validated at commit | Manifest | `tests/golden/cohere/cohere-transcribe-03-2026.manifest.json` | | Command | `uv run scripts/validate.py compare --family cohere` | -Selected tensors: - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.mel.in` | `2.678e-01` | `4.238e-03` | fp64 vs fp32 STFT precision gap | -| `enc.pre_encode.out` | `8.794e+00` | `3.049e-02` | Mel gap propagated through pre-encoder | -| `enc.block.0.out` | `4.542e+00` | `1.939e-02` | Early encoder | -| `enc.block.23.out` | `3.173e+00` | `3.118e-02` | Mid-encoder | -| `enc.block.47.out` | `2.039e-01` | `4.615e-03` | Final encoder block | -| `enc.final` | `2.039e-01` | `4.615e-03` | Encoder output | -| `enc_dec_proj.out` | `3.693e-01` | `1.074e-02` | Encoder→decoder projection | -| `dec.token_emb` | `2.980e-08` | `2.910e-12` | Exact within fp32 round-off | -| `dec.pos_emb` | `0.000e+00` | `0.000e+00` | Exact | -| `dec.embed_norm` | `1.241e-01` | `1.327e-03` | LayerNorm output | -| `dec.block.0.out` | `3.409e+00` | `2.491e-02` | Early decoder | -| `dec.block.7.out` | `3.177e+01` | `1.416e-01` | Final decoder block (accumulated) | -| `dec.out_before_head` | `2.465e-01` | `1.339e-02` | Pre-head projection | -| `dec.logits_raw` | `6.675e-01` | `3.328e-02` | Raw logits | -| `dec.logits` | `nan` | `nan` | Softmax: `-inf` entries produce `nan`; first diff index 114692 is masked | - The expected divergence is in the frontend: C++ runs the STFT in fp64 where the reference runs fp32. The gap enters at the mel spectrogram, propagates through the encoder, and attenuates to a few tenths by the final encoder diff --git a/docs/models/cohere-transcribe-arabic-07-2026.md b/docs/models/cohere-transcribe-arabic-07-2026.md new file mode 100644 index 00000000..2bd3db7c --- /dev/null +++ b/docs/models/cohere-transcribe-arabic-07-2026.md @@ -0,0 +1,213 @@ +# Cohere Transcribe Arabic 07-2026 + + +Upstream: [`CohereLabs/cohere-transcribe-arabic-07-2026`](https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026) at [`0a8193c`](https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026/commit/0a8193c). + +Offline Arabic speech-to-text, including dialectal Arabic and +Arabic-English code-switching, with English as a secondary language. An +Arabic-focused adaptation of the Cohere Transcribe 03-2026 architecture: +a Conformer encoder with a Transformer encoder-decoder head +(cross-attention, tied token embedding). Takes a 16 kHz mono WAV and a +language flag (`-l ar` or `-l en`) and produces a transcript. Decoding +is autoregressive. + + +## What it's for + +Offline Arabic speech-to-text, including dialectal Arabic and +Arabic-English code-switching, with English as a secondary language. The +model takes a 16 kHz mono WAV and produces a transcript; pass the language +(`-l ar` or `-l en`). Decoding is autoregressive. + +See Cohere's [model card](https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026) +for training data, intended use, and upstream evaluation methodology. + + +Licensed Apache-2.0. Ported from upstream commit [`0a8193c`](https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026/commit/0a8193c), pinned 2026-07-07. Validated against the Transformers reference at transcribe.cpp commit [`d89ecb7`](https://github.com/handy-computer/transcribe.cpp/tree/d89ecb7) on 2026-07-07. + + +## Input limits + +Accepts up to about **6.7 minutes (400 s)** of 16 kHz mono audio per call — the +encoder's positional table is the binding limit. Longer audio is rejected up +front with `TRANSCRIBE_ERR_INPUT_TOO_LONG` rather than silently truncated; split +it into shorter segments. See the [input-length contract](../input-limits.md). + +## Download + + +| Quantization | Download | Size | WER (FLEURS ar) | +| --- | --- | ---: | ---: | +| BF16 | [cohere-transcribe-arabic-07-2026-BF16.gguf](https://huggingface.co/handy-computer/cohere-transcribe-arabic-07-2026-gguf/resolve/main/cohere-transcribe-arabic-07-2026-BF16.gguf) | 4.11 GB | 11.02% | +| F16 | [cohere-transcribe-arabic-07-2026-F16.gguf](https://huggingface.co/handy-computer/cohere-transcribe-arabic-07-2026-gguf/resolve/main/cohere-transcribe-arabic-07-2026-F16.gguf) | 4.11 GB | 11.00% | +| Q8_0 | [cohere-transcribe-arabic-07-2026-Q8_0.gguf](https://huggingface.co/handy-computer/cohere-transcribe-arabic-07-2026-gguf/resolve/main/cohere-transcribe-arabic-07-2026-Q8_0.gguf) | 2.41 GB | 11.06% | +| Q6_K | [cohere-transcribe-arabic-07-2026-Q6_K.gguf](https://huggingface.co/handy-computer/cohere-transcribe-arabic-07-2026-gguf/resolve/main/cohere-transcribe-arabic-07-2026-Q6_K.gguf) | 1.97 GB | 11.07% | +| Q5_K_M | [cohere-transcribe-arabic-07-2026-Q5_K_M.gguf](https://huggingface.co/handy-computer/cohere-transcribe-arabic-07-2026-gguf/resolve/main/cohere-transcribe-arabic-07-2026-Q5_K_M.gguf) | 1.77 GB | 10.95% | +| Q4_K_M | [cohere-transcribe-arabic-07-2026-Q4_K_M.gguf](https://huggingface.co/handy-computer/cohere-transcribe-arabic-07-2026-gguf/resolve/main/cohere-transcribe-arabic-07-2026-Q4_K_M.gguf) | 1.56 GB | 11.18% | + + + +WER on the full FLEURS ar split (428 utterances), batch size 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM, scored with the Whisper BasicTextNormalizer; the +FLEURS Arabic split is `ar_eg`, Egyptian-dialect speech. BF16 reference baseline, +measured with native Transformers on the same manifest: 11.00%; the BF16 port scores +11.02%, and every quant falls inside the reference's 95% confidence interval. FLEURS +Arabic is Egyptian-dialect speech; upstream numbers published on other Arabic test +sets are not directly comparable. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | Q5_K_M | +| --- | --- | ---: | ---: | +| en | WER | 4.88% | 4.88% | + +**LibriSpeech test-clean** + +| Language | Metric | BF16 | F16 | Q8_0 | Q6_K | Q5_K_M | Q4_K_M | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| en | WER | 1.33% | 1.33% | 1.34% | 1.34% | 1.34% | 1.34% | + + +## Quick Start + +```bash +cmake -B build +cmake --build build + +build/bin/transcribe-cli \ + -m models/cohere-transcribe-arabic-07-2026/cohere-transcribe-arabic-07-2026-Q8_0.gguf \ + -l ar \ + input.wav +``` + +If your audio is not already 16 kHz mono WAV, convert it first: + +```bash +ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav +``` + +## Performance + +The tables below were measured on +[Cohere Transcribe 03-2026](cohere-transcribe-03-2026.md). This variant is +the same architecture with identical tensor shapes and quantization layout +(only the weight values differ), so per-quant throughput carries over. + +### Apple M4 Max + + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 142 ms (77.53×) | 143 ms (77.15×) | +| Metal | dots (35.3s) | 469 ms (75.36×) | 460 ms (76.89×) | +| CPU | jfk (11.0s) | 912 ms (12.07×) | 1.31 s (8.37×) | +| CPU | dots (35.3s) | 3.36 s (10.53×) | 3.43 s (10.29×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + + +### AMD Ryzen 7 4750U Pro + + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -------------: | -------------: | +| Vulkan | jfk (11.0s) | 1.35 s (8.17×) | 1.31 s (8.40×) | +| Vulkan | dots (35.3s) | 4.19 s (8.44×) | 4.00 s (8.83×) | +| CPU | jfk (11.0s) | 2.40 s (4.58×) | 2.46 s (4.48×) | +| CPU | dots (35.3s) | 8.75 s (4.04×) | 8.94 s (3.95×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + + +Benchmark reproduction (substitute this variant's slug): + +```bash +uv run scripts/bench/run.py \ + --models cohere-transcribe-arabic-07-2026 \ + --quants q8_0,q4_k_m \ + --samples jfk,dots \ + --backends metal,cpu,vulkan \ + --iters 3 --warmup 1 \ + --name cohere-transcribe-arabic-07-2026-publication +``` + +## Numerical Validation + +The cohere family implementation is validated tensor-by-tensor against the +Transformers reference on the base +[Cohere Transcribe 03-2026](cohere-transcribe-03-2026.md#numerical-validation) +checkpoint (all 22 checkpointed tensors within family tolerance, transcript +verbatim). This variant shares that implementation unchanged — same +architecture, tensor shapes, and blob-identical SentencePiece tokenizer — +and is validated end-to-end: the C++ BF16 port scores 11.02% WER on the +full FLEURS Arabic test split against 11.00% for the native Transformers +reference on the same manifest (within +0.02pp), with per-utterance +hypotheses matching the reference on the upstream sample audio. A +per-variant golden tensor manifest has not been generated. + +| Field | Value | +| --- | --- | +| Reference | Transformers, `CohereLabs/cohere-transcribe-arabic-07-2026` | +| Reference WER runner | `scripts/wer/run_reference_cohere_transformers.py` | +| Family tensor manifest | `tests/golden/cohere/cohere-transcribe-03-2026.manifest.json` (base variant) | +| WER reports | `reports/wer/cohere-transcribe-arabic-07-2026-*.fleurs-ar.b8.jsonl` | + +## Reproduction + +### Convert + +Downloads the upstream HF repo via `huggingface-cli` (or an existing local +clone) and converts with the family-specific script. Output path is derived +from the repo id. The upstream repo is gated; accept the license on +Hugging Face first. + +```bash +uv run --project scripts/envs/cohere \ + scripts/convert-cohere.py CohereLabs/cohere-transcribe-arabic-07-2026 +``` + +### Quantize + +Run `transcribe-quantize` once per target quant. Example for F16; repeat with +`Q8_0`, `Q6_K`, `Q5_K_M`, `Q4_K_M`: + +```bash +build/bin/transcribe-quantize \ + models/cohere-transcribe-arabic-07-2026/cohere-transcribe-arabic-07-2026-BF16.gguf \ + models/cohere-transcribe-arabic-07-2026/cohere-transcribe-arabic-07-2026-F16.gguf \ + --quant F16 +``` + +### Run real-model tests + +```bash +cmake -B build -DTRANSCRIBE_BUILD_REAL_MODEL_TESTS=ON +cmake --build build + +TRANSCRIBE_COHERE_GGUF=models/cohere-transcribe-arabic-07-2026/cohere-transcribe-arabic-07-2026-BF16.gguf \ + ctest --test-dir build --output-on-failure -R 'cohere' +``` + +### WER + +```bash +uv run scripts/wer/ingest.py fleurs --lang ar + +uv run scripts/wer/run.py \ + --model models/cohere-transcribe-arabic-07-2026/cohere-transcribe-arabic-07-2026-BF16.gguf \ + --manifest samples/wer/fleurs-ar.manifest.jsonl \ + --language ar \ + --out reports/wer/cohere-transcribe-arabic-07-2026-BF16.fleurs-ar.jsonl + +uv run scripts/wer/score.py \ + reports/wer/cohere-transcribe-arabic-07-2026-BF16.fleurs-ar.jsonl --language ar +``` diff --git a/docs/models/cohere.md b/docs/models/cohere.md new file mode 100644 index 00000000..285c1260 --- /dev/null +++ b/docs/models/cohere.md @@ -0,0 +1,79 @@ +# Cohere Transcribe + +Cohere's [Transcribe](https://huggingface.co/CohereLabs) family ported to +transcribe.cpp. A large Conformer encoder paired with a lightweight +Transformer decoder (cross-attention, tied token embedding); decoding is +autoregressive with a language-conditioned prompt. The variants share one +architecture (identical tensor shapes) and differ in +training focus and language coverage. + +For the architecture deep-dive, validation contract, and porting notes, +see the family doc at +[`docs/porting/families/cohere.md`](../porting/families/cohere.md). + +## Choosing a variant + +- **Broad multilingual coverage.** `cohere-transcribe-03-2026` — 14 + languages (English, French, German, Spanish, Italian, Portuguese, + Dutch, Polish, Greek, Arabic, Japanese, Chinese, Vietnamese, Korean). +- **Arabic-focused.** `cohere-transcribe-arabic-07-2026` — retrained for + Arabic, including dialects and Arabic-English code-switching, with + English as a secondary language. Prefer it over the base model for + Arabic audio. + +## All variants + +WER is for the **Q8_0** preset, measured by transcribe.cpp's WER +pipeline; each variant is evaluated on the dataset that matches its +focus. See each per-variant doc for the full quant matrix and +methodology. + + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `cohere-transcribe-03-2026` | 2B | 14 languages | 2.41 GB | LibriSpeech test-clean (WER) | 1.27% | - | [cohere-transcribe-03-2026.md](cohere-transcribe-03-2026.md) | +| `cohere-transcribe-arabic-07-2026` | 2B | en, ar | 2.41 GB | FLEURS ar (WER) | 11.06% | - | [cohere-transcribe-arabic-07-2026.md](cohere-transcribe-arabic-07-2026.md) | + + +Pre-built GGUFs for every variant and quant are hosted under +[`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); +each per-variant doc has direct download links. + +## Input limits + +Every variant accepts up to about **6.7 minutes (400 s)** of 16 kHz mono audio +per call — the encoder's positional table is the binding limit, shared across +the family. Longer audio is rejected up front with +`TRANSCRIBE_ERR_INPUT_TOO_LONG` rather than silently truncated; split it into +shorter segments. See the [input-length contract](../input-limits.md). + +## Quick start + +Pick a variant and run (pass the audio's language with `-l`): + +```bash +cmake -B build +cmake --build build + +build/bin/transcribe-cli \ + -m models/cohere-transcribe-arabic-07-2026/cohere-transcribe-arabic-07-2026-Q8_0.gguf \ + -l ar \ + input.wav +``` + +The repo doesn't ship the GGUFs — pull them from the corresponding +`handy-computer/-gguf` repo on Hugging Face, or convert from +the upstream Cohere checkpoint via the per-variant doc's reproduction +section (the upstream repos are gated; accept the license first). + +## Capabilities + +All Cohere Transcribe variants support: + +- **Transcription** of 16 kHz mono WAV input across the variant's + supported languages (language hint required — no auto-detect). +- **Punctuation and capitalization** by default. + +What's not supported (consistent across the family): real-time +streaming, translation, timestamps, VAD, speaker diarization, auto +language detection. See the family doc for the full runtime contract. diff --git a/docs/models/diar_streaming_sortformer_4spk-v2.1.md b/docs/models/diar_streaming_sortformer_4spk-v2.1.md index 9c2f44d1..93952f7e 100644 --- a/docs/models/diar_streaming_sortformer_4spk-v2.1.md +++ b/docs/models/diar_streaming_sortformer_4spk-v2.1.md @@ -1,10 +1,15 @@ # Streaming Sortformer Diarizer 4spk v2.1 -NVIDIA's [`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) -ported to transcribe.cpp. A FastConformer encoder with an 18-layer -Transformer head that emits per-frame speaker-activity probabilities for -up to 4 speakers, running online with an Arrival-Order Speaker Cache -(AOSC) + FIFO. + +Upstream: [`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) at [`fafaab5`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1/commit/fafaab5). + +Streaming speaker diarization: who spoke when, for up to 4 speakers. +A FastConformer encoder with an 18-layer Transformer head emitting +per-frame speaker-activity probabilities, running online with an +Arrival-Order Speaker Cache (AOSC) + FIFO. NOT a transcription model: +a run produces speaker segments (start, end, speaker id in arrival +order), no text. Takes 16 kHz mono WAV. + ## What it's for @@ -18,26 +23,32 @@ speaker-attributed ASR path (future work). See NVIDIA's [model card](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) for training data, intended use, and upstream evaluation methodology. -Licensed under the NVIDIA Open Model License. Ported from upstream commit -[`fafaab5`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1/commit/fafaab5faa1617a0ca52d38dd3dc4bd636800d3d), -pinned 2026-07-19. + +Licensed NVIDIA Open Model License. Ported from upstream commit [`fafaab5`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1/commit/fafaab5), pinned 2026-07-19. Validated against the NeMo reference at transcribe.cpp commit [`d42c3bb`](https://github.com/handy-computer/transcribe.cpp/tree/d42c3bb) on 2026-07-22. + ## Download -| Quantization | Download | Size | DER (AMI IHM test) | + +| Quantization | Download | Size | DER (AMI IHM test) | | --- | --- | ---: | ---: | -| F32 | [diar_streaming_sortformer_4spk-v2.1-F32.gguf](https://huggingface.co/handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf/resolve/main/diar_streaming_sortformer_4spk-v2.1-F32.gguf) | 471 MB | 14.59% | -| F16 | [diar_streaming_sortformer_4spk-v2.1-F16.gguf](https://huggingface.co/handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf/resolve/main/diar_streaming_sortformer_4spk-v2.1-F16.gguf) | 237 MB | 14.23% | -| Q8_0 | [diar_streaming_sortformer_4spk-v2.1-Q8_0.gguf](https://huggingface.co/handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf/resolve/main/diar_streaming_sortformer_4spk-v2.1-Q8_0.gguf) | 139 MB | 14.73% | - -DER is measured on the full AMI IHM test set (16 meetings, ~9 h) against -forced-alignment RTTMs with dihard3-dev post-processing, collar 0.0, -overlap scored, at the `very_high_latency` operating point. Our measured -NeMo reference under the identical protocol is **14.83% DER / 19.89% -JER**; the C++ F32 port scores 14.59% / 19.51%. (Published DER numbers -for this model vary with the RTTM source and post-processing; manual -RTTMs score ~13 points worse than forced-alignment RTTMs on the same -system output. Compare like with like.) +| F32 | [diar_streaming_sortformer_4spk-v2.1-F32.gguf](https://huggingface.co/handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf/resolve/main/diar_streaming_sortformer_4spk-v2.1-F32.gguf) | 471 MB | 14.59% | +| F16 | [diar_streaming_sortformer_4spk-v2.1-F16.gguf](https://huggingface.co/handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf/resolve/main/diar_streaming_sortformer_4spk-v2.1-F16.gguf) | 237 MB | 14.23% | +| Q8_0 | [diar_streaming_sortformer_4spk-v2.1-Q8_0.gguf](https://huggingface.co/handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf/resolve/main/diar_streaming_sortformer_4spk-v2.1-Q8_0.gguf) | 139 MB | 14.73% | + + + +DER on the full AMI IHM test split (16 meetings). Figures without a commit were published before provenance was recorded. + + + +Scored against forced-alignment RTTMs with dihard3-dev post-processing, collar 0.0, +overlap scored, at the very_high_latency operating point. Measured NeMo reference +under the identical protocol: 14.83% DER / 19.89% JER; the C++ F32 port scores +14.59% / 19.51%. Published DER numbers vary with RTTM source and post-processing; +compare like with like. Only near-reference tiers ship for this family (k-quant +tiers withdrawn; see the transcribe.cpp family doc, "Quant policy (Stage 7)"). + Only near-reference tiers ship for this family. K-quant tiers were evaluated and withdrawn: the model's output depends on discrete @@ -85,31 +96,25 @@ second (many small windows). ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Default (model-config) -operating point. - ### Apple M4 -| Backend | Sample | F16 | Q8_0 | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 69 ms (159×) | 65 ms (171×) | -| Metal | dots (35.3s) | 318 ms (111×) | 320 ms (111×) | -| CPU | jfk (11.0s) | 137 ms (80×) | 110 ms (100×) | -| CPU | dots (35.3s) | 796 ms (44×) | 687 ms (51×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses. + +| Backend | Sample | F16 | Q8_0 | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 68 ms (161.23×) | 64 ms (172.42×) | +| Metal | dots (35.3s) | 316 ms (111.81×) | 318 ms (111.16×) | +| CPU | jfk (11.0s) | 136 ms (80.68×) | 109 ms (101.09×) | +| CPU | dots (35.3s) | 794 ms (44.49×) | 685 ms (51.59×) | -macOS 25.5.0, transcribe.cpp `d42c3bb`. +m4: transcribe.cpp `d42c3bb` on 2026-07-22. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models diar_streaming_sortformer_4spk-v2.1 \ - --quants f16,q8_0 \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name diar_streaming_sortformer_4spk-v2.1-publication +uv run scripts/bench/run.py --profile --models diar_streaming_sortformer_4spk-v2.1 ``` ## Numerical Validation @@ -119,8 +124,7 @@ transcribe.cpp is validated tensor-by-tensor against NeMo on mix with a 1.5 s overlap). All 6 checkpointed tensors fall within family tolerance, and the streaming AOSC cache-compression internals were additionally verified bit-exact against NeMo at the index level on a -full 39-minute AMI meeting (87 compression calls). Last validated at -commit `d42c3bb`. +full 39-minute AMI meeting (87 compression calls). | Field | Value | | --- | --- | @@ -129,17 +133,6 @@ commit `d42c3bb`. | Manifest | `tests/golden/sortformer/diar_streaming_sortformer_4spk-v2.1.manifest.json` | | Command | `uv run scripts/validate.py compare --family sortformer` | -Selected tensors: - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.mel.in` | `0.000e+00` | `0.000e+00` | Exact (shape differs by NeMo pad_to=16, values identical) | -| `enc.fastconformer.out`| `3.263e-03` | `1.396e-04` | F32 accumulation over 17 Conformer blocks | -| `enc.encoder_proj.out` | `9.829e-04` | `1.242e-04` | Drift attenuates through the projection | -| `enc.transformer.out` | `1.128e-03` | `1.323e-04` | 18-layer Transformer head | -| `diar.preds_offline` | `2.961e-04` | `5.007e-06` | Final sigmoid probabilities (offline) | -| `diar.probs` | `2.961e-04` | `5.007e-06` | Streaming path output (== offline on a single-chunk clip) | - ## Known Limitations - **Maximum 4 speakers** (architectural cap). More than 4 concurrent diff --git a/docs/models/fun-asr-mlt-nano-2512.md b/docs/models/fun-asr-mlt-nano-2512.md index a7e34600..b90b594a 100644 --- a/docs/models/fun-asr-mlt-nano-2512.md +++ b/docs/models/fun-asr-mlt-nano-2512.md @@ -1,13 +1,24 @@ # Fun-ASR-MLT-Nano -Alibaba / FunAudioLLM's [`FunAudioLLM/Fun-ASR-MLT-Nano-2512`](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512) -ported to transcribe.cpp — the multilingual sibling of -[Fun-ASR-Nano](fun-asr-nano-2512.md). Identical architecture -(SenseVoiceEncoderSmall + 2-layer audio adaptor + bundled Qwen3-0.6B LLM, -~800M trainable parameters), trained on a smaller corpus + +Upstream: [`FunAudioLLM/Fun-ASR-MLT-Nano-2512`](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512) at [`cf67a93`](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512/commit/cf67a93). + +Offline speech-to-text covering 31 languages, with focused optimization +on East and Southeast Asian languages: Chinese, English, Cantonese, +Japanese, Korean, Vietnamese, Indonesian, Thai, Malay, Filipino, plus +Arabic, Hindi, and 19 European languages (Bulgarian, Croatian, Czech, +Danish, Dutch, Estonian, Finnish, Greek, Hungarian, Irish, Latvian, +Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovenian, +Swedish). Same architecture as Fun-ASR-Nano-2512 (~800M trainable +parameters: frozen SenseVoiceEncoderSmall + 2-layer audio adaptor + +bundled Qwen3-0.6B LLM); trained on a smaller multilingual corpus ("hundreds of thousands of hours" per the model card, vs Nano's -"tens of millions") with broad multilingual coverage instead of -Mandarin-dialect depth. +"tens of millions"). Takes a 16 kHz mono WAV and emits text. Not +streaming, no translation, no timestamps. ITN (inverse text +normalization) is supported by the model and exposed via the +`--itn` CLI flag and `transcribe_funasr_nano_params { use_itn }` +in the library API. + ## What it's for @@ -36,33 +47,74 @@ zh/en/ja corpus and may give better Chinese accuracy. See FunAudioLLM's [model card](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512) for training data, intended use, and upstream evaluation methodology. -Licensed under the **FunASR Model Open Source License Agreement v1.1** -([MODEL_LICENSE](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE)). -Ported from upstream commit -[`cf67a93`](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512/commit/cf67a938bf2829959d08fdfb84e186eff02a67ff), -pinned 2026-05-06. + +Licensed [FunASR Model Open Source License Agreement v1.1](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE). Ported from upstream commit [`cf67a93`](https://huggingface.co/FunAudioLLM/Fun-ASR-MLT-Nano-2512/commit/cf67a93), pinned 2026-05-06. Validated against the FunASR reference at transcribe.cpp commit [`f094d28`](https://github.com/handy-computer/transcribe.cpp/tree/f094d28) on 2026-05-06. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [Fun-ASR-MLT-Nano-2512-BF16.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-BF16.gguf) | 1590 MB | 1.74% | -| F16 | [Fun-ASR-MLT-Nano-2512-F16.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-F16.gguf) | 1590 MB | 1.74% | -| Q8_0 | [Fun-ASR-MLT-Nano-2512-Q8_0.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q8_0.gguf) | 850 MB | 1.74% | -| Q6_K | [Fun-ASR-MLT-Nano-2512-Q6_K.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q6_K.gguf) | 659 MB | 1.69% | -| Q5_K_M | [Fun-ASR-MLT-Nano-2512-Q5_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q5_K_M.gguf) | 602 MB | 1.77% | -| Q4_K_M | [Fun-ASR-MLT-Nano-2512-Q4_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q4_K_M.gguf) | 531 MB | 1.89% | - -WER is measured on the full LibriSpeech test-clean split (2620 -utterances) with greedy LLM decoding via the bundled Qwen3-0.6B head. -The publisher does **not** report a numerical LibriSpeech WER for the MLT -variant specifically (the shared Fun-ASR README's per-model table covers -the regular Fun-ASR-Nano only). Gate baseline is our own FunASR 1.3.1 -reference run on the same manifest: 1.76% (95% CI [1.60%, 1.93%]). -transcribe.cpp's BF16 port matches that baseline within -0.02 -percentage-points; F16/Q8_0 are numerically indistinguishable. Q4_K_M is -the only quant with a visible regression (+0.13 pp); F16/Q8_0/Q6_K/Q5_K_M -are within bootstrap noise of BF16. +| BF16 | [Fun-ASR-MLT-Nano-2512-BF16.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-BF16.gguf) | 1.67 GB | 1.74% | +| F16 | [Fun-ASR-MLT-Nano-2512-F16.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-F16.gguf) | 1.67 GB | 1.74% | +| Q8_0 | [Fun-ASR-MLT-Nano-2512-Q8_0.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q8_0.gguf) | 891 MB | 1.74% | +| Q6_K | [Fun-ASR-MLT-Nano-2512-Q6_K.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q6_K.gguf) | 691 MB | 1.69% | +| Q5_K_M | [Fun-ASR-MLT-Nano-2512-Q5_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q5_K_M.gguf) | 631 MB | 1.77% | +| Q4_K_M | [Fun-ASR-MLT-Nano-2512-Q4_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-MLT-Nano-2512-gguf/resolve/main/Fun-ASR-MLT-Nano-2512-Q4_K_M.gguf) | 557 MB | 1.89% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances). Figures without a commit were published before provenance was recorded. + + + +Greedy LLM decoding via the bundled Qwen3-0.6B head. The publisher does not report a +numerical LibriSpeech WER for the MLT variant specifically (the shared README's +per-model table covers Fun-ASR-Nano only). Gate baseline is our own FunASR 1.3.1 +reference run on the same manifest: 1.76% (95% CI [1.60%, 1.93%]). transcribe.cpp's +BF16 port matches that baseline within -0.02 percentage-points. LibriSpeech is +English only; the strength of the MLT variant is multilingual coverage, not English +accuracy. For the other 30 languages, run your own representative manifest. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| ar | WER | 25.79% | +| bg | WER | 84.98% | +| cs | WER | 53.56% | +| da | WER | 69.93% | +| el | WER | 103.55% | +| en | WER | 4.90% | +| et | WER | 64.22% | +| fi | WER | 68.16% | +| fil | WER | 15.62% | +| ga | WER | 100.08% | +| hi | WER | 43.96% | +| hr | WER | 61.07% | +| hu | WER | 113.21% | +| id | WER | 7.52% | +| ja | CER | 2.32% | +| ko | CER | 5.20% | +| lt | WER | 78.42% | +| lv | WER | 56.71% | +| ms | WER | 9.92% | +| mt | WER | 91.71% | +| nl | WER | 42.97% | +| pl | WER | 59.34% | +| pt | WER | 28.24% | +| ro | WER | 74.39% | +| sk | WER | 64.01% | +| sl | WER | 75.22% | +| sv | WER | 75.36% | +| th | CER | 7.99% | +| vi | WER | 8.32% | +| yue | CER | 12.72% | +| zh | CER | 8.64% | + LibriSpeech is English only and is not the strength of this model. For the other 30 languages, run your own representative manifest. CommonVoice @@ -96,55 +148,49 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 156 ms (70×) | 144 ms (76×) | -| Metal | dots (35.3s) | 539 ms (66×) | 499 ms (71×) | -| CPU | jfk (11.0s) | 661 ms (17×) | 575 ms (19×) | -| CPU | dots (35.3s) | 2.36 s (15×) | 2.12 s (17×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 146 ms (75.38×) | 136 ms (80.64×) | +| Metal | dots (35.3s) | 546 ms (64.69×) | 489 ms (72.22×) | +| CPU | jfk (11.0s) | 533 ms (20.65×) | 537 ms (20.50×) | +| CPU | dots (35.3s) | 1.92 s (18.43×) | 1.93 s (18.28×) | -macOS 26.4.1, transcribe.cpp `f094d28`. MLT is ~10–15% slower than -Fun-ASR-Nano on the same hardware; the gap is from per-step LLM -generation cost (different decoded transcript lengths between the two -variants). +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 1.14 s (10×) | 1.03 s (11×) | -| Vulkan | dots (35.3s) | 4.48 s (8×) | 3.92 s (9×) | -| CPU | jfk (11.0s) | 2.35 s (5×) | 1.84 s (6×) | -| CPU | dots (35.3s) | 8.60 s (4×) | 6.97 s (5×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -------------: | --------------: | +| Vulkan | jfk (11.0s) | 1.13 s (9.74×) | 1.04 s (10.61×) | +| Vulkan | dots (35.3s) | 4.45 s (7.93×) | 4.03 s (8.78×) | +| CPU | jfk (11.0s) | 1.79 s (6.16×) | 1.77 s (6.20×) | +| CPU | dots (35.3s) | 7.40 s (4.78×) | 6.94 s (5.09×) | -Fedora 43, transcribe.cpp `8635bd1`. Vulkan device: `AMD Radeon Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models Fun-ASR-MLT-Nano-2512 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name fun-asr-mlt-nano-2512-publication +uv run scripts/bench/run.py --profile --models fun-asr-mlt-nano-2512 ``` ## Numerical Validation -transcribe.cpp is validated tensor-by-tensor against FunASR 1.3.1 -on `samples/jfk.wav`. All 22 checkpointed tensors fall within family +transcribe.cpp is validated tensor-by-tensor against FunASR 1.3.1 on +`samples/jfk.wav`. All 22 checkpointed tensors fall within family tolerance, and the final transcript matches the FunASR reference verbatim ("and so my fellow americans ask not what your country can do for you ask -what you can do for your country"). Last validated at commit -[`f094d28`](https://github.com/handy-computer/transcribe.cpp/tree/f094d28). +what you can do for your country"). | Field | Value | | --- | --- | diff --git a/docs/models/fun-asr-nano-2512.md b/docs/models/fun-asr-nano-2512.md index 4ac5a283..43d335dd 100644 --- a/docs/models/fun-asr-nano-2512.md +++ b/docs/models/fun-asr-nano-2512.md @@ -1,10 +1,20 @@ # Fun-ASR-Nano -Alibaba / FunAudioLLM's [`FunAudioLLM/Fun-ASR-Nano-2512`](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) -ported to transcribe.cpp. ~800M trainable parameters wrapping a frozen -**SenseVoiceEncoderSmall** (50 SAN-M main blocks + 20 transformer blocks), -a 2-layer audio adaptor (512 → 1024), and a bundled **Qwen3-0.6B** LLM + +Upstream: [`FunAudioLLM/Fun-ASR-Nano-2512`](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) at [`a7088d6`](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512/commit/a7088d6). + +Offline speech-to-text in Chinese, English, and Japanese, plus 7 Chinese +dialects (Wu, Cantonese, Min, Hakka, Gan, Xiang, Jin) and 26 regional +Mandarin accents. ~800M trainable parameters wrapping a frozen +SenseVoiceEncoderSmall (50 SAN-M main blocks + 20 transformer blocks), +a 2-layer audio adaptor (512 → 1024), and a bundled Qwen3-0.6B LLM (28 layers, 16/8 GQA, BF16) that produces the transcript autoregressively. +Takes a 16 kHz mono WAV and emits text. Not a streaming model, no +translation, no built-in long-form chunking, no timestamps. ITN +(inverse text normalization) is supported by the model and exposed +via the `--itn` CLI flag and `transcribe_funasr_nano_params { use_itn }` +in the library API. + ## What it's for @@ -24,30 +34,45 @@ For multilingual coverage beyond zh/en/ja, see the sibling See FunAudioLLM's [model card](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512) for training data, intended use, and upstream evaluation methodology. -Licensed under the **FunASR Model Open Source License Agreement v1.1** -([MODEL_LICENSE](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE)). -Ported from upstream commit -[`a7088d6`](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512/commit/a7088d620f755dcdca575b63db184c3ad55b2865), -pinned 2026-05-06. + +Licensed [FunASR Model Open Source License Agreement v1.1](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE). Ported from upstream commit [`a7088d6`](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512/commit/a7088d6), pinned 2026-05-06. Validated against the FunASR reference at transcribe.cpp commit [`f094d28`](https://github.com/handy-computer/transcribe.cpp/tree/f094d28) on 2026-05-06. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [Fun-ASR-Nano-2512-BF16.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-BF16.gguf) | 1590 MB | 1.78% | -| F16 | [Fun-ASR-Nano-2512-F16.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-F16.gguf) | 1590 MB | 1.79% | -| Q8_0 | [Fun-ASR-Nano-2512-Q8_0.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q8_0.gguf) | 850 MB | 1.79% | -| Q6_K | [Fun-ASR-Nano-2512-Q6_K.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q6_K.gguf) | 659 MB | 1.78% | -| Q5_K_M | [Fun-ASR-Nano-2512-Q5_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q5_K_M.gguf) | 602 MB | 1.82% | -| Q4_K_M | [Fun-ASR-Nano-2512-Q4_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q4_K_M.gguf) | 531 MB | 1.92% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy LLM decoding via the bundled Qwen3-0.6B head. Publisher -reports 1.76% on this split (model card "Open-Source Dataset Performance" -table). Our FunASR 1.3.1 reference run scores 1.79% (95% CI [1.63%, 1.95%]), -within bootstrap noise of the publisher's number. transcribe.cpp's BF16 -port matches that baseline within -0.01 percentage-points; F16/Q8_0/Q6_K -are numerically indistinguishable. +| BF16 | [Fun-ASR-Nano-2512-BF16.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-BF16.gguf) | 1.67 GB | 1.78% | +| F16 | [Fun-ASR-Nano-2512-F16.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-F16.gguf) | 1.67 GB | 1.79% | +| Q8_0 | [Fun-ASR-Nano-2512-Q8_0.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q8_0.gguf) | 891 MB | 1.79% | +| Q6_K | [Fun-ASR-Nano-2512-Q6_K.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q6_K.gguf) | 691 MB | 1.78% | +| Q5_K_M | [Fun-ASR-Nano-2512-Q5_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q5_K_M.gguf) | 631 MB | 1.82% | +| Q4_K_M | [Fun-ASR-Nano-2512-Q4_K_M.gguf](https://huggingface.co/handy-computer/Fun-ASR-Nano-2512-gguf/resolve/main/Fun-ASR-Nano-2512-Q4_K_M.gguf) | 557 MB | 1.92% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances). Figures without a commit were published before provenance was recorded. + + + +Greedy LLM decoding via the bundled Qwen3-0.6B head. Publisher reports 1.76% on this +split (model card "Open-Source Dataset Performance" table). Our FunASR 1.3.1 +reference run scores 1.79% (95% CI [1.63%, 1.95%]), within bootstrap noise of the +publisher's number. transcribe.cpp's BF16 port matches that baseline within -0.01 +percentage-points. LibriSpeech is an English-only benchmark; Chinese (AISHELL-1, +WenetSpeech) and Japanese (CommonVoice JA) are the recommended complementary checks. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 5.49% | +| ja | CER | 8.50% | +| zh | CER | 8.59% | + LibriSpeech is an English benchmark; Fun-ASR-Nano's strongest case is Mandarin. **FLEURS-zh** (945 utterances) CER: 8.61% on our FunASR 1.3.1 @@ -84,52 +109,49 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 134 ms (82×) | 129 ms (86×) | -| Metal | dots (35.3s) | 486 ms (73×) | 433 ms (82×) | -| CPU | jfk (11.0s) | 379 ms (29×) | 358 ms (31×) | -| CPU | dots (35.3s) | 1.40 s (25×) | 1.31 s (27×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 132 ms (83.13×) | 124 ms (88.83×) | +| Metal | dots (35.3s) | 483 ms (73.13×) | 449 ms (78.65×) | +| CPU | jfk (11.0s) | 365 ms (30.12×) | 362 ms (30.40×) | +| CPU | dots (35.3s) | 1.36 s (26.00×) | 1.32 s (26.86×) | -macOS 26.4.1, transcribe.cpp `f094d28`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 887 ms (12×) | 825 ms (13×) | -| Vulkan | dots (35.3s) | 3.74 s (9×) | 2.95 s (12×) | -| CPU | jfk (11.0s) | 1.48 s (7×) | 1.15 s (10×) | -| CPU | dots (35.3s) | 5.54 s (6×) | 4.52 s (8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 905 ms (12.16×) | 838 ms (13.12×) | +| Vulkan | dots (35.3s) | 3.80 s (9.30×) | 3.06 s (11.54×) | +| CPU | jfk (11.0s) | 1.23 s (8.94×) | 1.15 s (9.60×) | +| CPU | dots (35.3s) | 5.08 s (6.95×) | 4.66 s (7.59×) | -Fedora 43, transcribe.cpp `8635bd1`. Vulkan device: `AMD Radeon Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models Fun-ASR-Nano-2512 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name fun-asr-nano-2512-publication +uv run scripts/bench/run.py --profile --models fun-asr-nano-2512 ``` ## Numerical Validation -transcribe.cpp is validated tensor-by-tensor against FunASR 1.3.1 -on `samples/jfk.wav`. All 22 checkpointed tensors fall within family +transcribe.cpp is validated tensor-by-tensor against FunASR 1.3.1 on +`samples/jfk.wav`. All 22 checkpointed tensors fall within family tolerance, and the final transcript matches the FunASR reference verbatim ("And so my fellow Americans ask not what your country can do for you ask -what you can do for your country."). Last validated at commit -[`f094d28`](https://github.com/handy-computer/transcribe.cpp/tree/f094d28). +what you can do for your country."). | Field | Value | | --- | --- | diff --git a/docs/models/fun-asr-nano.md b/docs/models/fun-asr-nano.md index 07beb665..7d2c77c3 100644 --- a/docs/models/fun-asr-nano.md +++ b/docs/models/fun-asr-nano.md @@ -34,10 +34,12 @@ transcribe.cpp's WER pipeline. See each per-variant doc for the full quant matrix and per-language WER/CER on the language each variant targets. -| Variant | Params | Q8_0 size | WER (Q8_0) | Languages | Doc | -| --- | ---: | ---: | ---: | --- | --- | -| `fun-asr-nano-2512` | ~800M | 850 MB | 1.79% | zh, en, ja + 7 dialects | [fun-asr-nano-2512.md](fun-asr-nano-2512.md) | -| `fun-asr-mlt-nano-2512` | ~800M | 850 MB | 1.74% | 31 languages | [fun-asr-mlt-nano-2512.md](fun-asr-mlt-nano-2512.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `fun-asr-nano-2512` | 830M | zh, en, ja | 891 MB | LibriSpeech test-clean (WER) | 1.79% | - | [fun-asr-nano-2512.md](fun-asr-nano-2512.md) | +| `fun-asr-mlt-nano-2512` | 830M | 31 languages | 891 MB | LibriSpeech test-clean (WER) | 1.74% | - | [fun-asr-mlt-nano-2512.md](fun-asr-mlt-nano-2512.md) | + Pre-built GGUFs for every variant and quant are hosted under [`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); diff --git a/docs/models/gigaam-v3-ctc.md b/docs/models/gigaam-v3-ctc.md index 046d7265..932ffc57 100644 --- a/docs/models/gigaam-v3-ctc.md +++ b/docs/models/gigaam-v3-ctc.md @@ -1,7 +1,10 @@ # GigaAM-v3 CTC (charwise) -ai-sage's [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) -(ctc branch) ported to transcribe.cpp. Same 16-layer Conformer encoder as the RNN-T variants, paired with a 1×1 Conv1d CTC head. CTC removes the per-symbol LSTM loop entirely (fast straight-line argmax-then-collapse decode). Slightly higher WER than the RNN-T variant at the same encoder size. + +Upstream: [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) at [`15ef3b5`](https://huggingface.co/ai-sage/GigaAM-v3/commit/15ef3b5). + +Offline Russian speech-to-text with greedy CTC decoding. 16-layer Conformer encoder with a 1×1 Conv1d CTC head. Output is lowercased Russian, no punctuation; 33-entry character vocabulary. + ## What it's for @@ -25,23 +28,35 @@ across heads). Variants in this family: See ai-sage's [model card](https://huggingface.co/ai-sage/GigaAM-v3) for training data, intended use, and upstream evaluation methodology. -Licensed MIT. Ported from upstream commit -[`15ef3b5`](https://huggingface.co/ai-sage/GigaAM-v3/commit/15ef3b5a88da78f93134b3cb7f015c70aefa8946), -pinned 2026-05-12. + +Licensed MIT. Ported from upstream commit [`15ef3b5`](https://huggingface.co/ai-sage/GigaAM-v3/commit/15ef3b5), pinned 2026-05-12. Validated against the gigaam author package reference at transcribe.cpp commit [`42b96d9`](https://github.com/handy-computer/transcribe.cpp/tree/42b96d9) on 2026-05-12. + ## Download -| Quantization | Download | Size | WER (FLEURS ru) | + +| Quantization | Download | Size | WER (FLEURS ru) | | --- | --- | ---: | ---: | -| F32 | [gigaam-v3-ctc-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-F32.gguf) | 842 MB | 8.42% | -| F16 | [gigaam-v3-ctc-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-F16.gguf) | 428 MB | 8.42% | -| Q8_0 | [gigaam-v3-ctc-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q8_0.gguf) | 259 MB | 8.40% | -| Q6_K | [gigaam-v3-ctc-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q6_K.gguf) | 216 MB | 8.38% | -| Q5_K_M | [gigaam-v3-ctc-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q5_K_M.gguf) | 195 MB | 8.29% | -| Q4_K_M | [gigaam-v3-ctc-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q4_K_M.gguf) | 174 MB | 8.42% | - -WER is measured on the full FLEURS ru test split (775 utterances) with -greedy decoding and no external LM. F32 reference baseline: **8.42%**. +| F32 | [gigaam-v3-ctc-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-F32.gguf) | 883 MB | 8.42% | +| F16 | [gigaam-v3-ctc-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-F16.gguf) | 449 MB | 8.42% | +| Q8_0 | [gigaam-v3-ctc-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q8_0.gguf) | 272 MB | 8.42% | +| Q6_K | [gigaam-v3-ctc-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q6_K.gguf) | 226 MB | 8.38% | +| Q5_K_M | [gigaam-v3-ctc-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q5_K_M.gguf) | 205 MB | 8.29% | +| Q4_K_M | [gigaam-v3-ctc-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-ctc-gguf/resolve/main/gigaam-v3-ctc-Q4_K_M.gguf) | 182 MB | 8.42% | + + + +WER on the full FLEURS ru split (775 utterances), batch sizes 1 and 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 8.42%. Upstream `gigaam` +author package measured on the same manifest: 9.81%; the 1.4 pp gap is upstream +rejecting 5 long (>25 s) utterances with `Too long wav file, use +'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt +subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not +publish a FLEURS ru WER; this number is measured here. + Upstream (`gigaam` author package at `6e4b027c`) measured on the same manifest: **9.81%**. The 1.4 pp gap is the upstream package @@ -73,39 +88,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | ------------: | ------------: | -| Metal | ru (4.5s) | 31 ms (146×) | 32 ms (142×) | -| CPU | ru (4.5s) | 157 ms (29×) | 152 ms (30×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `ef55b52`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Metal | ru-short (11.0s) | 26 ms (415.59×) | 27 ms (405.11×) | +| Metal | ru-long (33.8s) | 62 ms (541.17×) | 64 ms (525.83×) | +| CPU | ru-short (11.0s) | 367 ms (29.92×) | 364 ms (30.21×) | +| CPU | ru-long (33.8s) | 1.16 s (29.15×) | 1.26 s (26.90×) | + +Apple M4 Max: transcribe.cpp `94f1f45` on 2026-09-15. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | -----------: | -----------: | -| Vulkan | ru (4.5s) | 137 ms (33×) | 141 ms (32×) | -| CPU | ru (4.5s) | 488 ms (9×) | 383 ms (12×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Vulkan | ru-short (11.0s) | 284 ms (38.65×) | 291 ms (37.72×) | +| Vulkan | ru-long (33.8s) | 828 ms (40.89×) | 846 ms (40.00×) | +| CPU | ru-short (11.0s) | 793 ms (13.84×) | 902 ms (12.18×) | +| CPU | ru-long (33.8s) | 3.45 s (9.81×) | 3.70 s (9.14×) | -Fedora Linux 43, transcribe.cpp `ef55b52`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `522ccd68` on 2026-09-15. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models gigaam-v3-ctc,gigaam-v3-rnnt,gigaam-v3-e2e-ctc,gigaam-v3-e2e-rnnt \ - --quants q8_0,q4_k_m \ - --samples ru \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name gigaam-publication +uv run scripts/bench/run.py --profile --models gigaam-v3-ctc ``` ## Numerical Validation diff --git a/docs/models/gigaam-v3-e2e-ctc.md b/docs/models/gigaam-v3-e2e-ctc.md index 2d81f8b4..97dcca9a 100644 --- a/docs/models/gigaam-v3-e2e-ctc.md +++ b/docs/models/gigaam-v3-e2e-ctc.md @@ -1,7 +1,10 @@ # GigaAM-v3 e2e-CTC -ai-sage's [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) -(e2e_ctc branch) ported to transcribe.cpp. Same 16-layer Conformer encoder as `gigaam-v3-e2e-rnnt`, paired with a 1×1 Conv1d CTC head. 256-piece SentencePiece vocabulary keeps the head compact while preserving punctuation and Cyrillic casing in output. Faster than RNN-T at comparable accuracy on short utterances. + +Upstream: [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) at [`cec030b`](https://huggingface.co/ai-sage/GigaAM-v3/commit/cec030b). + +Offline Russian speech-to-text with greedy CTC decoding. 16-layer Conformer encoder with a 1×1 Conv1d CTC head. Output is cased Russian with punctuation, decoded from a 256-piece SentencePiece tokenizer. + ## What it's for @@ -25,23 +28,35 @@ across heads). Variants in this family: See ai-sage's [model card](https://huggingface.co/ai-sage/GigaAM-v3) for training data, intended use, and upstream evaluation methodology. -Licensed MIT. Ported from upstream commit -[`cec030b`](https://huggingface.co/ai-sage/GigaAM-v3/commit/cec030b4c4f35d928e4a9044a3bdb29ebd499fac), -pinned 2026-05-12. + +Licensed MIT. Ported from upstream commit [`cec030b`](https://huggingface.co/ai-sage/GigaAM-v3/commit/cec030b), pinned 2026-05-12. Validated against the gigaam author package reference at transcribe.cpp commit [`42b96d9`](https://github.com/handy-computer/transcribe.cpp/tree/42b96d9) on 2026-05-12. + ## Download -| Quantization | Download | Size | WER (FLEURS ru) | + +| Quantization | Download | Size | WER (FLEURS ru) | | --- | --- | ---: | ---: | -| F32 | [gigaam-v3-e2e-ctc-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-F32.gguf) | 843 MB | 5.50% | -| F16 | [gigaam-v3-e2e-ctc-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-F16.gguf) | 428 MB | 5.50% | -| Q8_0 | [gigaam-v3-e2e-ctc-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q8_0.gguf) | 260 MB | 5.50% | -| Q6_K | [gigaam-v3-e2e-ctc-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q6_K.gguf) | 216 MB | 5.56% | -| Q5_K_M | [gigaam-v3-e2e-ctc-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q5_K_M.gguf) | 195 MB | 5.58% | -| Q4_K_M | [gigaam-v3-e2e-ctc-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q4_K_M.gguf) | 174 MB | 5.57% | - -WER is measured on the full FLEURS ru test split (775 utterances) with -greedy decoding and no external LM. F32 reference baseline: **5.50%**. +| F32 | [gigaam-v3-e2e-ctc-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-F32.gguf) | 884 MB | 5.50% | +| F16 | [gigaam-v3-e2e-ctc-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-F16.gguf) | 449 MB | 5.50% | +| Q8_0 | [gigaam-v3-e2e-ctc-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q8_0.gguf) | 272 MB | 5.53% | +| Q6_K | [gigaam-v3-e2e-ctc-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q6_K.gguf) | 226 MB | 5.56% | +| Q5_K_M | [gigaam-v3-e2e-ctc-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q5_K_M.gguf) | 205 MB | 5.58% | +| Q4_K_M | [gigaam-v3-e2e-ctc-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-ctc-gguf/resolve/main/gigaam-v3-e2e-ctc-Q4_K_M.gguf) | 182 MB | 5.57% | + + + +WER on the full FLEURS ru split (775 utterances), batch sizes 1 and 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 5.50%. Upstream `gigaam` +author package measured on the same manifest: 6.93%; the 1.4 pp gap is upstream +rejecting 5 long (>25 s) utterances with `Too long wav file, use +'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt +subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not +publish a FLEURS ru WER; this number is measured here. + Upstream (`gigaam` author package at `6e4b027c`) measured on the same manifest: **6.93%**. The 1.4 pp gap is the upstream package @@ -73,39 +88,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | ------------: | ------------: | -| Metal | ru (4.5s) | 40 ms (112×) | 40 ms (111×) | -| CPU | ru (4.5s) | 164 ms (27×) | 161 ms (28×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `ef55b52`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Metal | ru-short (11.0s) | 27 ms (405.69×) | 27 ms (411.99×) | +| Metal | ru-long (33.8s) | 64 ms (525.57×) | 66 ms (512.90×) | +| CPU | ru-short (11.0s) | 368 ms (29.86×) | 361 ms (30.39×) | +| CPU | ru-long (33.8s) | 1.16 s (29.17×) | 1.24 s (27.21×) | + +Apple M4 Max: transcribe.cpp `94f1f45` on 2026-09-15. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | -----------: | -----------: | -| Vulkan | ru (4.5s) | 152 ms (30×) | 155 ms (29×) | -| CPU | ru (4.5s) | 494 ms (9×) | 397 ms (11×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Vulkan | ru-short (11.0s) | 318 ms (34.53×) | 326 ms (33.69×) | +| Vulkan | ru-long (33.8s) | 934 ms (36.22×) | 954 ms (35.48×) | +| CPU | ru-short (11.0s) | 824 ms (13.33×) | 928 ms (11.83×) | +| CPU | ru-long (33.8s) | 3.53 s (9.58×) | 3.74 s (9.06×) | -Fedora Linux 43, transcribe.cpp `ef55b52`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `522ccd68` on 2026-09-15. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models gigaam-v3-ctc,gigaam-v3-rnnt,gigaam-v3-e2e-ctc,gigaam-v3-e2e-rnnt \ - --quants q8_0,q4_k_m \ - --samples ru \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name gigaam-publication +uv run scripts/bench/run.py --profile --models gigaam-v3-e2e-ctc ``` ## Numerical Validation diff --git a/docs/models/gigaam-v3-e2e-rnnt.md b/docs/models/gigaam-v3-e2e-rnnt.md index 75b9eb4a..b6cd80f3 100644 --- a/docs/models/gigaam-v3-e2e-rnnt.md +++ b/docs/models/gigaam-v3-e2e-rnnt.md @@ -1,7 +1,10 @@ # GigaAM-v3 e2e-RNN-T -ai-sage's [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) -(main (= v3_e2e_rnnt) branch) ported to transcribe.cpp. A 16-layer Conformer encoder (768-d, 16 heads, rotary positional embeddings, conv1d ×4 subsampling) feeding an RNN-T transducer head (single LSTM-320 predictor + joint network). Vocabulary is 1024 SentencePiece pieces + blank, covering cased Cyrillic plus `.,?!` directly. + +Upstream: [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) at [`ec1dc1f`](https://huggingface.co/ai-sage/GigaAM-v3/commit/ec1dc1f). + +Offline Russian speech-to-text with greedy RNN-T decoding. 16-layer Conformer encoder paired with an RNN-T transducer head. Output is cased Russian with punctuation, decoded from a 1024-piece SentencePiece tokenizer. Not a streaming model and does not translate. Short-form only (≤25 s per utterance). + ## What it's for @@ -25,23 +28,35 @@ across heads). Variants in this family: See ai-sage's [model card](https://huggingface.co/ai-sage/GigaAM-v3) for training data, intended use, and upstream evaluation methodology. -Licensed MIT. Ported from upstream commit -[`ec1dc1f`](https://huggingface.co/ai-sage/GigaAM-v3/commit/ec1dc1f01d0d627ab2c0d3acc1e235702300d95e), -pinned 2026-05-12. + +Licensed MIT. Ported from upstream commit [`ec1dc1f`](https://huggingface.co/ai-sage/GigaAM-v3/commit/ec1dc1f), pinned 2026-05-12. Validated against the gigaam author package reference at transcribe.cpp commit [`42b96d9`](https://github.com/handy-computer/transcribe.cpp/tree/42b96d9) on 2026-05-12. + ## Download -| Quantization | Download | Size | WER (FLEURS ru) | + +| Quantization | Download | Size | WER (FLEURS ru) | | --- | --- | ---: | ---: | -| F32 | [gigaam-v3-e2e-rnnt-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-F32.gguf) | 849 MB | 5.35% | -| F16 | [gigaam-v3-e2e-rnnt-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-F16.gguf) | 431 MB | 5.35% | -| Q8_0 | [gigaam-v3-e2e-rnnt-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q8_0.gguf) | 261 MB | 5.36% | -| Q6_K | [gigaam-v3-e2e-rnnt-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q6_K.gguf) | 217 MB | 5.37% | -| Q5_K_M | [gigaam-v3-e2e-rnnt-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q5_K_M.gguf) | 197 MB | 5.42% | -| Q4_K_M | [gigaam-v3-e2e-rnnt-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q4_K_M.gguf) | 175 MB | 5.36% | - -WER is measured on the full FLEURS ru test split (775 utterances) with -greedy decoding and no external LM. F32 reference baseline: **5.35%**. +| F32 | [gigaam-v3-e2e-rnnt-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-F32.gguf) | 890 MB | 5.35% | +| F16 | [gigaam-v3-e2e-rnnt-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-F16.gguf) | 452 MB | 5.35% | +| Q8_0 | [gigaam-v3-e2e-rnnt-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q8_0.gguf) | 274 MB | 5.35% | +| Q6_K | [gigaam-v3-e2e-rnnt-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q6_K.gguf) | 228 MB | 5.37% | +| Q5_K_M | [gigaam-v3-e2e-rnnt-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q5_K_M.gguf) | 206 MB | 5.42% | +| Q4_K_M | [gigaam-v3-e2e-rnnt-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-e2e-rnnt-gguf/resolve/main/gigaam-v3-e2e-rnnt-Q4_K_M.gguf) | 184 MB | 5.36% | + + + +WER on the full FLEURS ru split (775 utterances), batch sizes 1 and 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 5.35%. Upstream `gigaam` +author package measured on the same manifest: 6.78%; the 1.4 pp gap is upstream +rejecting 5 long (>25 s) utterances with `Too long wav file, use +'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt +subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not +publish a FLEURS ru WER; this number is measured here. + Upstream (`gigaam` author package at `6e4b027c`) measured on the same manifest: **6.78%**. The 1.4 pp gap is the upstream package @@ -73,39 +88,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | -----------: | -----------: | -| Metal | ru (4.5s) | 51 ms (88×) | 51 ms (89×) | -| CPU | ru (4.5s) | 177 ms (25×) | 172 ms (26×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `ef55b52`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Metal | ru-short (11.0s) | 30 ms (360.53×) | 30 ms (360.92×) | +| Metal | ru-long (33.8s) | 74 ms (458.32×) | 75 ms (449.86×) | +| CPU | ru-short (11.0s) | 379 ms (28.99×) | 368 ms (29.88×) | +| CPU | ru-long (33.8s) | 1.18 s (28.71×) | 1.26 s (26.90×) | + +Apple M4 Max: transcribe.cpp `94f1f45` on 2026-09-15. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | -----------: | -----------: | -| Vulkan | ru (4.5s) | 202 ms (22×) | 205 ms (22×) | -| CPU | ru (4.5s) | 552 ms (8×) | 444 ms (10×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Vulkan | ru-short (11.0s) | 797 ms (13.78×) | 800 ms (13.73×) | +| Vulkan | ru-long (33.8s) | 2.08 s (16.26×) | 2.08 s (16.24×) | +| CPU | ru-short (11.0s) | 1.30 s (8.47×) | 1.40 s (7.87×) | +| CPU | ru-long (33.8s) | 4.65 s (7.28×) | 4.85 s (6.98×) | -Fedora Linux 43, transcribe.cpp `ef55b52`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `522ccd68` on 2026-09-15. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models gigaam-v3-ctc,gigaam-v3-rnnt,gigaam-v3-e2e-ctc,gigaam-v3-e2e-rnnt \ - --quants q8_0,q4_k_m \ - --samples ru \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name gigaam-publication +uv run scripts/bench/run.py --profile --models gigaam-v3-e2e-rnnt ``` ## Numerical Validation diff --git a/docs/models/gigaam-v3-rnnt.md b/docs/models/gigaam-v3-rnnt.md index 62a72b72..5d0a45fe 100644 --- a/docs/models/gigaam-v3-rnnt.md +++ b/docs/models/gigaam-v3-rnnt.md @@ -1,7 +1,10 @@ # GigaAM-v3 RNN-T (charwise) -ai-sage's [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) -(rnnt branch) ported to transcribe.cpp. Same 16-layer Conformer encoder as `gigaam-v3-e2e-rnnt`, paired with an RNN-T transducer head fine-tuned on lowercased no-punctuation text. Charwise tokenizer (33 entries + blank) keeps the head tiny and the output normalized. + +Upstream: [`ai-sage/GigaAM-v3`](https://huggingface.co/ai-sage/GigaAM-v3) at [`c7f128b`](https://huggingface.co/ai-sage/GigaAM-v3/commit/c7f128b). + +Offline Russian speech-to-text with greedy RNN-T decoding. Same 16-layer Conformer encoder as the e2e variant, fine-tuned to emit lowercased Russian with no punctuation; 33-entry character vocabulary. + ## What it's for @@ -25,23 +28,35 @@ across heads). Variants in this family: See ai-sage's [model card](https://huggingface.co/ai-sage/GigaAM-v3) for training data, intended use, and upstream evaluation methodology. -Licensed MIT. Ported from upstream commit -[`c7f128b`](https://huggingface.co/ai-sage/GigaAM-v3/commit/c7f128b8accdd9624df905e5c2d7b7a48c27c0d8), -pinned 2026-05-12. + +Licensed MIT. Ported from upstream commit [`c7f128b`](https://huggingface.co/ai-sage/GigaAM-v3/commit/c7f128b), pinned 2026-05-12. Validated against the gigaam author package reference at transcribe.cpp commit [`42b96d9`](https://github.com/handy-computer/transcribe.cpp/tree/42b96d9) on 2026-05-12. + ## Download -| Quantization | Download | Size | WER (FLEURS ru) | + +| Quantization | Download | Size | WER (FLEURS ru) | | --- | --- | ---: | ---: | -| F32 | [gigaam-v3-rnnt-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-F32.gguf) | 846 MB | 8.08% | -| F16 | [gigaam-v3-rnnt-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-F16.gguf) | 430 MB | 8.08% | -| Q8_0 | [gigaam-v3-rnnt-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q8_0.gguf) | 260 MB | 8.08% | -| Q6_K | [gigaam-v3-rnnt-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q6_K.gguf) | 217 MB | 8.07% | -| Q5_K_M | [gigaam-v3-rnnt-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q5_K_M.gguf) | 196 MB | 8.12% | -| Q4_K_M | [gigaam-v3-rnnt-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q4_K_M.gguf) | 175 MB | 8.12% | - -WER is measured on the full FLEURS ru test split (775 utterances) with -greedy decoding and no external LM. F32 reference baseline: **8.08%**. +| F32 | [gigaam-v3-rnnt-F32.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-F32.gguf) | 888 MB | 8.08% | +| F16 | [gigaam-v3-rnnt-F16.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-F16.gguf) | 451 MB | 8.08% | +| Q8_0 | [gigaam-v3-rnnt-Q8_0.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q8_0.gguf) | 273 MB | 8.07% | +| Q6_K | [gigaam-v3-rnnt-Q6_K.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q6_K.gguf) | 227 MB | 8.07% | +| Q5_K_M | [gigaam-v3-rnnt-Q5_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q5_K_M.gguf) | 206 MB | 8.12% | +| Q4_K_M | [gigaam-v3-rnnt-Q4_K_M.gguf](https://huggingface.co/handy-computer/gigaam-v3-rnnt-gguf/resolve/main/gigaam-v3-rnnt-Q4_K_M.gguf) | 183 MB | 8.12% | + + + +WER on the full FLEURS ru split (775 utterances), batch sizes 1 and 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding, no external LM. F32 reference baseline: 8.08%. Upstream `gigaam` +author package measured on the same manifest: 9.46%; the 1.4 pp gap is upstream +rejecting 5 long (>25 s) utterances with `Too long wav file, use +'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt +subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not +publish a FLEURS ru WER; this number is measured here. + Upstream (`gigaam` author package at `6e4b027c`) measured on the same manifest: **9.46%**. The 1.4 pp gap is the upstream package @@ -73,39 +88,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | ------------: | ------------: | -| Metal | ru (4.5s) | 41 ms (110×) | 43 ms (105×) | -| CPU | ru (4.5s) | 167 ms (27×) | 166 ms (27×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `ef55b52`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Metal | ru-short (11.0s) | 28 ms (385.49×) | 30 ms (367.06×) | +| Metal | ru-long (33.8s) | 70 ms (483.39×) | 70 ms (484.17×) | +| CPU | ru-short (11.0s) | 376 ms (29.22×) | 371 ms (29.57×) | +| CPU | ru-long (33.8s) | 1.18 s (28.58×) | 1.28 s (26.46×) | + +Apple M4 Max: transcribe.cpp `94f1f45` on 2026-09-15. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ---------- | -----------: | -----------: | -| Vulkan | ru (4.5s) | 179 ms (25×) | 184 ms (25×) | -| CPU | ru (4.5s) | 511 ms (9×) | 420 ms (11×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ---------------- | --------------: | --------------: | +| Vulkan | ru-short (11.0s) | 428 ms (25.67×) | 437 ms (25.11×) | +| Vulkan | ru-long (33.8s) | 1.22 s (27.81×) | 1.24 s (27.34×) | +| CPU | ru-short (11.0s) | 930 ms (11.81×) | 1.04 s (10.55×) | +| CPU | ru-long (33.8s) | 3.81 s (8.87×) | 4.07 s (8.32×) | -Fedora Linux 43, transcribe.cpp `ef55b52`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `522ccd68` on 2026-09-15. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models gigaam-v3-ctc,gigaam-v3-rnnt,gigaam-v3-e2e-ctc,gigaam-v3-e2e-rnnt \ - --quants q8_0,q4_k_m \ - --samples ru \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name gigaam-publication +uv run scripts/bench/run.py --profile --models gigaam-v3-rnnt ``` ## Numerical Validation diff --git a/docs/models/gigaam.md b/docs/models/gigaam.md index 1a1ae077..1a9cdbdf 100644 --- a/docs/models/gigaam.md +++ b/docs/models/gigaam.md @@ -36,12 +36,14 @@ WER is on FLEURS Russian (`fleurs-ru`) for the **Q8_0** preset, measured by transcribe.cpp's WER pipeline. See each per-variant doc for the full quant matrix. -| Variant | Decoder | Output | Params | Q8_0 size | WER (Q8_0) | Doc | -| --- | --- | --- | ---: | ---: | ---: | --- | -| `gigaam-v3-e2e-rnnt` | RNN-T | cased + punctuated | ~180M | 261 MB | 5.36% | [gigaam-v3-e2e-rnnt.md](gigaam-v3-e2e-rnnt.md) | -| `gigaam-v3-e2e-ctc` | CTC | cased + punctuated | ~180M | 260 MB | 5.50% | [gigaam-v3-e2e-ctc.md](gigaam-v3-e2e-ctc.md) | -| `gigaam-v3-rnnt` | RNN-T | lowercased, no-punctuation | ~180M | 260 MB | 8.08% | [gigaam-v3-rnnt.md](gigaam-v3-rnnt.md) | -| `gigaam-v3-ctc` | CTC | lowercased, no-punctuation | ~180M | 259 MB | 8.40% | [gigaam-v3-ctc.md](gigaam-v3-ctc.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `gigaam-v3-e2e-rnnt` | 223M | ru | 274 MB | FLEURS ru (WER) | 5.35% | token timestamps | [gigaam-v3-e2e-rnnt.md](gigaam-v3-e2e-rnnt.md) | +| `gigaam-v3-e2e-ctc` | 221M | ru | 272 MB | FLEURS ru (WER) | 5.53% | token timestamps | [gigaam-v3-e2e-ctc.md](gigaam-v3-e2e-ctc.md) | +| `gigaam-v3-rnnt` | 222M | ru | 273 MB | FLEURS ru (WER) | 8.07% | token timestamps | [gigaam-v3-rnnt.md](gigaam-v3-rnnt.md) | +| `gigaam-v3-ctc` | 221M | ru | 272 MB | FLEURS ru (WER) | 8.42% | token timestamps | [gigaam-v3-ctc.md](gigaam-v3-ctc.md) | + Pre-built GGUFs for every variant and quant are hosted under [`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); diff --git a/docs/models/granite-4.0-1b-speech.md b/docs/models/granite-4.0-1b-speech.md index 7f415dbc..a4f267b7 100644 --- a/docs/models/granite-4.0-1b-speech.md +++ b/docs/models/granite-4.0-1b-speech.md @@ -1,9 +1,17 @@ # Granite Speech 4.0-1b -IBM's [`ibm-granite/granite-4.0-1b-speech`](https://huggingface.co/ibm-granite/granite-4.0-1b-speech) -ported to transcribe.cpp. An audio-LLM: a Conformer encoder with block-local -Shaw attention, a BLIP-2 Q-Former projector, and the Granite-4.0-1b-base LLM -as an autoregressive decoder. + +Upstream: [`ibm-granite/granite-4.0-1b-speech`](https://huggingface.co/ibm-granite/granite-4.0-1b-speech) at [`bd87ab8`](https://huggingface.co/ibm-granite/granite-4.0-1b-speech/commit/bd87ab8). + +Offline multilingual speech-to-text. IBM Granite Speech 4.0-1b is an +audio-LLM: a Conformer encoder with block-local Shaw attention, a BLIP-2 +Q-Former projector, and the Granite-4.0-1b-base LLM as an autoregressive +decoder. Takes a 16 kHz mono WAV and produces a transcript; the LLM half is +what writes the text. Transcribes English, French, German, Spanish, +Portuguese, and Japanese. Translates between English and each of those +five other languages in either direction (en ↔ fr, en ↔ de, en ↔ es, +en ↔ pt, en ↔ ja) — always via English, no direct fr↔de etc. + ## What it's for @@ -20,27 +28,46 @@ etc. Pass the target language as a BCP-47 code via `--translate See IBM's [model card](https://huggingface.co/ibm-granite/granite-4.0-1b-speech) for training data, intended use, and upstream evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`bd87ab8`](https://huggingface.co/ibm-granite/granite-4.0-1b-speech/commit/bd87ab862416353633ea431fe49b1614003623c5), -pinned 2026-05-17. + +Licensed Apache-2.0. Ported from upstream commit [`bd87ab8`](https://huggingface.co/ibm-granite/granite-4.0-1b-speech/commit/bd87ab8), pinned 2026-05-17. Validated against the Transformers reference at transcribe.cpp commit [`275332d`](https://github.com/handy-computer/transcribe.cpp/tree/275332d) on 2026-05-17. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [granite-4.0-1b-speech-BF16.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-BF16.gguf) | 4.63 GB | 1.42% | -| F16 | [granite-4.0-1b-speech-F16.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-F16.gguf) | 4.63 GB | 1.42% | -| Q8_0 | [granite-4.0-1b-speech-Q8_0.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q8_0.gguf) | 2.56 GB | 1.44% | -| Q6_K | [granite-4.0-1b-speech-Q6_K.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q6_K.gguf) | 2.02 GB | 1.41% | -| Q5_K_M | [granite-4.0-1b-speech-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q5_K_M.gguf) | 1.83 GB | 1.42% | -| Q4_K_M | [granite-4.0-1b-speech-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q4_K_M.gguf) | 1.60 GB | 1.48% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with -greedy decoding. The BF16 reference baseline (transformers, re-run locally -with the model-card prompt `USER: <|audio|>can you transcribe the speech -into a written format?\n ASSISTANT:`) is 1.42%, matching IBM's published -Open ASR Leaderboard number exactly. Text normalizer: Whisper -`EnglishTextNormalizer`, the same normalizer Open ASR Leaderboard uses. +| BF16 | [granite-4.0-1b-speech-BF16.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-BF16.gguf) | 4.63 GB | 1.42% | +| F16 | [granite-4.0-1b-speech-F16.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-F16.gguf) | 4.63 GB | 1.42% | +| Q8_0 | [granite-4.0-1b-speech-Q8_0.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q8_0.gguf) | 2.56 GB | 1.44% | +| Q6_K | [granite-4.0-1b-speech-Q6_K.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q6_K.gguf) | 2.02 GB | 1.41% | +| Q5_K_M | [granite-4.0-1b-speech-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q5_K_M.gguf) | 1.83 GB | 1.42% | +| Q4_K_M | [granite-4.0-1b-speech-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-4.0-1b-speech-gguf/resolve/main/granite-4.0-1b-speech-Q4_K_M.gguf) | 1.60 GB | 1.48% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding. BF16 reference baseline (re-run locally with the model card's exact +prompt): 1.42% — matches the upstream Open ASR Leaderboard number exactly. Text +normalizer: Whisper `EnglishTextNormalizer`, the same normalizer Open ASR +Leaderboard uses. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 7.29% | +| en | WER | 4.66% | +| es | WER | 5.91% | +| fr | WER | 8.44% | +| ja | CER | 6.44% | +| pt | WER | 9.60% | + ## Quick Start @@ -70,58 +97,35 @@ build/bin/transcribe-cli \ ## Performance -Cells are wall-clock latency, with speedup over realtime in parentheses. - ### Apple M4 Max -Mean over 3 iterations after 1 warmup. - -**Metal** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 254 ms (43×) | 284 ms (39×) | -| dots (35.3s) | 928 ms (38×) | 1.05 s (34×) | - -**CPU** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 1.48 s (7.4×) | 1.80 s (6.1×) | -| dots (35.3s) | 5.15 s (6.9×) | 5.96 s (5.9×) | - -macOS 26.4, transcribe.cpp `de05c43`. + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -### Apple M4 +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 136 ms (80.65×) | 169 ms (65.11×) | +| Metal | dots (35.3s) | 502 ms (70.32×) | 472 ms (74.81×) | +| CPU | jfk (11.0s) | 1.32 s (8.31×) | 1.39 s (7.94×) | +| CPU | dots (35.3s) | 4.20 s (8.41×) | 4.23 s (8.36×) | -Mean over 5 iterations after 2 warmups. Q8_0. - -| Backend | Sample | Q8_0 | -| ------- | ----------- | ----------------: | -| Metal | jfk (11.0s) | 959 ms (11×) | -| CPU | jfk (11.0s) | 2.44 s (5×) | - -macOS 26.1, transcribe.cpp `275332d`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U (Vega 8 iGPU) -Mean over 3 iterations after 1 warmup. - -**Vulkan (RADV)** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 3.47 s (3.2×) | 3.71 s (3.0×) | -| dots (35.3s) | 11.37 s (3.1×) | 12.30 s (2.9×) | - -**CPU** + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 5.21 s (2.1×) | 6.72 s (1.6×) | -| dots (35.3s) | 18.08 s (1.9×) | 24.21 s (1.5×) | +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 2.54 s (4.33×) | 2.54 s (4.32×) | +| Vulkan | dots (35.3s) | 7.15 s (4.94×) | 6.84 s (5.16×) | +| CPU | jfk (11.0s) | 3.96 s (2.78×) | 3.90 s (2.82×) | +| CPU | dots (35.3s) | 12.93 s (2.73×) | 12.72 s (2.78×) | -Linux 6.18 (Fedora 43), transcribe.cpp `dbe5814`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + ## Capabilities diff --git a/docs/models/granite-speech-4.1-2b-nar.md b/docs/models/granite-speech-4.1-2b-nar.md index 4201f8f7..3d529aba 100644 --- a/docs/models/granite-speech-4.1-2b-nar.md +++ b/docs/models/granite-speech-4.1-2b-nar.md @@ -1,12 +1,18 @@ # Granite Speech 4.1-2b NAR -IBM's [`ibm-granite/granite-speech-4.1-2b-nar`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar) -ported to transcribe.cpp. The non-autoregressive editor variant of -Granite-Speech. Shares the Conformer audio encoder with the AR Granite- -Speech family but pairs it with a custom MLP-with-attention projector and -the Granite-4.0-1b LLM used as a bidirectional editor (causal mask -disabled). One forward pass produces logits over the full transcript; -CTC decode yields the final text — no token-by-token loop. + +Upstream: [`ibm-granite/granite-speech-4.1-2b-nar`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar) at [`99a4df9`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar/commit/99a4df9). + +Offline multilingual speech-to-text in a single non-autoregressive editor +pass. IBM Granite Speech 4.1-2b NAR shares the Conformer audio encoder +with the AR Granite-Speech family but pairs it with a custom MLP-with- +attention projector and the Granite-4.0-1b LLM used as a bidirectional +editor (causal mask disabled). One forward pass produces logits over the +full transcript; CTC decode yields the final text. No token-by-token loop. +Takes a 16 kHz mono WAV and produces a transcript. English plus French, +German, Spanish, and Portuguese; ASR only (no translation, no +timestamps). + ## What it's for @@ -17,33 +23,53 @@ only — no translation, no timestamps, no diarization. See IBM's [model card](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar) for training data, intended use, and upstream evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`99a4df9`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar/commit/99a4df9007ac5682f9daa093fb7008ff606e9a5d), -pinned 2026-05-24 (single-file `modeling_granite_speech_nar.py` snapshot — -the README's canonical inference target). + +Licensed Apache-2.0. Ported from upstream commit [`99a4df9`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-nar/commit/99a4df9), pinned 2026-05-24. Validated against the Transformers reference at transcribe.cpp commit [`c53af2c`](https://github.com/handy-computer/transcribe.cpp/tree/c53af2c) on 2026-05-24. + + +The pinned revision is the single-file `modeling_granite_speech_nar.py` +snapshot, the README's canonical inference target. ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [granite-speech-4.1-2b-nar-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-BF16.gguf) | 4.20 GB | 1.29% | -| F16 | [granite-speech-4.1-2b-nar-F16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-F16.gguf) | 4.21 GB | 1.29% | -| Q8_0 | [granite-speech-4.1-2b-nar-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q8_0.gguf) | 2.33 GB | 1.29% | -| Q6_K | [granite-speech-4.1-2b-nar-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q6_K.gguf) | 1.84 GB | 1.29% | -| Q5_K_M | [granite-speech-4.1-2b-nar-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q5_K_M.gguf) | 1.66 GB | 1.25% | -| Q4_K_M | [granite-speech-4.1-2b-nar-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q4_K_M.gguf) | 1.45 GB | 1.35% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances). -BF16 reference baseline (transformers `model.transcribe`, MPS, re-run -locally): 1.28% — matches the upstream model card's 1.29% to within -sampling noise. Text normalizer: Whisper `EnglishTextNormalizer`. F16, -Q8_0, and Q6_K all score the same 1.29% as BF16 — the editor is very -robust to weight quantization down through Q5_K_M, where the WER -actually dips slightly (1.25%, within overlapping 95% CI of REF). -Reference reproduction follows the model card path verbatim -(`AutoProcessor` + `AutoModel.transcribe` + `processor.batch_decode`) -at HF revision `99a4df9`; the older snapshot's bidirectional-mask patch -is obsolete in this snapshot. +| BF16 | [granite-speech-4.1-2b-nar-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-BF16.gguf) | 4.51 GB | 1.29% | +| F16 | [granite-speech-4.1-2b-nar-F16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-F16.gguf) | 4.52 GB | 1.29% | +| Q8_0 | [granite-speech-4.1-2b-nar-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q8_0.gguf) | 2.50 GB | 1.29% | +| Q6_K | [granite-speech-4.1-2b-nar-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q6_K.gguf) | 1.98 GB | 1.29% | +| Q5_K_M | [granite-speech-4.1-2b-nar-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q5_K_M.gguf) | 1.78 GB | 1.28% | +| Q4_K_M | [granite-speech-4.1-2b-nar-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-nar-gguf/resolve/main/granite-speech-4.1-2b-nar-Q4_K_M.gguf) | 1.56 GB | 1.34% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +BF16 reference baseline (transformers `model.transcribe`, MPS, re-run locally): +1.28% — matches the upstream model card's 1.29% to within sampling noise. Text +normalizer: Whisper `EnglishTextNormalizer`, the same normalizer Open ASR +Leaderboard uses. Reference reproduction follows the model card path verbatim +(`AutoProcessor` + `AutoModel.transcribe` + `processor.batch_decode`) at HF revision +`99a4df9` (single-file `modeling_granite_speech_nar.py` snapshot, the README's +canonical target); no mask patching is required because the NAR LM uses +`create_bidirectional_mask()` natively. F16, Q8_0, and Q6_K all match BF16's 1.29%; +Q5_K_M dips slightly to 1.25% (within overlapping CIs). + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 6.07% | +| en | WER | 5.33% | +| es | WER | 4.08% | +| fr | WER | 6.76% | +| pt | WER | 5.57% | + ## Quick Start @@ -68,64 +94,35 @@ editor handles language detection implicitly. ## Performance -Cells are wall-clock latency, with speedup over realtime in parentheses. -NAR is faster than the AR variants on GPU backends because there is no -autoregressive step loop — a single bidirectional forward through 40 LLM -layers replaces the per-token decode graph. - ### Apple M4 Max -Mean over 3 iterations after 1 warmup. - -**Metal** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 209 ms (53×) | 196 ms (56×) | -| dots (35.3s) | 664 ms (53×) | 635 ms (56×) | - -**CPU** + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 1.87 s (5.9×) | 1.99 s (5.5×) | -| dots (35.3s) | 6.50 s (5.4×) | 7.71 s (4.6×) | +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 149 ms (73.82×) | 153 ms (71.92×) | +| Metal | dots (35.3s) | 466 ms (75.73×) | 466 ms (75.88×) | +| CPU | jfk (11.0s) | 1.59 s (6.92×) | 1.67 s (6.60×) | +| CPU | dots (35.3s) | 5.27 s (6.70×) | 5.75 s (6.15×) | -macOS 26.4, transcribe.cpp `de05c43`. - -### Apple M4 - -Mean over 5 iterations after 2 warmups. Q8_0. - -| Backend | Sample | Q8_0 | -| ------- | ----------- | ----------------: | -| Metal | jfk (11.0s) | 614 ms (18×) | -| CPU | jfk (11.0s) | 2.55 s (4×) | - -macOS 26.1, transcribe.cpp `275332d`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U (Vega 8 iGPU) -Mean over 3 iterations after 1 warmup. - -**Vulkan (RADV)** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 3.16 s (3.5×) | 3.06 s (3.6×) | -| dots (35.3s) | 9.85 s (3.6×) | 9.57 s (3.7×) | - -**CPU** + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 5.71 s (1.9×) | 7.05 s (1.6×) | -| dots (35.3s) | 20.39 s (1.7×) | 24.81 s (1.4×) | +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 2.68 s (4.10×) | 2.77 s (3.96×) | +| Vulkan | dots (35.3s) | 9.09 s (3.89×) | 9.02 s (3.92×) | +| CPU | jfk (11.0s) | 4.76 s (2.31×) | 4.98 s (2.21×) | +| CPU | dots (35.3s) | 17.88 s (1.98×) | 17.84 s (1.98×) | -Linux 6.18 (Fedora 43), transcribe.cpp `dbe5814`. NAR's Vulkan RTF stays -flat across short and long samples (jfk and dots both ~3.6×) because the -single bidirectional LLM pass dominates over the encoder; on CPU the -encoder dominates so RTF tapers slightly with sequence length. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + ## Capabilities diff --git a/docs/models/granite-speech-4.1-2b-plus.md b/docs/models/granite-speech-4.1-2b-plus.md index a3cc164a..0e51855d 100644 --- a/docs/models/granite-speech-4.1-2b-plus.md +++ b/docs/models/granite-speech-4.1-2b-plus.md @@ -1,12 +1,24 @@ # Granite Speech 4.1-2b-plus -IBM's [`ibm-granite/granite-speech-4.1-2b-plus`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus) -ported to transcribe.cpp. The timestamp-and-diarization variant of the + +Upstream: [`ibm-granite/granite-speech-4.1-2b-plus`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus) at [`edd3bf5`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus/commit/edd3bf5). + +Offline multilingual speech-to-text with word-level timestamps. IBM Granite +Speech 4.1-2b-plus is the timestamp-and-diarization variant of the Granite-Speech family. Same architecture as the base 4.1-2b (Conformer encoder, BLIP-2 Q-Former projector, Granite-4.0-1b autoregressive LLM decoder) with two changes: the encoder concatenates mid-layer (idx 3) and -final-layer hidden states (doubling the projector K/V input from 1024 to -2048), and the LM token embeddings are tied with the lm_head. +final-layer hidden states (`cat_hidden_layers=[3]`, doubling the projector +K/V input from 1024 to 2048), and the LM token embeddings are tied with +the lm_head. Takes a 16 kHz mono WAV and produces a transcript, with +`--timestamps word` returning structured per-word timestamps (parsed from the +model's `[T:N]` centisecond markers), or `--diarize` returning structured +speaker-attributed turns from its separate SAA prompt. Those two prompt tasks +cannot be combined. Transcribes English, +French, German, Spanish, and Portuguese (no Japanese on this variant). +This variant is transcription-only: unlike the base granite-speech-4.1-2b, +it does not perform speech translation. + ## What it's for @@ -24,30 +36,50 @@ speech translation. See IBM's [model card](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus) for training data, intended use, and upstream evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`edd3bf5`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus/commit/edd3bf54fbb06d8e263aa0c1939321d67b073f86), -pinned 2026-05-17. + +Licensed Apache-2.0. Ported from upstream commit [`edd3bf5`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus/commit/edd3bf5), pinned 2026-05-17. Validated against the Transformers reference at transcribe.cpp commit [`275332d`](https://github.com/handy-computer/transcribe.cpp/tree/275332d) on 2026-05-17. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [granite-speech-4.1-2b-plus-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-BF16.gguf) | 4.23 GB | 1.49% | -| F16 | [granite-speech-4.1-2b-plus-F16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-F16.gguf) | 4.23 GB | 1.48% | -| Q8_0 | [granite-speech-4.1-2b-plus-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q8_0.gguf) | 2.35 GB | 1.50% | -| Q6_K | [granite-speech-4.1-2b-plus-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q6_K.gguf) | 1.86 GB | 1.46% | -| Q5_K_M | [granite-speech-4.1-2b-plus-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q5_K_M.gguf) | 1.69 GB | 1.48% | -| Q4_K_M | [granite-speech-4.1-2b-plus-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q4_K_M.gguf) | 1.49 GB | 1.56% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with -greedy decoding and the model-card chat template (system prompt + leading- -space user instruction + `add_generation_prompt=True`). BF16 reference -baseline (transformers, re-run locally with that exact prompt): 1.48%; -0.04pp above upstream's published 1.44%, within bootstrap CI overlap and -likely a chat-template / normalization difference on the publisher side. -Text normalizer: Whisper `EnglishTextNormalizer`. The transcribe.cpp runtime -hard-codes the correct chat template; the WER quoted here is what the C++ -runtime actually scores. +| BF16 | [granite-speech-4.1-2b-plus-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-BF16.gguf) | 4.23 GB | 1.49% | +| F16 | [granite-speech-4.1-2b-plus-F16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-F16.gguf) | 4.23 GB | 1.48% | +| Q8_0 | [granite-speech-4.1-2b-plus-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q8_0.gguf) | 2.35 GB | 1.50% | +| Q6_K | [granite-speech-4.1-2b-plus-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q6_K.gguf) | 1.86 GB | 1.46% | +| Q5_K_M | [granite-speech-4.1-2b-plus-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q5_K_M.gguf) | 1.69 GB | 1.48% | +| Q4_K_M | [granite-speech-4.1-2b-plus-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-plus-gguf/resolve/main/granite-speech-4.1-2b-plus-Q4_K_M.gguf) | 1.49 GB | 1.56% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding with the model-card chat template (system prompt + leading-space +user instruction + `add_generation_prompt=True`). BF16 reference baseline (re-run +locally with that exact prompt): 1.48%; 0.04pp above upstream's published 1.44%, +within bootstrap CI overlap and likely a chat-template / normalization difference on +the publisher side. Text normalizer: Whisper `EnglishTextNormalizer`, the same +normalizer Open ASR Leaderboard uses. The `add_generation_prompt=True` is +load-bearing — without it the model emits 25-27 empty hypotheses on short test-clean +clips and WER blows up to ~26%. The transcribe.cpp runtime hard-codes the prompt +correctly; this note only matters if you reproduce the reference. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 8.06% | +| en | WER | 4.46% | +| es | WER | 6.53% | +| fr | WER | 8.82% | +| pt | WER | 10.61% | + ## Quick Start @@ -93,58 +125,35 @@ words: 22 ## Performance -Cells are wall-clock latency, with speedup over realtime in parentheses. - ### Apple M4 Max -Mean over 3 iterations after 1 warmup. - -**Metal** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 280 ms (39×) | 308 ms (36×) | -| dots (35.3s) | 1.02 s (34×) | 1.18 s (30×) | - -**CPU** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 1.87 s (5.9×) | 2.04 s (5.4×) | -| dots (35.3s) | 5.71 s (6.2×) | 6.91 s (5.1×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4, transcribe.cpp `de05c43`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | --------------: | +| Metal | jfk (11.0s) | 134 ms (82.33×) | 137 ms (80.36×) | +| Metal | dots (35.3s) | 350 ms (100.92×) | 353 ms (99.98×) | +| CPU | jfk (11.0s) | 1.47 s (7.46×) | 1.50 s (7.33×) | +| CPU | dots (35.3s) | 3.85 s (9.18×) | 4.14 s (8.54×) | -### Apple M4 - -Mean over 5 iterations after 2 warmups. Q8_0. - -| Backend | Sample | Q8_0 | -| ------- | ----------- | ----------------: | -| Metal | jfk (11.0s) | 1.00 s (11×) | -| CPU | jfk (11.0s) | 2.44 s (5×) | - -macOS 26.1, transcribe.cpp `275332d`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U (Vega 8 iGPU) -Mean over 3 iterations after 1 warmup. - -**Vulkan (RADV)** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 3.63 s (3.0×) | 3.85 s (2.9×) | -| dots (35.3s) | 12.29 s (2.9×) | 13.42 s (2.6×) | - -**CPU** + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 6.08 s (1.8×) | 7.80 s (1.4×) | -| dots (35.3s) | 20.45 s (1.7×) | 26.23 s (1.3×) | +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 2.58 s (4.27×) | 2.58 s (4.26×) | +| Vulkan | dots (35.3s) | 7.04 s (5.02×) | 7.53 s (4.69×) | +| CPU | jfk (11.0s) | 4.32 s (2.55×) | 4.34 s (2.53×) | +| CPU | dots (35.3s) | 13.35 s (2.65×) | 13.10 s (2.70×) | -Linux 6.18 (Fedora 43), transcribe.cpp `dbe5814`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + ## Capabilities diff --git a/docs/models/granite-speech-4.1-2b.md b/docs/models/granite-speech-4.1-2b.md index d0fd3b9c..556b6269 100644 --- a/docs/models/granite-speech-4.1-2b.md +++ b/docs/models/granite-speech-4.1-2b.md @@ -1,10 +1,18 @@ # Granite Speech 4.1-2b -IBM's [`ibm-granite/granite-speech-4.1-2b`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b) -ported to transcribe.cpp. An audio-LLM with the same architecture as -4.0-1b (Conformer encoder with block-local Shaw attention, BLIP-2 Q-Former -projector, Granite-4.0-1b-base autoregressive LLM decoder) and improved -punctuation/casing over 4.0-1b. + +Upstream: [`ibm-granite/granite-speech-4.1-2b`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b) at [`8f4bb5f`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b/commit/8f4bb5f). + +Offline multilingual speech-to-text. IBM Granite Speech 4.1-2b is an +audio-LLM with the same architecture as 4.0-1b (Conformer encoder with +block-local Shaw attention, BLIP-2 Q-Former projector, Granite-4.0-1b-base +autoregressive LLM decoder) and improved punctuation and casing over 4.0-1b. +Takes a 16 kHz mono WAV and produces a transcript. Transcribes English, +French, German, Spanish, Portuguese, and Japanese. Translates between +English and each of those five other languages in either direction +(en ↔ fr, en ↔ de, en ↔ es, en ↔ pt, en ↔ ja) — always via English, no +direct fr↔de etc. + ## What it's for @@ -21,27 +29,47 @@ etc. Pass the target language as a BCP-47 code via `--translate See IBM's [model card](https://huggingface.co/ibm-granite/granite-speech-4.1-2b) for training data, intended use, and upstream evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`8f4bb5f`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b/commit/8f4bb5f31ae98971bd218169f00065a041d20058), -pinned 2026-05-17. + +Licensed Apache-2.0. Ported from upstream commit [`8f4bb5f`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b/commit/8f4bb5f), pinned 2026-05-17. Validated against the Transformers reference at transcribe.cpp commit [`275332d`](https://github.com/handy-computer/transcribe.cpp/tree/275332d) on 2026-05-17. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [granite-speech-4.1-2b-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-BF16.gguf) | 4.63 GB | 1.31% | -| F16 | [granite-speech-4.1-2b-F16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-F16.gguf) | 4.63 GB | 1.32% | -| Q8_0 | [granite-speech-4.1-2b-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q8_0.gguf) | 2.56 GB | 1.32% | -| Q6_K | [granite-speech-4.1-2b-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q6_K.gguf) | 2.02 GB | 1.29% | -| Q5_K_M | [granite-speech-4.1-2b-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q5_K_M.gguf) | 1.83 GB | 1.33% | -| Q4_K_M | [granite-speech-4.1-2b-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q4_K_M.gguf) | 1.60 GB | 1.37% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with -greedy decoding and the model-card prompt `transcribe the speech with proper -punctuation and capitalization.`. BF16 reference baseline (transformers, -re-run locally with that prompt): 1.31% — 0.02pp below upstream's published -1.33%, within bootstrap CI overlap. Text normalizer: Whisper -`EnglishTextNormalizer`, the same normalizer Open ASR Leaderboard uses. +| BF16 | [granite-speech-4.1-2b-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-BF16.gguf) | 4.63 GB | 1.31% | +| F16 | [granite-speech-4.1-2b-F16.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-F16.gguf) | 4.63 GB | 1.32% | +| Q8_0 | [granite-speech-4.1-2b-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q8_0.gguf) | 2.56 GB | 1.32% | +| Q6_K | [granite-speech-4.1-2b-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q6_K.gguf) | 2.02 GB | 1.29% | +| Q5_K_M | [granite-speech-4.1-2b-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q5_K_M.gguf) | 1.83 GB | 1.33% | +| Q4_K_M | [granite-speech-4.1-2b-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-4.1-2b-gguf/resolve/main/granite-speech-4.1-2b-Q4_K_M.gguf) | 1.60 GB | 1.37% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding. BF16 reference baseline (re-run locally with the model card's exact +prompt): 1.31% — 0.02pp below upstream's published 1.33%, likely a minor +normalization difference on the publisher side and well within bootstrap CI overlap. +Text normalizer: Whisper `EnglishTextNormalizer`, the same normalizer Open ASR +Leaderboard uses. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 6.25% | +| en | WER | 4.14% | +| es | WER | 5.48% | +| fr | WER | 7.61% | +| ja | CER | 6.30% | +| pt | WER | 9.80% | + ## Quick Start @@ -71,58 +99,35 @@ build/bin/transcribe-cli \ ## Performance -Cells are wall-clock latency, with speedup over realtime in parentheses. - ### Apple M4 Max -Mean over 3 iterations after 1 warmup. - -**Metal** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 272 ms (40×) | 303 ms (36×) | -| dots (35.3s) | 1.00 s (35×) | 1.16 s (30×) | - -**CPU** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 1.67 s (6.6×) | 1.85 s (5.9×) | -| dots (35.3s) | 5.49 s (6.4×) | 6.22 s (5.7×) | - -macOS 26.4, transcribe.cpp `de05c43`. + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -### Apple M4 +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 145 ms (75.92×) | 139 ms (79.11×) | +| Metal | dots (35.3s) | 445 ms (79.32×) | 458 ms (77.22×) | +| CPU | jfk (11.0s) | 1.33 s (8.30×) | 1.37 s (8.05×) | +| CPU | dots (35.3s) | 4.13 s (8.56×) | 4.26 s (8.30×) | -Mean over 5 iterations after 2 warmups. Q8_0. - -| Backend | Sample | Q8_0 | -| ------- | ----------- | ----------------: | -| Metal | jfk (11.0s) | 954 ms (12×) | -| CPU | jfk (11.0s) | 2.45 s (4×) | - -macOS 26.1, transcribe.cpp `275332d`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U (Vega 8 iGPU) -Mean over 3 iterations after 1 warmup. - -**Vulkan (RADV)** - -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 3.58 s (3.1×) | 3.84 s (2.9×) | -| dots (35.3s) | 11.79 s (3.0×) | 13.08 s (2.7×) | - -**CPU** + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -| Sample | Q4_K_M | Q8_0 | -| ------------ | ---------------: | ---------------: | -| jfk (11.0s) | 5.58 s (2.0×) | 7.19 s (1.5×) | -| dots (35.3s) | 19.49 s (1.8×) | 25.50 s (1.4×) | +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 2.56 s (4.29×) | 2.55 s (4.31×) | +| Vulkan | dots (35.3s) | 7.15 s (4.94×) | 6.80 s (5.20×) | +| CPU | jfk (11.0s) | 3.86 s (2.85×) | 3.89 s (2.83×) | +| CPU | dots (35.3s) | 13.06 s (2.70×) | 12.59 s (2.81×) | -Linux 6.18 (Fedora 43), transcribe.cpp `dbe5814`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + ## Capabilities diff --git a/docs/models/granite-speech-5.0-470m-turboctc-nc.md b/docs/models/granite-speech-5.0-470m-turboctc-nc.md index 6cc66eb9..fefe8be9 100644 --- a/docs/models/granite-speech-5.0-470m-turboctc-nc.md +++ b/docs/models/granite-speech-5.0-470m-turboctc-nc.md @@ -1,14 +1,12 @@ # Granite Speech 5.0 470M TurboCTC NC -IBM's [`ibm-granite/granite-speech-5.0-470m-turboctc-nc`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc-nc) -ported to transcribe.cpp. A 470M-parameter Granite Conformer encoder with a -self-conditioned CTC head. + +Upstream: [`ibm-granite/granite-speech-5.0-470m-turboctc-nc`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc-nc) at [`0eb7b4f`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc-nc/commit/0eb7b4f). -> **Non-commercial.** Licensed -> [CC-BY-NC-SA-4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/), not -> Apache-2.0. Research and non-commercial use only, and ShareAlike means these -> GGUFs carry the same terms. For commercial use take the Apache-2.0 sibling, -> [`granite-speech-5.0-470m-turboctc`](granite-speech-5.0-470m-turboctc.md). +Offline English speech-to-text, research and non-commercial use only. A Granite Conformer +encoder with a self-conditioned CTC head. Takes a 16 kHz mono WAV and produces a +transcript. Not a streaming model. English only. + ## What it's for @@ -19,27 +17,39 @@ Same architecture as the Apache-2.0 sibling, trained on more data (~75,000 h vs ~60,000 h). IBM reports 4.85% aggregate WER across the 8 Open ASR leaderboard test sets for this model, against 5.00% for the sibling. -Licensed CC-BY-NC-SA-4.0. Ported from upstream commit -[`0eb7b4f`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc-nc/commit/0eb7b4fe726a294815dc45d342860465b5af68ef), -pinned 2026-09-12. + +Licensed CC-BY-NC-SA-4.0. Ported from upstream commit [`0eb7b4f`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc-nc/commit/0eb7b4f), pinned 2026-09-12. Validated against the transformers reference at transcribe.cpp commit [`f1d0e10`](https://github.com/handy-computer/transcribe.cpp/tree/f1d0e10) on 2026-09-12. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [granite-speech-5.0-470m-turboctc-nc-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-BF16.gguf) | 948 MB | 1.29% | -| F16 | [granite-speech-5.0-470m-turboctc-nc-F16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-F16.gguf) | 949 MB | 1.29% | -| Q8_0 | [granite-speech-5.0-470m-turboctc-nc-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q8_0.gguf) | 506 MB | 1.30% | -| Q6_K | [granite-speech-5.0-470m-turboctc-nc-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q6_K.gguf) | 392 MB | 1.28% | -| Q5_K_M | [granite-speech-5.0-470m-turboctc-nc-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q5_K_M.gguf) | 336 MB | 1.29% | -| Q4_K_M | [granite-speech-5.0-470m-turboctc-nc-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q4_K_M.gguf) | 279 MB | 1.33% | - -Measured on the full LibriSpeech test-clean split (2620 utterances), greedy CTC -decoding, no external LM. Reference baseline (transformers 5.17.0, F32, CPU): -**1.29%**, 95% CI [1.15%, 1.42%]. Every tier falls inside that CI, so the -ordering between them is not meaningful. Q4_K_M is the weakest at 1.33%; if you -want a small file without that, use Q5_K_M. Clean read speech only, not checked -on noisy or accented audio. +| BF16 | [granite-speech-5.0-470m-turboctc-nc-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-BF16.gguf) | 948 MB | 1.29% | +| F16 | [granite-speech-5.0-470m-turboctc-nc-F16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-F16.gguf) | 949 MB | 1.28% | +| Q8_0 | [granite-speech-5.0-470m-turboctc-nc-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q8_0.gguf) | 506 MB | 1.29% | +| Q6_K | [granite-speech-5.0-470m-turboctc-nc-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q6_K.gguf) | 392 MB | 1.29% | +| Q5_K_M | [granite-speech-5.0-470m-turboctc-nc-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q5_K_M.gguf) | 336 MB | 1.29% | +| Q4_K_M | [granite-speech-5.0-470m-turboctc-nc-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-nc-Q4_K_M.gguf) | 279 MB | 1.34% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 8, timestamps none, language hint `en`, decoded on cuda. Measured at transcribe.cpp `9daf396`. + + + +Greedy CTC decoding, no external LM. Measured reference baseline (transformers +5.17.0, F32, CPU): 1.29%, 95% CI [1.15, 1.42]. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.30% | + ## Quick Start @@ -51,41 +61,35 @@ build/bin/transcribe-cli \ ## Performance -Wall-clock latency (mean over 3 iterations after 1 warmup), with speedup over -realtime in parentheses. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 37.0 ms (297×) | 38.1 ms (289×) | -| Metal | dots (35.3s) | 85.3 ms (414×) | 87.2 ms (405×) | -| CPU | jfk (11.0s) | 233 ms (47×) | 233 ms (47×) | -| CPU | dots (35.3s) | 696 ms (51×) | 687 ms (52×) | - -macOS 26.6.2, transcribe.cpp `144ccad`. + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -### AMD Ryzen 7 PRO 4750U +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 35 ms (312.54×) | 37 ms (295.13×) | +| Metal | dots (35.3s) | 83 ms (424.15×) | 87 ms (407.09×) | +| CPU | jfk (11.0s) | 244 ms (45.18×) | 255 ms (43.17×) | +| CPU | dots (35.3s) | 746 ms (47.39×) | 764 ms (46.26×) | -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 631 ms (17×) | 647 ms (17×) | -| Vulkan | dots (35.3s) | 1.49 s (24×) | 1.52 s (23×) | -| CPU | jfk (11.0s) | 702 ms (16×) | 654 ms (17×) | -| CPU | dots (35.3s) | 2.27 s (16×) | 2.26 s (16×) | +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + -Fedora 43, transcribe.cpp `3a5ed01`. Vulkan device: `AMD Radeon Graphics (RADV RENOIR)`. +### AMD Ryzen 7 PRO 4750U -### Apple M4 + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 100 ms (110×) | 103 ms (107×) | -| Metal | dots (35.3s) | 271 ms (131×) | 278 ms (127×) | -| CPU | jfk (11.0s) | 383 ms (29×) | 414 ms (27×) | -| CPU | dots (35.3s) | 1.17 s (30×) | 1.26 s (28×) | +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 583 ms (18.88×) | 605 ms (18.18×) | +| Vulkan | dots (35.3s) | 1.32 s (26.77×) | 1.34 s (26.45×) | +| CPU | jfk (11.0s) | 614 ms (17.91×) | 653 ms (16.84×) | +| CPU | dots (35.3s) | 2.13 s (16.59×) | 2.12 s (16.63×) | -macOS 26.5.1, transcribe.cpp `54b241e`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Q8_0 is usually a little faster than Q4_K_M despite being 1.8× the size, so pick Q4_K_M for footprint rather than speed. Cost is linear in audio length, so @@ -97,8 +101,7 @@ identical output, but buys almost nothing here. transcribe.cpp is validated tensor-by-tensor against Hugging Face transformers on `samples/jfk.wav` and `samples/dots.wav`. All 35 checkpointed tensors fall within variant tolerance, and the final transcript matches the reference -verbatim. Last validated at commit -[`f1d0e10`](https://github.com/handy-computer/transcribe.cpp/tree/f1d0e10). +verbatim. | Field | Value | | --- | --- | @@ -108,17 +111,6 @@ verbatim. Last validated at commit | Tolerances | `tests/tolerances/granite5_ctc-nc.json` | | Command | `uv run scripts/validate.py all --family granite5_ctc --variant granite-speech-5.0-470m-turboctc-nc` | -Selected tensors (the `dots` case, 441 output frames): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `mel.in` | `8.821e-06` | `1.007e-07` | Production C++ frontend, no injection | -| `enc.input_linear.out` | `1.851e-02` | `1.080e-03` | 0.18% of RMS | -| `enc.block.7.out` | `8.640e-03` | `9.496e-04` | Before the self-conditioning injection | -| `enc.ctc.mid_logits` | `4.356e-02` | `3.568e-03` | Mid-layer CTC head | -| `enc.block.15.out` | `2.187e-01` | `1.625e-03` | Max is one frame of 441; mean is 0.25% of RMS | -| `enc.ctc_logits` | `7.307e-01` | `6.497e-03` | 0.09% of RMS | - Drift is BF16 matmul accumulation, amplified by a massive-activation channel in encoder blocks 10-13 that sits ~20x above the rest of its tensor. It is concentrated on single frames rather than spread out, and it does not reach the diff --git a/docs/models/granite-speech-5.0-470m-turboctc.md b/docs/models/granite-speech-5.0-470m-turboctc.md index a880a102..4157f12d 100644 --- a/docs/models/granite-speech-5.0-470m-turboctc.md +++ b/docs/models/granite-speech-5.0-470m-turboctc.md @@ -1,17 +1,21 @@ # Granite Speech 5.0 470M TurboCTC -IBM's [`ibm-granite/granite-speech-5.0-470m-turboctc`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc) -ported to transcribe.cpp. A 470M-parameter Granite Conformer encoder with a -self-conditioned CTC head. + +Upstream: [`ibm-granite/granite-speech-5.0-470m-turboctc`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc) at [`18ca3c1`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc/commit/18ca3c1). + +Offline English speech-to-text. A Granite Conformer encoder with a self-conditioned CTC +head. Takes a 16 kHz mono WAV and produces a transcript. Not a streaming model. English +only. + ## What it's for Offline English speech-to-text. Takes a 16 kHz mono WAV and produces a transcript. Not a streaming model. English only, and it does not translate. -Licensed Apache-2.0. Ported from upstream commit -[`18ca3c1`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc/commit/18ca3c1de6cd092b5a30c39fb0f04550b38ed1a0), -pinned 2026-09-12. + +Licensed Apache-2.0. Ported from upstream commit [`18ca3c1`](https://huggingface.co/ibm-granite/granite-speech-5.0-470m-turboctc/commit/18ca3c1), pinned 2026-09-12. Validated against the transformers reference at transcribe.cpp commit [`b9427cf`](https://github.com/handy-computer/transcribe.cpp/tree/b9427cf) on 2026-09-12. + There is also a non-commercial sibling, [`granite-speech-5.0-470m-turboctc-nc`](granite-speech-5.0-470m-turboctc-nc.md), @@ -20,20 +24,33 @@ this one for anything commercial. ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [granite-speech-5.0-470m-turboctc-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-BF16.gguf) | 948 MB | 1.33% | -| F16 | [granite-speech-5.0-470m-turboctc-F16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-F16.gguf) | 948 MB | 1.33% | -| Q8_0 | [granite-speech-5.0-470m-turboctc-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q8_0.gguf) | 506 MB | 1.34% | -| Q6_K | [granite-speech-5.0-470m-turboctc-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q6_K.gguf) | 392 MB | 1.33% | -| Q5_K_M | [granite-speech-5.0-470m-turboctc-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q5_K_M.gguf) | 336 MB | 1.34% | -| Q4_K_M | [granite-speech-5.0-470m-turboctc-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q4_K_M.gguf) | 279 MB | 1.34% | - -Measured on the full LibriSpeech test-clean split (2620 utterances), greedy CTC -decoding, no external LM. Reference baseline (transformers 5.17.0, F32, CPU): -**1.33%**, 95% CI [1.20%, 1.47%]. Every tier falls inside that CI, so the -ordering between them is not meaningful and quantizing down to Q4_K_M costs -very little. Clean read speech only, not checked on noisy or accented audio. +| BF16 | [granite-speech-5.0-470m-turboctc-BF16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-BF16.gguf) | 948 MB | 1.34% | +| F16 | [granite-speech-5.0-470m-turboctc-F16.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-F16.gguf) | 948 MB | 1.33% | +| Q8_0 | [granite-speech-5.0-470m-turboctc-Q8_0.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q8_0.gguf) | 506 MB | 1.33% | +| Q6_K | [granite-speech-5.0-470m-turboctc-Q6_K.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q6_K.gguf) | 392 MB | 1.33% | +| Q5_K_M | [granite-speech-5.0-470m-turboctc-Q5_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q5_K_M.gguf) | 336 MB | 1.34% | +| Q4_K_M | [granite-speech-5.0-470m-turboctc-Q4_K_M.gguf](https://huggingface.co/handy-computer/granite-speech-5.0-470m-turboctc-gguf/resolve/main/granite-speech-5.0-470m-turboctc-Q4_K_M.gguf) | 279 MB | 1.35% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 8, timestamps none, language hint `en`, decoded on cuda. Measured at transcribe.cpp `9daf396`. + + + +Greedy CTC decoding, no external LM. Measured reference baseline (transformers +5.17.0, F32, CPU): 1.33%, 95% CI [1.20, 1.47]. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.61% | + ## Quick Start @@ -45,43 +62,35 @@ build/bin/transcribe-cli \ ## Performance -Wall-clock latency (mean over 3 iterations after 1 warmup), with speedup over -realtime in parentheses. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 37.5 ms (293×) | 38.9 ms (283×) | -| Metal | dots (35.3s) | 85.6 ms (413×) | 87.9 ms (402×) | -| CPU | jfk (11.0s) | 232 ms (48×) | 233 ms (47×) | -| CPU | dots (35.3s) | 703 ms (50×) | 689 ms (51×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.6.2, transcribe.cpp `54b241e`. - -### AMD Ryzen 7 PRO 4750U +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 36 ms (301.30×) | 39 ms (284.51×) | +| Metal | dots (35.3s) | 86 ms (410.20×) | 95 ms (371.09×) | +| CPU | jfk (11.0s) | 243 ms (45.19×) | 254 ms (43.29×) | +| CPU | dots (35.3s) | 752 ms (46.95×) | 769 ms (45.93×) | -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 652 ms (17×) | 647 ms (17×) | -| Vulkan | dots (35.3s) | 1.48 s (24×) | 1.51 s (23×) | -| CPU | jfk (11.0s) | 696 ms (16×) | 663 ms (17×) | -| CPU | dots (35.3s) | 2.28 s (16×) | 2.25 s (16×) | +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + -Fedora 43, transcribe.cpp `3a5ed01`. Vulkan device: `AMD Radeon Graphics (RADV RENOIR)`. +### AMD Ryzen 7 PRO 4750U -### Apple M4 + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ----------------- | ------------: | ------------: | -| Metal | jfk (11.0s) | 100 ms (110×) | 103 ms (107×) | -| Metal | dots (35.3s) | 272 ms (130×) | 279 ms (127×) | -| Metal | dots-full (305.9s)| 2.33 s (131×) | 2.38 s (128×) | -| CPU | jfk (11.0s) | 388 ms (28×) | 423 ms (26×) | -| CPU | dots (35.3s) | 1.19 s (30×) | 1.27 s (28×) | -| CPU | dots-full (305.9s)| 8.30 s (37×) | 9.73 s (31×) | +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 588 ms (18.69×) | 588 ms (18.72×) | +| Vulkan | dots (35.3s) | 1.31 s (27.04×) | 1.31 s (26.89×) | +| CPU | jfk (11.0s) | 615 ms (17.90×) | 652 ms (16.87×) | +| CPU | dots (35.3s) | 2.13 s (16.58×) | 2.14 s (16.51×) | -macOS 26.5.1, transcribe.cpp `f2d5e31`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Q8_0 is usually a little faster than Q4_K_M despite being 1.8× the size, so pick Q4_K_M for footprint rather than speed. Cost is linear in audio length: @@ -94,8 +103,7 @@ supported and gives identical output, but buys almost nothing here. transcribe.cpp is validated tensor-by-tensor against Hugging Face transformers on `samples/jfk.wav` and `samples/dots.wav`. All 35 checkpointed tensors fall within family tolerance, and the final transcript matches the reference -verbatim. Last validated at commit -[`b9427cf`](https://github.com/handy-computer/transcribe.cpp/tree/b9427cf). +verbatim. | Field | Value | | --- | --- | @@ -105,17 +113,6 @@ verbatim. Last validated at commit | Tolerances | `tests/tolerances/granite5_ctc.json` | | Command | `uv run scripts/validate.py all --family granite5_ctc --variant granite-speech-5.0-470m-turboctc` | -Selected tensors (the `dots` case, 441 output frames): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `mel.in` | `8.821e-06` | `1.007e-07` | Production C++ frontend, no injection | -| `enc.input_linear.out` | `1.372e-02` | `1.097e-03` | 0.18% of RMS | -| `enc.block.7.out` | `1.316e-02` | `1.011e-03` | Before the self-conditioning injection | -| `enc.ctc.mid_logits` | `6.251e-02` | `3.951e-03` | Mid-layer CTC head | -| `enc.block.15.out` | `2.190e-02` | `1.370e-03` | Final encoder output | -| `enc.ctc_logits` | `1.513e-01` | `5.442e-03` | 0.06% of RMS | - Drift is BF16 matmul accumulation compounding over 16 blocks, and stays small relative to tensor magnitude. It does not reach the output: zero argmax differences against the reference on either sample, and 2619 of 2620 diff --git a/docs/models/granite-speech-5.0-turboctc.md b/docs/models/granite-speech-5.0-turboctc.md index 010cd741..46c190fa 100644 --- a/docs/models/granite-speech-5.0-turboctc.md +++ b/docs/models/granite-speech-5.0-turboctc.md @@ -28,10 +28,12 @@ Apache one unless you specifically want the NC weights. ## All variants -| Variant | Licence | WER (test-clean) | Card | -| --- | --- | ---: | --- | -| `granite-speech-5.0-470m-turboctc` | Apache-2.0 | 1.33% | [card](granite-speech-5.0-470m-turboctc.md) | -| `granite-speech-5.0-470m-turboctc-nc` | CC-BY-NC-SA-4.0 | 1.29% | [card](granite-speech-5.0-470m-turboctc-nc.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `granite-speech-5.0-470m-turboctc` | 473M | en | 506 MB | LibriSpeech test-clean (WER) | 1.33% | - | [granite-speech-5.0-470m-turboctc.md](granite-speech-5.0-470m-turboctc.md) | +| `granite-speech-5.0-470m-turboctc-nc` | 473M | en | 506 MB | LibriSpeech test-clean (WER) | 1.29% | - | [granite-speech-5.0-470m-turboctc-nc.md](granite-speech-5.0-470m-turboctc-nc.md) | + Both ship BF16, F16, Q8_0, Q6_K, Q5_K_M and Q4_K_M, from 948 MB down to 279 MB. Download links are on the per-variant cards. diff --git a/docs/models/granite-speech.md b/docs/models/granite-speech.md index 24dcf9e8..b24f328f 100644 --- a/docs/models/granite-speech.md +++ b/docs/models/granite-speech.md @@ -42,12 +42,14 @@ WER is on LibriSpeech test-clean for the **Q8_0** preset, measured by transcribe.cpp's WER pipeline. See each per-variant doc for the full quant matrix. -| Variant | Decode mode | Params | Q8_0 size | WER (Q8_0) | Languages | Extras | Doc | -| --- | --- | ---: | ---: | ---: | --- | --- | --- | -| `granite-4.0-1b-speech` | AR (audio-LLM) | ~3B† | 2.56 GB | 1.44% | en, fr, de, es, pt, ja | translate (en ↔ ASR langs; en → it/zh) | [granite-4.0-1b-speech.md](granite-4.0-1b-speech.md) | -| `granite-speech-4.1-2b` | AR (audio-LLM) | ~3B† | 2.56 GB | 1.32% | en, fr, de, es, pt, ja | translate (en ↔ ASR langs; en → it/zh) | [granite-speech-4.1-2b.md](granite-speech-4.1-2b.md) | -| `granite-speech-4.1-2b-plus` | AR (audio-LLM) | ~3B† | 2.35 GB | 1.50% | en, fr, de, es, pt | word timestamps (ASR only) | [granite-speech-4.1-2b-plus.md](granite-speech-4.1-2b-plus.md) | -| `granite-speech-4.1-2b-nar` | NAR (editor) | ~3B† | 2.33 GB | 1.29% | en, fr, de, es, pt | (ASR only) | [granite-speech-4.1-2b-nar.md](granite-speech-4.1-2b-nar.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `granite-4.0-1b-speech` | 2.3B | 6 languages | 2.56 GB | LibriSpeech test-clean (WER) | 1.44% | translate | [granite-4.0-1b-speech.md](granite-4.0-1b-speech.md) | +| `granite-speech-4.1-2b` | 2.3B | 6 languages | 2.56 GB | LibriSpeech test-clean (WER) | 1.32% | translate | [granite-speech-4.1-2b.md](granite-speech-4.1-2b.md) | +| `granite-speech-4.1-2b-plus` | 2.1B | 5 languages | 2.35 GB | LibriSpeech test-clean (WER) | 1.50% | diarize, word timestamps | [granite-speech-4.1-2b-plus.md](granite-speech-4.1-2b-plus.md) | +| `granite-speech-4.1-2b-nar` | 2.3B | 5 languages | 2.50 GB | LibriSpeech test-clean (WER) | 1.29% | - | [granite-speech-4.1-2b-nar.md](granite-speech-4.1-2b-nar.md) | + † Parameter counts include the Conformer audio encoder, the projector, and the Granite-4.0-1b text LM. The "1b" / "2b" in IBM's variant names diff --git a/docs/models/medasr.md b/docs/models/medasr.md index f28569d0..cc7fb0cf 100644 --- a/docs/models/medasr.md +++ b/docs/models/medasr.md @@ -1,6 +1,10 @@ # MedASR -Google's [`google/medasr`](https://huggingface.co/google/medasr) ported to transcribe.cpp. 105M-parameter encoder-CTC for medical-dictation English ASR. 17-layer Conformer encoder with RoPE attention (rope_theta=10000), macaron FFNs (residual scalars [1.5, 0.5]), BatchNorm conv module (kernel=32, residual scalars [2.0, 1.0]), and a Linear 512→512 CTC head over a SentencePiece BPE vocabulary. + +Upstream: [`google/medasr`](https://huggingface.co/google/medasr) at [`ae1e484`](https://huggingface.co/google/medasr/commit/ae1e484). + +Offline English speech-to-text optimized for medical dictation (radiology, internal medicine, family medicine). 17-layer Conformer encoder with RoPE attention, macaron FFNs, and a 512-token SentencePiece CTC head. Greedy CTC decode; no language model, no beam search. + ## What it's for @@ -8,9 +12,11 @@ Offline English speech-to-text optimized for medical dictation (radiology, inter Trained on ~5,000 hours of de-identified physician dictations on top of a LibriHeavy 50k-hour pretrain. The upstream model card flags lower accuracy on non-native accents and a male-skewed speaker distribution. -Licensed under the [Health AI Developer Foundations terms](https://developers.google.com/health-ai-developer-foundations/terms). The upstream repo is gated; you must accept the HF terms before download. +The upstream repo is gated; you must accept the HF terms before download. -Ported from upstream commit [`ae1e484`](https://huggingface.co/google/medasr/commit/ae1e4845b4b07479735d93e1e591e566435b7104), pinned 2026-06-04. + +Licensed [Health AI Developer Foundations](https://developers.google.com/health-ai-developer-foundations/terms). Ported from upstream commit [`ae1e484`](https://huggingface.co/google/medasr/commit/ae1e484), pinned 2026-06-04. Validated against the transformers @ 65dc2615 (dev; v5.0.0 unreleased) reference at transcribe.cpp commit [`782abfd`](https://github.com/handy-computer/transcribe.cpp/tree/782abfd) on 2026-06-04. + ## Input limits @@ -22,18 +28,40 @@ long recordings for best results. See the ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [medasr-F32.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-F32.gguf) | 417 MB | 17.88% | -| F16 | [medasr-F16.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-F16.gguf) | 202 MB | 17.88% | -| Q8_0 | [medasr-Q8_0.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q8_0.gguf) | 122 MB | 17.86% | -| Q6_K | [medasr-Q6_K.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q6_K.gguf) | 101 MB | 17.93% | -| Q5_K_M | [medasr-Q5_K_M.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q5_K_M.gguf) | 90 MB | 17.91% | -| Q4_K_M | [medasr-Q4_K_M.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q4_K_M.gguf) | 79 MB | 18.14% | - -**Recommended default: Q8_0.** Smallest preset with no statistically detectable WER degradation versus F32 (122 MB; +0.00 pp within bootstrap CI). Q4_K_M shows a real +0.26 pp degradation on LibriSpeech and is shipped for completeness but **not recommended** — prefer Q5_K_M if you need smaller than Q8_0. - -WER measured on the full LibriSpeech test-clean split (2,620 utterances) with greedy CTC decoding and no external LM. F32 reference baseline (HuggingFace transformers, Mac MPS): **17.88%**; transcribe.cpp F32 matches exactly. Absolute WER is higher than general-purpose ASR (e.g. Whisper-base ≈ 5%) because the model is fine-tuned for medical dictation — on the publisher's internal RAD-DICT / GENERAL-DICT / FM-DICT datasets the model scores 6.6%–9.3%, but those datasets are not publicly reproducible. See [`reports/wer/medasr.test-clean.summary.md`](../../reports/wer/medasr.test-clean.summary.md) for the full sweep. +| F32 | [medasr-F32.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-F32.gguf) | 421 MB | 17.88% | +| F16 | [medasr-F16.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-F16.gguf) | 211 MB | 17.88% | +| Q8_0 | [medasr-Q8_0.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q8_0.gguf) | 128 MB | 17.86% | +| Q6_K | [medasr-Q6_K.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q6_K.gguf) | 106 MB | 17.93% | +| Q5_K_M | [medasr-Q5_K_M.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q5_K_M.gguf) | 94 MB | 17.91% | +| Q4_K_M | [medasr-Q4_K_M.gguf](https://huggingface.co/handy-computer/medasr-gguf/resolve/main/medasr-Q4_K_M.gguf) | 83 MB | 18.14% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy CTC decoding, no external LM. F32 reference baseline (HuggingFace +transformers, Mac MPS): 17.88%; transcribe.cpp F32 matches exactly. Absolute WER is +higher than general-purpose ASR (e.g. Whisper-base ~5%) because the model is +fine-tuned for medical dictation — on the publisher's internal RAD-DICT / +GENERAL-DICT / FM-DICT datasets the model scores 6.6%–9.3%, but those datasets are +not publicly reproducible. Q8_0 is the recommended default (smallest preset with no +statistically detectable WER degradation); Q4_K_M shows a real +0.26 pp degradation +and is shipped for completeness but not recommended — prefer Q5_K_M if you need +smaller than Q8_0. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 37.48% | + ## Quick Start @@ -55,36 +83,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------- | ------------: | ------------: | -| Metal | jfk (11.0 s) | 38 ms (290×) | 44 ms (248×) | -| Metal | dots (35.3 s) | 84 ms (419×) | 90 ms (394×) | -| CPU | jfk (11.0 s) | 161 ms (68×) | 180 ms (61×) | -| CPU | dots (35.3 s) | 558 ms (63×) | 623 ms (57×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 17 ms (658.45×) | 16 ms (667.76×) | +| Metal | dots (35.3s) | 39 ms (915.05×) | 40 ms (893.98×) | +| CPU | jfk (11.0s) | 162 ms (68.09×) | 179 ms (61.33×) | +| CPU | dots (35.3s) | 564 ms (62.60×) | 632 ms (55.88×) | -macOS 26.5, transcribe.cpp `8139a4b`. Metal device: `Apple M4 Max`. Mel pipeline uses the shared `MelFrontend` (Accelerate vDSP fp64 FFT + cblas_sgemm); encoder is the conformer + RoPE + BatchNorm-conv graph in `src/arch/medasr/encoder.cpp`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------- | ------------: | ------------: | -| Vulkan | jfk (11.0 s) | 163 ms (68×) | 174 ms (63×) | -| Vulkan | dots (35.3 s) | 481 ms (74×) | 495 ms (71×) | -| CPU | jfk (11.0 s) | 543 ms (20×) | 488 ms (23×) | -| CPU | dots (35.3 s) | 1.84 s (19×) | 1.63 s (22×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 160 ms (68.93×) | 157 ms (70.04×) | +| Vulkan | dots (35.3s) | 480 ms (73.59×) | 484 ms (72.95×) | +| CPU | jfk (11.0s) | 400 ms (27.48×) | 466 ms (23.58×) | +| CPU | dots (35.3s) | 1.46 s (24.25×) | 1.50 s (23.58×) | -Fedora 43, transcribe.cpp `79d139a`. Vulkan device: `AMD Radeon Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models medasr --quants q8_0,q4_k_m --samples jfk,dots \ - --backends metal,cpu,vulkan --iters 3 --warmup 1 --name medasr-publication +uv run scripts/bench/run.py --profile --models medasr ``` ## Numerical Validation diff --git a/docs/models/moonshine-base.md b/docs/models/moonshine-base.md index 71a5629f..eb4ba383 100644 --- a/docs/models/moonshine-base.md +++ b/docs/models/moonshine-base.md @@ -1,10 +1,15 @@ # Moonshine base -Useful Sensors' [`UsefulSensors/moonshine-base`](https://huggingface.co/UsefulSensors/moonshine-base) -ported to transcribe.cpp. A 61M-parameter encoder-decoder transformer that -consumes raw 16 kHz PCM directly (no STFT, no mel filterbank) via a three-layer -Conv1d stem. Wider and deeper than moonshine-tiny (8 encoder / 8 decoder -layers, hidden size 416, intermediate 1664, partial RoPE 0.62). + +Upstream: [`UsefulSensors/moonshine-base`](https://huggingface.co/UsefulSensors/moonshine-base) at [`7a73d8d`](https://huggingface.co/UsefulSensors/moonshine-base/commit/7a73d8d). + +Useful Sensors Moonshine base — an encoder-decoder transformer for English +speech recognition. Consumes raw 16 kHz PCM directly via a three-layer Conv1d +stem (no STFT, no mel) and emits transcript-only output. Wider and deeper +than moonshine-tiny (8 encoder / 8 decoder layers, hidden size 416, partial +RoPE 0.62). English-only; no translation, no language detection, no +timestamps. + ## What it's for @@ -18,32 +23,40 @@ timestamps. See the [upstream model card](https://huggingface.co/UsefulSensors/moonshine-base) for training data, intended use, and the original evaluation methodology. -Licensed MIT. Ported from upstream commit -[`7a73d8d`](https://huggingface.co/UsefulSensors/moonshine-base/commit/7a73d8d55ac0ba2ef3ae761593f6784b51f96dcf), -pinned 2026-05-05. Validated against the transformers reference at -transcribe.cpp commit -[`07a8a84`](https://github.com/handy-computer/transcribe.cpp/tree/07a8a84) -on 2026-05-05. + +Licensed MIT. Ported from upstream commit [`7a73d8d`](https://huggingface.co/UsefulSensors/moonshine-base/commit/7a73d8d), pinned 2026-05-05. Validated against the transformers reference at transcribe.cpp commit [`07a8a84`](https://github.com/handy-computer/transcribe.cpp/tree/07a8a84) on 2026-05-05. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [moonshine-base-F32.gguf](https://huggingface.co/handy-computer/moonshine-base-gguf/resolve/main/moonshine-base-F32.gguf) | 236 MB | 3.28% | -| F16 | [moonshine-base-F16.gguf](https://huggingface.co/handy-computer/moonshine-base-gguf/resolve/main/moonshine-base-F16.gguf) | 126 MB | 3.28% | -| Q8_0 | [moonshine-base-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-base-gguf/resolve/main/moonshine-base-Q8_0.gguf) | 74 MB | 3.26% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with -the transcribe.cpp default decode (greedy, `num_beams=1`, `max_length=194` — -matching the upstream `generation_config`). Upstream reports 3.27% on the same -split (Moonshine paper, Table 2; also Open ASR Leaderboard). Our F32 reference -baseline lands at 3.28%, identical to upstream within rounding and well within -the ±1.00 pp Stage 7 acceptance gate. Q8_0 lands at 3.26%, slightly under F32 -— that delta sits inside the 95% bootstrap CI and is noise, not a real -improvement. Only F16 and Q8_0 are shipped as derived presets: at -moonshine-base's shapes (hidden 416, intermediate 1664, vocab 32768) none of -the dimensions divide the k-quant super-block size of 256, so Q6_K / Q5_K_M / -Q4_K_M would all fall back to Q8_0 storage and be near-duplicates. +| F32 | [moonshine-base-F32.gguf](https://huggingface.co/handy-computer/moonshine-base-gguf/resolve/main/moonshine-base-F32.gguf) | 248 MB | 3.28% | +| F16 | [moonshine-base-F16.gguf](https://huggingface.co/handy-computer/moonshine-base-gguf/resolve/main/moonshine-base-F16.gguf) | 132 MB | 3.28% | +| Q8_0 | [moonshine-base-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-base-gguf/resolve/main/moonshine-base-Q8_0.gguf) | 77 MB | 3.26% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=194, +matching the upstream generation_config). Upstream reports 3.27% on the same split +(Moonshine paper, Table 2; also Open ASR Leaderboard). Our F32 reference baseline +lands at 3.28%, identical to upstream within rounding and well within the ±1.00 pp +Stage 7 acceptance gate. Q8_0 lands at 3.26%, slightly under F32 — that delta sits +inside the 95% bootstrap CI and is noise, not a real improvement. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 12.25% | + ## Quick Start @@ -64,43 +77,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 5 iterations after 2 warmups), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above -(2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | -| ------- | ------------ | -----------: | -| Metal | jfk (11.0s) | 96 ms (115×) | -| Metal | dots (35.3s) | 811 ms (44×) | -| CPU | jfk (11.0s) | 98 ms (112×) | -| CPU | dots (35.3s) | 721 ms (49×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | +| ------- | ------------ | ---------------: | +| Metal | jfk (11.0s) | 89 ms (123.33×) | +| Metal | dots (35.3s) | 740 ms (47.76×) | +| CPU | jfk (11.0s) | 100 ms (109.83×) | +| CPU | dots (35.3s) | 690 ms (51.17×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | -| ------- | ------------ | ------------: | -| Vulkan | jfk (11.0s) | 218 ms (50×) | -| Vulkan | dots (35.3s) | 1.85 s (19×) | -| CPU | jfk (11.0s) | 331 ms (33×) | -| CPU | dots (35.3s) | 3.17 s (11×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Vulkan | jfk (11.0s) | 216 ms (50.90×) | +| Vulkan | dots (35.3s) | 1.74 s (20.28×) | +| CPU | jfk (11.0s) | 306 ms (35.95×) | +| CPU | dots (35.3s) | 3.20 s (11.03×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models moonshine-base \ - --quants q8_0 \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 5 --warmup 2 \ - --name moonshine-publication +uv run scripts/bench/run.py --profile --models moonshine-base ``` ## Numerical Validation diff --git a/docs/models/moonshine-streaming-medium.md b/docs/models/moonshine-streaming-medium.md index 4bb12d4a..7021abc4 100644 --- a/docs/models/moonshine-streaming-medium.md +++ b/docs/models/moonshine-streaming-medium.md @@ -1,11 +1,15 @@ # Moonshine Streaming Medium -Useful Sensors' [`UsefulSensors/moonshine-streaming-medium`](https://huggingface.co/UsefulSensors/moonshine-streaming-medium) -ported to transcribe.cpp. A 245M-parameter encoder-decoder English ASR model -designed for streaming use (ergodic encoder + sliding-window attention, -50 Hz time-domain frontend). Same family as the tiny and small variants; -deepest of the three (14 / 14 layers) and widest hidden dims (encoder 768 / -decoder 640). + +Upstream: [`UsefulSensors/moonshine-streaming-medium`](https://huggingface.co/UsefulSensors/moonshine-streaming-medium) at [`57b8436`](https://huggingface.co/UsefulSensors/moonshine-streaming-medium/commit/57b8436). + +English speech-to-text in both one-shot and streaming modes. An encoder-decoder +ASR model designed for streaming use (ergodic encoder + sliding-window +attention, 50 Hz time-domain frontend). Same family as moonshine-streaming-tiny +and moonshine-streaming-small; deepest of the three (14 / 14 layers) and widest +hidden dims (encoder 768 / decoder 640). Takes a 16 kHz mono WAV and produces a +transcript. No translation, no multilingual capability, no timestamps. + ## What it's for @@ -16,27 +20,41 @@ multilingual capability, and does not emit timestamps. See Useful Sensors' [model card](https://huggingface.co/UsefulSensors/moonshine-streaming-medium) for training data, intended use, and upstream evaluation methodology. -Licensed MIT. Ported from upstream commit -[`57b8436`](https://huggingface.co/UsefulSensors/moonshine-streaming-medium/commit/57b843633a8c183cadf6699ffa761377a933a866), -pinned 2026-05-06. + +Licensed MIT. Ported from upstream commit [`57b8436`](https://huggingface.co/UsefulSensors/moonshine-streaming-medium/commit/57b8436), pinned 2026-05-06. Validated against the HF Transformers v5.7.0 reference at transcribe.cpp commit [`0d312ce`](https://github.com/handy-computer/transcribe.cpp/tree/0d312ce) on 2026-05-06. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [moonshine-streaming-medium-F32.gguf](https://huggingface.co/handy-computer/moonshine-streaming-medium-gguf/resolve/main/moonshine-streaming-medium-F32.gguf) | 1015 MB | 2.16% | -| F16 | [moonshine-streaming-medium-F16.gguf](https://huggingface.co/handy-computer/moonshine-streaming-medium-gguf/resolve/main/moonshine-streaming-medium-F16.gguf) | 509 MB | 2.16% | -| Q8_0 | [moonshine-streaming-medium-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-streaming-medium-gguf/resolve/main/moonshine-streaming-medium-Q8_0.gguf) | 282 MB | 2.16% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference -baseline: 2.16%. Quants are numerically indistinguishable from F32 on this -manifest. Useful Sensors' self-reported number on this split is 2.08% from -the Open ASR Leaderboard table; the +0.08pp residual matches the same -scoring / text-normalization difference seen across the tiny and small -variants (where the tiny cross-check against the HF Transformers reference -on the same manifest landed within 0.01pp of our port), and is not a -numerical drift in the port. +| F32 | [moonshine-streaming-medium-F32.gguf](https://huggingface.co/handy-computer/moonshine-streaming-medium-gguf/resolve/main/moonshine-streaming-medium-F32.gguf) | 1.07 GB | 2.16% | +| F16 | [moonshine-streaming-medium-F16.gguf](https://huggingface.co/handy-computer/moonshine-streaming-medium-gguf/resolve/main/moonshine-streaming-medium-F16.gguf) | 534 MB | 2.16% | +| Q8_0 | [moonshine-streaming-medium-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-streaming-medium-gguf/resolve/main/moonshine-streaming-medium-Q8_0.gguf) | 296 MB | 2.16% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances). Figures without a commit were published before provenance was recorded. + + + +Greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference baseline: 2.16%. +Quants are numerically indistinguishable from F32 on this manifest. Useful Sensors' +self-reported number on this split is 2.08% from the Open ASR Leaderboard table; the ++0.08pp residual matches the same scoring / text-normalization difference seen +across the tiny and small variants (cross-checked against HF Transformers on tiny +and found to be at 99.6% identical hypotheses to our port), and is not a numerical +drift. Q6_K / Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 7.87% | + **One utterance the model cannot end.** A single LibriSpeech test-clean clip — `7176-92135-0020` (7.2 s; reference *"DOUBLE NINE TWO THREE ELSINORE DOUBLE NINE @@ -76,43 +94,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 5 iterations after 2 warmups), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above -(2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | -| ------- | ------------ | -----------: | -| Metal | jfk (11.0s) | 124 ms (89×) | -| Metal | dots (35.3s) | 952 ms (37×) | -| CPU | jfk (11.0s) | 281 ms (39×) | -| CPU | dots (35.3s) | 1.11 s (32×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `0d312ce`. +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Metal | jfk (11.0s) | 127 ms (86.61×) | +| Metal | dots (35.3s) | 978 ms (36.12×) | +| CPU | jfk (11.0s) | 235 ms (46.77×) | +| CPU | dots (35.3s) | 1.12 s (31.52×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | -| ------- | ------------ | ------------: | -| Vulkan | jfk (11.0s) | 570 ms (19×) | -| Vulkan | dots (35.3s) | 4.01 s (9×) | -| CPU | jfk (11.0s) | 1.07 s (10×) | -| CPU | dots (35.3s) | 6.50 s (5×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Vulkan | jfk (11.0s) | 560 ms (19.66×) | +| Vulkan | dots (35.3s) | 3.97 s (8.90×) | +| CPU | jfk (11.0s) | 817 ms (13.47×) | +| CPU | dots (35.3s) | 6.03 s (5.86×) | -Fedora 43, transcribe.cpp `f243f34`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models moonshine-streaming-medium \ - --quants q8_0 \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 5 --warmup 2 \ - --name moonshine-streaming-publication +uv run scripts/bench/run.py --profile --models moonshine-streaming-medium ``` ## Numerical Validation @@ -121,8 +136,7 @@ transcribe.cpp is validated tensor-by-tensor against the HF Transformers reference (`MoonshineStreamingForConditionalGeneration`, fp32 inference, `attn_implementation="eager"`) on `samples/jfk.wav`. All contract tensors fall within family tolerance, and the final transcript matches the -reference. Last validated at commit -[`0d312ce`](https://github.com/handy-computer/transcribe.cpp/tree/0d312ce). +reference. | Field | Value | | --- | --- | diff --git a/docs/models/moonshine-streaming-small.md b/docs/models/moonshine-streaming-small.md index 9368e237..35b69bea 100644 --- a/docs/models/moonshine-streaming-small.md +++ b/docs/models/moonshine-streaming-small.md @@ -1,11 +1,15 @@ # Moonshine Streaming Small -Useful Sensors' [`UsefulSensors/moonshine-streaming-small`](https://huggingface.co/UsefulSensors/moonshine-streaming-small) -ported to transcribe.cpp. A 123M-parameter encoder-decoder English ASR model -designed for streaming use (ergodic encoder + sliding-window attention, -50 Hz time-domain frontend). Same family as the tiny and medium variants; -deeper encoder/decoder (10 / 10 layers vs 6 / 6 for tiny) and wider hidden -dims (encoder 620 / decoder 512). + +Upstream: [`UsefulSensors/moonshine-streaming-small`](https://huggingface.co/UsefulSensors/moonshine-streaming-small) at [`2c03650`](https://huggingface.co/UsefulSensors/moonshine-streaming-small/commit/2c03650). + +English speech-to-text in both one-shot and streaming modes. An encoder-decoder +ASR model designed for streaming use (ergodic encoder + sliding-window +attention, 50 Hz time-domain frontend). Same family as +moonshine-streaming-tiny; deeper encoder/decoder (10 / 10 layers) and wider +hidden dims (encoder 620 / decoder 512). Takes a 16 kHz mono WAV and produces a +transcript. No translation, no multilingual capability, no timestamps. + ## What it's for @@ -16,26 +20,41 @@ multilingual capability, and does not emit timestamps. See Useful Sensors' [model card](https://huggingface.co/UsefulSensors/moonshine-streaming-small) for training data, intended use, and upstream evaluation methodology. -Licensed MIT. Ported from upstream commit -[`2c03650`](https://huggingface.co/UsefulSensors/moonshine-streaming-small/commit/2c036506f23a09c18df5a50057599ba6d9280999), -pinned 2026-05-06. + +Licensed MIT. Ported from upstream commit [`2c03650`](https://huggingface.co/UsefulSensors/moonshine-streaming-small/commit/2c03650), pinned 2026-05-06. Validated against the HF Transformers v5.7.0 reference at transcribe.cpp commit [`0d312ce`](https://github.com/handy-computer/transcribe.cpp/tree/0d312ce) on 2026-05-06. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [moonshine-streaming-small-F32.gguf](https://huggingface.co/handy-computer/moonshine-streaming-small-gguf/resolve/main/moonshine-streaming-small-F32.gguf) | 536 MB | 2.53% | -| F16 | [moonshine-streaming-small-F16.gguf](https://huggingface.co/handy-computer/moonshine-streaming-small-gguf/resolve/main/moonshine-streaming-small-F16.gguf) | 269 MB | 2.53% | -| Q8_0 | [moonshine-streaming-small-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-streaming-small-gguf/resolve/main/moonshine-streaming-small-Q8_0.gguf) | 189 MB | 2.54% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference -baseline: 2.53%. Useful Sensors' self-reported number on this split is -2.49% from the Open ASR Leaderboard table; the +0.04pp residual matches -the same scoring / text-normalization difference seen on the tiny variant -where we cross-checked against the HF Transformers reference (4.52% on the -same manifest, 99.6% identical hypotheses to our F32) and confirmed it is -not a numerical drift in the port. +| F32 | [moonshine-streaming-small-F32.gguf](https://huggingface.co/handy-computer/moonshine-streaming-small-gguf/resolve/main/moonshine-streaming-small-F32.gguf) | 562 MB | 2.53% | +| F16 | [moonshine-streaming-small-F16.gguf](https://huggingface.co/handy-computer/moonshine-streaming-small-gguf/resolve/main/moonshine-streaming-small-F16.gguf) | 282 MB | 2.53% | +| Q8_0 | [moonshine-streaming-small-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-streaming-small-gguf/resolve/main/moonshine-streaming-small-Q8_0.gguf) | 199 MB | 2.54% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances). Figures without a commit were published before provenance was recorded. + + + +Greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference baseline: 2.53%. +Useful Sensors' self-reported number on this split is 2.49% from the Open ASR +Leaderboard table; the +0.04pp residual matches the same scoring / +text-normalization difference seen on the tiny variant where we cross-checked +against the HF Transformers reference (4.52% on the same manifest, 99.6% identical +hypotheses to our F32) and confirmed it is not a numerical drift in the port. Q6_K / +Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 8.55% | + Q6_K / Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. @@ -58,43 +77,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 5 iterations after 2 warmups), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above -(2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | -| ------- | ------------ | -----------: | -| Metal | jfk (11.0s) | 82 ms (134×) | -| Metal | dots (35.3s) | 612 ms (58×) | -| CPU | jfk (11.0s) | 174 ms (63×) | -| CPU | dots (35.3s) | 699 ms (51×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `0d312ce`. +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Metal | jfk (11.0s) | 85 ms (129.36×) | +| Metal | dots (35.3s) | 603 ms (58.59×) | +| CPU | jfk (11.0s) | 172 ms (63.75×) | +| CPU | dots (35.3s) | 722 ms (48.91×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | -| ------- | ------------ | ------------: | -| Vulkan | jfk (11.0s) | 349 ms (32×) | -| Vulkan | dots (35.3s) | 2.38 s (15×) | -| CPU | jfk (11.0s) | 735 ms (15×) | -| CPU | dots (35.3s) | 4.00 s (9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Vulkan | jfk (11.0s) | 374 ms (29.38×) | +| Vulkan | dots (35.3s) | 2.50 s (14.16×) | +| CPU | jfk (11.0s) | 620 ms (17.75×) | +| CPU | dots (35.3s) | 3.79 s (9.32×) | -Fedora 43, transcribe.cpp `f243f34`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models moonshine-streaming-small \ - --quants q8_0 \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 5 --warmup 2 \ - --name moonshine-streaming-publication +uv run scripts/bench/run.py --profile --models moonshine-streaming-small ``` ## Numerical Validation @@ -103,8 +119,7 @@ transcribe.cpp is validated tensor-by-tensor against the HF Transformers reference (`MoonshineStreamingForConditionalGeneration`, fp32 inference, `attn_implementation="eager"`) on `samples/jfk.wav`. All contract tensors fall within family tolerance, and the final transcript matches the -reference. Last validated at commit -[`0d312ce`](https://github.com/handy-computer/transcribe.cpp/tree/0d312ce). +reference. | Field | Value | | --- | --- | diff --git a/docs/models/moonshine-streaming-tiny.md b/docs/models/moonshine-streaming-tiny.md index 05882c21..dfba8f1d 100644 --- a/docs/models/moonshine-streaming-tiny.md +++ b/docs/models/moonshine-streaming-tiny.md @@ -1,9 +1,14 @@ # Moonshine Streaming Tiny -Useful Sensors' [`UsefulSensors/moonshine-streaming-tiny`](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny) -ported to transcribe.cpp. A 34M-parameter encoder-decoder English ASR model -designed for streaming use (ergodic encoder + sliding-window attention, -50 Hz time-domain frontend). + +Upstream: [`UsefulSensors/moonshine-streaming-tiny`](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny) at [`f8e9dfd`](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny/commit/f8e9dfd). + +English speech-to-text in both one-shot and streaming modes. An +encoder-decoder ASR model designed for streaming use (ergodic encoder + +sliding-window attention, 50 Hz time-domain frontend). Takes a 16 kHz mono WAV +and produces a transcript. No translation, no multilingual capability, no +timestamps. + ## What it's for @@ -14,26 +19,41 @@ multilingual capability, and does not emit timestamps. See Useful Sensors' [model card](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny) for training data, intended use, and upstream evaluation methodology. -Licensed MIT. Ported from upstream commit -[`f8e9dfd`](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny/commit/f8e9dfd8c562c257c151a907b7b7f2fe8ff8511a), -pinned 2026-05-06. + +Licensed MIT. Ported from upstream commit [`f8e9dfd`](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny/commit/f8e9dfd), pinned 2026-05-06. Validated against the HF Transformers v5.7.0 reference at transcribe.cpp commit [`0d312ce`](https://github.com/handy-computer/transcribe.cpp/tree/0d312ce) on 2026-05-06. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [moonshine-streaming-tiny-F32.gguf](https://huggingface.co/handy-computer/moonshine-streaming-tiny-gguf/resolve/main/moonshine-streaming-tiny-F32.gguf) | 169 MB | 4.53% | -| F16 | [moonshine-streaming-tiny-F16.gguf](https://huggingface.co/handy-computer/moonshine-streaming-tiny-gguf/resolve/main/moonshine-streaming-tiny-F16.gguf) | 85 MB | 4.53% | -| Q8_0 | [moonshine-streaming-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-streaming-tiny-gguf/resolve/main/moonshine-streaming-tiny-Q8_0.gguf) | 48 MB | 4.52% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference -baseline: 4.53%. The HF Transformers reference scored on the same manifest -in the same regime lands at 4.52% with 99.6% byte-identical hypotheses to -our F32, so the port is at exact parity with the reference. Useful Sensors' -self-reported number on this split is 4.49% from the Open ASR Leaderboard -table; the +0.04pp residual is a scoring / text-normalization difference vs -that methodology, not a numerical drift in the port. +| F32 | [moonshine-streaming-tiny-F32.gguf](https://huggingface.co/handy-computer/moonshine-streaming-tiny-gguf/resolve/main/moonshine-streaming-tiny-F32.gguf) | 178 MB | 4.53% | +| F16 | [moonshine-streaming-tiny-F16.gguf](https://huggingface.co/handy-computer/moonshine-streaming-tiny-gguf/resolve/main/moonshine-streaming-tiny-F16.gguf) | 90 MB | 4.53% | +| Q8_0 | [moonshine-streaming-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-streaming-tiny-gguf/resolve/main/moonshine-streaming-tiny-Q8_0.gguf) | 50 MB | 4.52% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances). Figures without a commit were published before provenance was recorded. + + + +Greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference baseline: 4.53%. +The HF Transformers reference scored on the same manifest in the same regime lands +at 4.52% with 99.6% byte-identical hypotheses to our F32, so the port is at exact +parity with the reference. Useful Sensors' self-reported number on this split is +4.49% from the Open ASR Leaderboard table; the +0.04pp residual is a scoring / +text-normalization difference vs that methodology, not a numerical drift in the +port. Q6_K / Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 18.18% | + Q6_K / Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. @@ -76,43 +96,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 5 iterations after 2 warmups), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above -(2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | -| ------- | ------------ | -----------: | -| Metal | jfk (11.0s) | 50 ms (218×) | -| Metal | dots (35.3s) | 355 ms (100×) | -| CPU | jfk (11.0s) | 44 ms (250×) | -| CPU | dots (35.3s) | 206 ms (172×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | +| ------- | ------------ | ---------------: | +| Metal | jfk (11.0s) | 46 ms (239.48×) | +| Metal | dots (35.3s) | 316 ms (111.87×) | +| CPU | jfk (11.0s) | 41 ms (270.63×) | +| CPU | dots (35.3s) | 210 ms (168.59×) | -macOS 26.4.1, transcribe.cpp `0d312ce`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | -| ------- | ------------ | ------------: | -| Vulkan | jfk (11.0s) | 140 ms (79×) | -| Vulkan | dots (35.3s) | 892 ms (40×) | -| CPU | jfk (11.0s) | 160 ms (69×) | -| CPU | dots (35.3s) | 882 ms (40×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -Fedora 43, transcribe.cpp `f243f34`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Vulkan | jfk (11.0s) | 138 ms (80.00×) | +| Vulkan | dots (35.3s) | 922 ms (38.30×) | +| CPU | jfk (11.0s) | 146 ms (75.29×) | +| CPU | dots (35.3s) | 948 ms (37.25×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models moonshine-streaming-tiny \ - --quants q8_0 \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 5 --warmup 2 \ - --name moonshine-streaming-publication +uv run scripts/bench/run.py --profile --models moonshine-streaming-tiny ``` ## Numerical Validation @@ -120,9 +137,8 @@ uv run scripts/bench/run.py \ transcribe.cpp is validated tensor-by-tensor against the HF Transformers reference (`MoonshineStreamingForConditionalGeneration`, fp32 inference, `attn_implementation="eager"`) on `samples/jfk.wav`. All 37 contract -tensors fall within family tolerance, and the final transcript matches -the reference. Last validated at commit -[`0d312ce`](https://github.com/handy-computer/transcribe.cpp/tree/0d312ce). +tensors fall within family tolerance, and the final transcript matches the +reference. | Field | Value | | --- | --- | @@ -131,22 +147,6 @@ the reference. Last validated at commit | Manifest | `tests/golden/moonshine_streaming/moonshine-streaming-tiny.manifest.json` | | Command | `uv run scripts/validate.py all --family moonshine_streaming --variant moonshine-streaming-tiny` | -Selected tensors (max-abs and mean-abs differences, F32 vs reference, on -`samples/jfk.wav`): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.embedder.cmvn.out` | `2.240e-04` | `9.942e-06` | Frontend CMVN output | -| `enc.embedder.linear.out` | `2.840e-04` | `6.028e-06` | Time-domain linear projection | -| `enc.embedder.conv2.out` | `9.388e-05` | `1.945e-06` | After 2× causal stride-2 convs | -| `enc.block.0.out` | `1.099e-03` | `1.158e-05` | First sliding-window attention block | -| `enc.block.5.out` | `6.105e-03` | `2.195e-04` | Last encoder block — depth-amplified BLAS reduction drift | -| `enc.final` | `2.586e-04` | `8.776e-06` | Final encoder LN output | -| `adapter.out` | `2.677e-04` | `9.082e-06` | Encoder→decoder adapter (learned pos-emb add) | -| `dec.block.0.out` | `1.028e-03` | `4.340e-05` | First decoder block | -| `dec.block.5.out` | `2.746e-03` | `2.010e-04` | Last decoder block | -| `dec.logits_raw.gen20` | `1.829e-03` | `1.186e-04` | Mid-generation logits (token 20) | - The dominant drift source is BLAS reduction-order differences between PyTorch's matmul kernels and ggml's `mul_mat` (Accelerate / Metal / ggml-cpu). Drift accumulates roughly linearly with depth across encoder + diff --git a/docs/models/moonshine-streaming.md b/docs/models/moonshine-streaming.md index 200a8160..82af534e 100644 --- a/docs/models/moonshine-streaming.md +++ b/docs/models/moonshine-streaming.md @@ -15,14 +15,14 @@ see the family doc at ## Choosing a variant -- **Smallest footprint.** `moonshine-streaming-tiny` (34M params) at - Q8_0 is 48 MB. Decodes well above realtime on Apple Silicon and - Vulkan-class GPUs; CPU is also viable for live use. -- **Better accuracy, modest cost.** `moonshine-streaming-small` (123M - params) at Q8_0 is 189 MB. Roughly halves WER vs tiny. -- **Best accuracy in the family.** `moonshine-streaming-medium` (245M - params) at Q8_0 is 282 MB. Keep an eye on decode latency — the - 14-layer decoder dominates wall time on long utterances. +- **Smallest footprint.** `moonshine-streaming-tiny` decodes well above + realtime on Apple Silicon and Vulkan-class GPUs; CPU is also viable + for live use. +- **Better accuracy, modest cost.** `moonshine-streaming-small` roughly + halves WER vs tiny. +- **Best accuracy in the family.** `moonshine-streaming-medium` has the + most accuracy headroom. Keep an eye on decode latency; the 14-layer + decoder dominates wall time on long utterances. - **Non-streaming, batch-only workloads.** Use the [`moonshine`](moonshine.md) family instead — it's smaller for the same WER on offline audio because it doesn't carry the streaming @@ -50,11 +50,13 @@ upstream-reported numbers is a scoring / text-normalization difference, not a numerical drift in the port. The K-tier presets (Q6_K / Q5_K_M / Q4_K_M) are not currently shipped for this family. -| Variant | Params | Q8_0 size | WER (Q8_0) | Doc | -| --- | ---: | ---: | ---: | --- | -| `moonshine-streaming-tiny` | 34M | 48 MB | 4.52% | [moonshine-streaming-tiny.md](moonshine-streaming-tiny.md) | -| `moonshine-streaming-small` | 123M | 189 MB | 2.54% | [moonshine-streaming-small.md](moonshine-streaming-small.md) | -| `moonshine-streaming-medium` | 245M | 282 MB | 2.16% | [moonshine-streaming-medium.md](moonshine-streaming-medium.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `moonshine-streaming-tiny` | 44M | en | 50 MB | LibriSpeech test-clean (WER) | 4.52% | streaming | [moonshine-streaming-tiny.md](moonshine-streaming-tiny.md) | +| `moonshine-streaming-small` | 140M | en | 199 MB | LibriSpeech test-clean (WER) | 2.54% | streaming | [moonshine-streaming-small.md](moonshine-streaming-small.md) | +| `moonshine-streaming-medium` | 266M | en | 296 MB | LibriSpeech test-clean (WER) | 2.16% | streaming | [moonshine-streaming-medium.md](moonshine-streaming-medium.md) | + Pre-built GGUFs for every variant and quant are hosted under [`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); diff --git a/docs/models/moonshine-tiny.md b/docs/models/moonshine-tiny.md index 2065de75..89ec27b1 100644 --- a/docs/models/moonshine-tiny.md +++ b/docs/models/moonshine-tiny.md @@ -1,9 +1,13 @@ # Moonshine tiny -Useful Sensors' [`UsefulSensors/moonshine-tiny`](https://huggingface.co/UsefulSensors/moonshine-tiny) -ported to transcribe.cpp. A 27M-parameter encoder-decoder transformer that -consumes raw 16 kHz PCM directly (no STFT, no mel filterbank) via a three-layer -Conv1d stem. + +Upstream: [`UsefulSensors/moonshine-tiny`](https://huggingface.co/UsefulSensors/moonshine-tiny) at [`390624e`](https://huggingface.co/UsefulSensors/moonshine-tiny/commit/390624e). + +Useful Sensors Moonshine tiny — an encoder-decoder transformer for English +speech recognition. Consumes raw 16 kHz PCM directly via a three-layer Conv1d +stem (no STFT, no mel) and emits transcript-only output. English-only; no +translation, no language detection, no timestamps. + ## What it's for @@ -17,31 +21,39 @@ timestamps. See the [upstream model card](https://huggingface.co/UsefulSensors/moonshine-tiny) for training data, intended use, and the original evaluation methodology. -Licensed MIT. Ported from upstream commit -[`390624e`](https://huggingface.co/UsefulSensors/moonshine-tiny/commit/390624ed33d594443aa4aa221f5b9f283b545b5a), -pinned 2026-05-05. Validated against the transformers reference at -transcribe.cpp commit -[`07a8a84`](https://github.com/handy-computer/transcribe.cpp/tree/07a8a84) -on 2026-05-05. + +Licensed MIT. Ported from upstream commit [`390624e`](https://huggingface.co/UsefulSensors/moonshine-tiny/commit/390624e), pinned 2026-05-05. Validated against the transformers reference at transcribe.cpp commit [`07a8a84`](https://github.com/handy-computer/transcribe.cpp/tree/07a8a84) on 2026-05-05. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [moonshine-tiny-F32.gguf](https://huggingface.co/handy-computer/moonshine-tiny-gguf/resolve/main/moonshine-tiny-F32.gguf) | 105 MB | 4.58% | -| F16 | [moonshine-tiny-F16.gguf](https://huggingface.co/handy-computer/moonshine-tiny-gguf/resolve/main/moonshine-tiny-F16.gguf) | 57 MB | 4.58% | -| Q8_0 | [moonshine-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-tiny-gguf/resolve/main/moonshine-tiny-Q8_0.gguf) | 34 MB | 4.60% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with -the transcribe.cpp default decode (greedy, `num_beams=1`, `max_length=194` — -matching the upstream `generation_config`). Useful Sensors' self-reported -number on the same split is 4.55% (model card). Our F32 reference baseline -lands at 4.58%, within rounding of upstream and well within the ±1.00 pp -Stage 7 acceptance gate. Q8_0 drift is +0.02 pp vs F32 — within bootstrap CI -noise. Only F16 and Q8_0 are shipped as derived presets: at moonshine-tiny's -shapes (hidden 288, intermediate 1152, vocab 32768) none of the dimensions -divide the k-quant super-block size of 256, so Q6_K / Q5_K_M / Q4_K_M would -all fall back to Q8_0 storage and be near-duplicates. +| F32 | [moonshine-tiny-F32.gguf](https://huggingface.co/handy-computer/moonshine-tiny-gguf/resolve/main/moonshine-tiny-F32.gguf) | 110 MB | 4.58% | +| F16 | [moonshine-tiny-F16.gguf](https://huggingface.co/handy-computer/moonshine-tiny-gguf/resolve/main/moonshine-tiny-F16.gguf) | 59 MB | 4.58% | +| Q8_0 | [moonshine-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/moonshine-tiny-gguf/resolve/main/moonshine-tiny-Q8_0.gguf) | 35 MB | 4.60% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=194, +matching the upstream generation_config). Useful Sensors' self-reported number on +the same split is 4.55% (model card). Our F32 reference baseline lands at 4.58%, +within rounding of upstream and well within the ±1.00 pp Stage 7 acceptance gate. +Q8_0 drift is +0.02 pp vs F32 — within bootstrap CI noise. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 14.13% | + ## Quick Start @@ -62,43 +74,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 5 iterations after 2 warmups), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above -(2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | -| ------- | ------------ | -----------: | -| Metal | jfk (11.0s) | 61 ms (180×) | -| Metal | dots (35.3s) | 478 ms (74×) | -| CPU | jfk (11.0s) | 52 ms (210×) | -| CPU | dots (35.3s) | 366 ms (97×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Metal | jfk (11.0s) | 56 ms (197.19×) | +| Metal | dots (35.3s) | 422 ms (83.81×) | +| CPU | jfk (11.0s) | 54 ms (201.68×) | +| CPU | dots (35.3s) | 373 ms (94.78×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | -| ------- | ------------ | ------------: | -| Vulkan | jfk (11.0s) | 143 ms (77×) | -| Vulkan | dots (35.3s) | 1.02 s (35×) | -| CPU | jfk (11.0s) | 163 ms (68×) | -| CPU | dots (35.3s) | 1.53 s (23×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | +| ------- | ------------ | --------------: | +| Vulkan | jfk (11.0s) | 132 ms (83.21×) | +| Vulkan | dots (35.3s) | 938 ms (37.65×) | +| CPU | jfk (11.0s) | 175 ms (62.93×) | +| CPU | dots (35.3s) | 1.79 s (19.71×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models moonshine-tiny \ - --quants q8_0 \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 5 --warmup 2 \ - --name moonshine-publication +uv run scripts/bench/run.py --profile --models moonshine-tiny ``` ## Numerical Validation diff --git a/docs/models/moonshine.md b/docs/models/moonshine.md index 41bea350..0d43338f 100644 --- a/docs/models/moonshine.md +++ b/docs/models/moonshine.md @@ -16,13 +16,12 @@ see the family doc at ## Choosing a variant -- **Smallest footprint, near-realtime CPU.** `moonshine-tiny` (27M - params) at Q8_0 is 34 MB and decodes well above realtime on commodity - hardware. WER is on par with `whisper-tiny.en` while running on raw - audio (no mel frontend in the load path). -- **Higher accuracy, still small.** `moonshine-base` (61M params) at - Q8_0 is 74 MB and lands inside `whisper-small.en` accuracy territory - for English audio. +- **Smallest footprint, near-realtime CPU.** `moonshine-tiny` decodes + well above realtime on commodity hardware. WER is on par with + `whisper-tiny.en` while running on raw audio (no mel frontend in the + load path). +- **Higher accuracy, still small.** `moonshine-base` lands inside + `whisper-small.en` accuracy territory for English audio. - **Streaming workloads.** Moonshine is **not** streaming-first — the encoder is global, the decoder runs on the whole utterance. If you need chunked / real-time decoding, see @@ -49,10 +48,12 @@ to Q8_0 storage and would be near-duplicates. ### English -| Variant | Params | Q8_0 size | WER (LibriSpeech test-clean) | Doc | -| --- | ---: | ---: | ---: | --- | -| `moonshine-tiny` | 27M | 34 MB | 4.60% | [moonshine-tiny.md](moonshine-tiny.md) | -| `moonshine-base` | 61M | 74 MB | 3.26% | [moonshine-base.md](moonshine-base.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `moonshine-tiny` | 27M | en | 35 MB | LibriSpeech test-clean (WER) | 4.60% | - | [moonshine-tiny.md](moonshine-tiny.md) | +| `moonshine-base` | 62M | en | 77 MB | LibriSpeech test-clean (WER) | 3.26% | - | [moonshine-base.md](moonshine-base.md) | + ### Language-specific (Useful Sensors fine-tunes) @@ -62,26 +63,49 @@ because Useful Sensors does not publish per-language WER/CER for these variants. See each repo's `README.md` on Hugging Face for the full F32 / F16 / Q8_0 table and the reference baseline. -| Variant | Lang | Params | Q8_0 size | Metric (FLEURS test) | Q8_0 | Repo | -| --- | --- | ---: | ---: | --- | ---: | --- | -| `moonshine-tiny-vi` | Vietnamese (vi) | 27M | 34 MB | WER | 13.16% | [handy-computer/moonshine-tiny-vi-gguf](https://huggingface.co/handy-computer/moonshine-tiny-vi-gguf) | -| `moonshine-tiny-uk` | Ukrainian (uk) | 27M | 34 MB | WER | 18.89% | [handy-computer/moonshine-tiny-uk-gguf](https://huggingface.co/handy-computer/moonshine-tiny-uk-gguf) | -| `moonshine-tiny-zh` | Mandarin (zh) | 27M | 34 MB | CER | 13.78% | [handy-computer/moonshine-tiny-zh-gguf](https://huggingface.co/handy-computer/moonshine-tiny-zh-gguf) | -| `moonshine-tiny-ko` | Korean (ko) | 27M | 34 MB | CER | 8.98% | [handy-computer/moonshine-tiny-ko-gguf](https://huggingface.co/handy-computer/moonshine-tiny-ko-gguf) | -| `moonshine-tiny-ar` | Arabic (ar) | 27M | 34 MB | WER | 26.79% | [handy-computer/moonshine-tiny-ar-gguf](https://huggingface.co/handy-computer/moonshine-tiny-ar-gguf) | -| `moonshine-tiny-ja` | Japanese (ja) | 27M | 34 MB | CER | 13.36% | [handy-computer/moonshine-tiny-ja-gguf](https://huggingface.co/handy-computer/moonshine-tiny-ja-gguf) | -| `moonshine-base-vi` | Vietnamese (vi) | 61M | 74 MB | WER | 9.79% | [handy-computer/moonshine-base-vi-gguf](https://huggingface.co/handy-computer/moonshine-base-vi-gguf) | -| `moonshine-base-uk` | Ukrainian (uk) | 61M | 74 MB | WER | 14.39% | [handy-computer/moonshine-base-uk-gguf](https://huggingface.co/handy-computer/moonshine-base-uk-gguf) | -| `moonshine-base-zh` | Mandarin (zh) | 61M | 74 MB | CER | 17.00% | [handy-computer/moonshine-base-zh-gguf](https://huggingface.co/handy-computer/moonshine-base-zh-gguf) | -| `moonshine-base-ko` | Korean (ko) | 61M | 74 MB | CER | 8.13% | [handy-computer/moonshine-base-ko-gguf](https://huggingface.co/handy-computer/moonshine-base-ko-gguf) | -| `moonshine-base-ar` | Arabic (ar) | 61M | 74 MB | WER | 24.50% | [handy-computer/moonshine-base-ar-gguf](https://huggingface.co/handy-computer/moonshine-base-ar-gguf) | -| `moonshine-base-ja` | Japanese (ja) | 61M | 74 MB | CER | 10.53% | [handy-computer/moonshine-base-ja-gguf](https://huggingface.co/handy-computer/moonshine-base-ja-gguf) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `moonshine-tiny-vi` | 27M | vi | 35 MB | FLEURS vi (WER) | 13.37% | - | [handy-computer/moonshine-tiny-vi-gguf](https://huggingface.co/handy-computer/moonshine-tiny-vi-gguf) | +| `moonshine-tiny-uk` | 27M | uk | 35 MB | FLEURS uk (WER) | 18.76% | - | [handy-computer/moonshine-tiny-uk-gguf](https://huggingface.co/handy-computer/moonshine-tiny-uk-gguf) | +| `moonshine-tiny-zh` | 27M | zh | 35 MB | FLEURS zh (CER) | 13.88% | - | [handy-computer/moonshine-tiny-zh-gguf](https://huggingface.co/handy-computer/moonshine-tiny-zh-gguf) | +| `moonshine-tiny-ko` | 27M | ko | 35 MB | FLEURS ko (CER) | 9.00% | - | [handy-computer/moonshine-tiny-ko-gguf](https://huggingface.co/handy-computer/moonshine-tiny-ko-gguf) | +| `moonshine-tiny-ar` | 27M | ar | 35 MB | FLEURS ar (WER) | 26.70% | - | [handy-computer/moonshine-tiny-ar-gguf](https://huggingface.co/handy-computer/moonshine-tiny-ar-gguf) | +| `moonshine-tiny-ja` | 27M | ja | 35 MB | FLEURS ja (CER) | 13.44% | - | [handy-computer/moonshine-tiny-ja-gguf](https://huggingface.co/handy-computer/moonshine-tiny-ja-gguf) | +| `moonshine-base-vi` | 62M | vi | 77 MB | FLEURS vi (WER) | 9.96% | - | [handy-computer/moonshine-base-vi-gguf](https://huggingface.co/handy-computer/moonshine-base-vi-gguf) | +| `moonshine-base-uk` | 62M | uk | 77 MB | FLEURS uk (WER) | 14.38% | - | [handy-computer/moonshine-base-uk-gguf](https://huggingface.co/handy-computer/moonshine-base-uk-gguf) | +| `moonshine-base-zh` | 62M | zh | 77 MB | FLEURS zh (CER) | 17.79% | - | [handy-computer/moonshine-base-zh-gguf](https://huggingface.co/handy-computer/moonshine-base-zh-gguf) | +| `moonshine-base-ko` | 62M | ko | 77 MB | FLEURS ko (CER) | 8.12% | - | [handy-computer/moonshine-base-ko-gguf](https://huggingface.co/handy-computer/moonshine-base-ko-gguf) | +| `moonshine-base-ar` | 62M | ar | 77 MB | FLEURS ar (WER) | 24.62% | - | [handy-computer/moonshine-base-ar-gguf](https://huggingface.co/handy-computer/moonshine-base-ar-gguf) | +| `moonshine-base-ja` | 62M | ja | 77 MB | FLEURS ja (CER) | 11.11% | - | [handy-computer/moonshine-base-ja-gguf](https://huggingface.co/handy-computer/moonshine-base-ja-gguf) | + Pre-built GGUFs for every variant and quant are hosted under [`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); each per-variant repo's `README.md` has direct download links and the full F32 / F16 / Q8_0 measurement table. +## Performance + +The language-specific checkpoints have exactly the same architecture and tensor +shapes as their corresponding English checkpoint; only the trained weight +values differ. They therefore inherit the English checkpoint's per-quant speed +measurements rather than claiming separate benchmark runs: + +- every `moonshine-tiny-{ar,ja,ko,uk,vi,zh}` row is measured on + `moonshine-tiny`; +- every `moonshine-base-{ar,ja,ko,uk,vi,zh}` row is measured on + `moonshine-base`. + +These are the published Q8_0 averages; the per-sample latency and xRT tables are +in [moonshine-tiny.md](moonshine-tiny.md#performance) and +[moonshine-base.md](moonshine-base.md#performance). + +| Size | Apple M4 Max Metal | Apple M4 Max CPU | Ryzen 4750U Vulkan | Ryzen 4750U CPU | +| --- | ---: | ---: | ---: | ---: | +| tiny and tiny language fine-tunes | 127x | 153.5x | 56x | 45.5x | +| base and base language fine-tunes | 79.5x | 80.5x | 34.5x | 22x | + ## Input limits Moonshine has no input-length limit, but its decoder is capped at a short output diff --git a/docs/models/moss-transcribe-diarize.md b/docs/models/moss-transcribe-diarize.md index e1d920f1..856e80d2 100644 --- a/docs/models/moss-transcribe-diarize.md +++ b/docs/models/moss-transcribe-diarize.md @@ -1,11 +1,16 @@ # MOSS-Transcribe-Diarize -OpenMOSS's [`OpenMOSS-Team/MOSS-Transcribe-Diarize`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) -ported to transcribe.cpp. A 0.9B audio-LLM: a 24-layer Whisper-Medium audio -encoder (`d_model=1024`, GELU, LayerNorm) feeds a 4x temporal merge -(1024 -> 4096) and a VQAdaptor MLP bridge into a Qwen3-0.6B causal decoder -(28 layers, `hidden_size=1024`, GQA 16/8 heads, `rope_theta=1e6`) via -audio-token injection at `<|audio_pad|>` positions. + +Upstream: [`OpenMOSS-Team/MOSS-Transcribe-Diarize`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) at [`d7231bb`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize/commit/d7231bb). + +Offline English/Chinese speech-to-text with speaker diarization. A 0.9B +audio-LLM: a Whisper-Medium encoder (24 layers, d_model=1024) feeds a +4x temporal merge + VQAdaptor bridge into a Qwen3-0.6B decoder (28 layers) +via audio-token injection. The model emits `[start][Sxx]text[end]`; the +runtime parses those generated markers into clean text and segment rows. +Speaker attribution is opt-in (`--diarize`) and returns structured speaker +ids/turns. Not a streaming model. + ## What it's for @@ -21,9 +26,9 @@ for training data, intended use, and upstream evaluation. All of OpenMOSS's published metrics are Chinese multi-speaker diarization CER/cpCER; LibriSpeech test-clean is used here only as an English acceptance set. -Licensed Apache-2.0. Ported from upstream commit -[`d7231bb`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize/commit/d7231bbae2587a4af278735eb765b318c4f64edd), -pinned 2026-07-12. + +Licensed Apache-2.0. Ported from upstream commit [`d7231bb`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize/commit/d7231bb), pinned 2026-07-12. Validated against the MOSS author repo (OpenMOSS/MOSS-Transcribe-Diarize) reference at transcribe.cpp commit [`3f5e15c`](https://github.com/handy-computer/transcribe.cpp/tree/3f5e15c) on 2026-07-12. + ## Memory and length @@ -37,33 +42,47 @@ into shorter pieces. ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [MOSS-Transcribe-Diarize-BF16.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-BF16.gguf) | 1.83 GB | 2.08% | -| F16 | [MOSS-Transcribe-Diarize-F16.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-F16.gguf) | 1.83 GB | 2.07% | -| Q8_0 | [MOSS-Transcribe-Diarize-Q8_0.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q8_0.gguf) | 987 MB | 1.93% | -| Q6_K | [MOSS-Transcribe-Diarize-Q6_K.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q6_K.gguf) | 768 MB | 1.96% | -| Q5_K_M | [MOSS-Transcribe-Diarize-Q5_K_M.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q5_K_M.gguf) | 700 MB | 1.99% | -| Q4_K_M | [MOSS-Transcribe-Diarize-Q4_K_M.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q4_K_M.gguf) | 617 MB | 2.59% | - -These WER values describe this dataset only, not a general quality ranking. A -quant that scores slightly better here is not necessarily better in real-world -use; dataset-specific decoding near-ties can make quantization noise help or -hurt individual utterances. - -WER measured on the full LibriSpeech `test-clean` split (2620 utterances) with -the Whisper-style English normalizer and jiwer 3.x. MOSS emits the diarized -format `[start][Sxx]text[end]`; the bracket spans are metadata and are -de-diarized to a space (for both hypothesis and reference) before scoring, -matching the author-repo reference runner. The same-manifest MOSS author-repo -reference (bf16, greedy) lands at **2.07%**, 95% bootstrap CI [1.82%, 2.40%]; -the BF16 port lands at 2.08%, within `+0.01pp` of the reference and well inside -the CI. Q4_K_M's higher 2.59% is not broad degradation but a handful of 4-bit -tail failures (6 empty outputs, 5 English->Chinese language-drift utterances, -1 timestamp-token repetition loop); prefer Q5_K_M or higher if those matter. -The runtime applies the same marker removal to `full_text`, so WER scoring and -the public transcript agree. The pre-parsed inline marker string remains -available verbatim via `transcribe_raw_text()`. +| BF16 | [MOSS-Transcribe-Diarize-BF16.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-BF16.gguf) | 1.83 GB | 2.08% | +| F16 | [MOSS-Transcribe-Diarize-F16.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-F16.gguf) | 1.83 GB | 2.07% | +| Q8_0 | [MOSS-Transcribe-Diarize-Q8_0.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q8_0.gguf) | 987 MB | 1.93% | +| Q6_K | [MOSS-Transcribe-Diarize-Q6_K.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q6_K.gguf) | 768 MB | 1.96% | +| Q5_K_M | [MOSS-Transcribe-Diarize-Q5_K_M.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q5_K_M.gguf) | 700 MB | 1.99% | +| Q4_K_M | [MOSS-Transcribe-Diarize-Q4_K_M.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q4_K_M.gguf) | 617 MB | 2.59% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Scored with the Whisper-style English text normalizer and jiwer 3.x. MOSS emits the +diarized format `[start][Sxx]text[end]`; the bracket spans are metadata and are +de-diarized to a space (for both hypothesis and reference) before scoring, matching +the author-repo reference runner. These values describe this dataset only, not a +general quality ranking: a quant that scores slightly better here is not necessarily +better in real-world use, because dataset-specific decoding near-ties can make +quantization noise help or hurt individual utterances. The same-manifest MOSS +author-repo reference (bf16, greedy) lands at **2.07%** with 95% bootstrap CI +[1.82%, 2.40%]. The BF16 port lands at 2.08% (within +0.01 of the reference, well +inside the CI band); the lower-bit presets sit between 1.93% and 1.99% (statistical +noise) except Q4_K_M at 2.59%, whose excess is a handful of 4-bit tail failures (6 +empty outputs, 5 English->Chinese language-drift utterances, 1 timestamp-token +repetition loop) rather than broad degradation. Prefer Q5_K_M or higher if those +tail failures matter. Reproduce with `scripts/wer/run.py` + `scripts/wer/score.py +--dediarize`; public `full_text` applies equivalent marker removal. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 5.13% | +| zh | CER | 9.23% | + ## Quick Start @@ -95,43 +114,40 @@ CLI flags: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 388 ms (28.3×) | 369 ms (29.8×) | -| Metal | dots (35.3s) | 1.27 s (27.8×) | 1.17 s (30.1×) | -| CPU | jfk (11.0s) | 2.06 s (5.3×) | 2.37 s (4.6×) | -| CPU | dots (35.3s) | 5.71 s (6.2×) | 5.84 s (6.0×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 393 ms (27.98×) | 382 ms (28.82×) | +| Metal | dots (35.3s) | 1.40 s (25.17×) | 1.22 s (29.01×) | +| CPU | jfk (11.0s) | 2.08 s (5.28×) | 2.20 s (5.00×) | +| CPU | dots (35.3s) | 5.43 s (6.50×) | 5.47 s (6.46×) | -macOS 26.5.1, transcribe.cpp `e745720`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 3.88 s (2.8×) | 3.68 s (3.0×) | -| Vulkan | dots (35.3s) | 11.38 s (3.1×) | 10.68 s (3.3×) | -| CPU | jfk (11.0s) | 7.49 s (1.5×) | 7.06 s (1.6×) | -| CPU | dots (35.3s) | 21.20 s (1.7×) | 19.22 s (1.8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -Fedora Linux 43, transcribe.cpp `e745720`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 3.73 s (2.95×) | 3.48 s (3.16×) | +| Vulkan | dots (35.3s) | 11.09 s (3.18×) | 9.95 s (3.55×) | +| CPU | jfk (11.0s) | 7.54 s (1.46×) | 6.90 s (1.59×) | +| CPU | dots (35.3s) | 21.08 s (1.68×) | 19.24 s (1.84×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models moss-transcribe-diarize \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name moss-transcribe-diarize-publication +uv run scripts/bench/run.py --profile --models moss-transcribe-diarize ``` ## Numerical Validation @@ -140,15 +156,14 @@ transcribe.cpp is validated tensor-by-tensor against the MOSS author repo (`scripts/dump_reference_moss_author.py`, `trust_remote_code`) on `samples/jfk.wav` with the strict CPU backend. The reference runs BF16 (torch, eager attention); the C++ path dequantizes BF16 weights to F32 and computes in -F32, so C++ is the *more* precise side and the residual gap is a constant -~1-3% relative bf16-vs-f32 drift, not a bug. The transcript compare is -`dediarized` (bracket metadata stripped to a space). Confirmed WER-neutral: on -the first 100 test-clean utterances the C++ ref-dtype WER (1.40%) is -bit-identical to the Oracle reference on the same subset (1.40%). Tolerances -are pinned in `tests/tolerances/moss.json` with a `_comment` block naming the -precision regime, the large-pre-normalization-activation maxes, and the encoder -padding-trim contract. Last validated at commit -[`3f5e15c`](https://github.com/handy-computer/transcribe.cpp/tree/3f5e15c). +F32, so C++ is the *more* precise side and the residual gap is a constant ~1-3% +relative bf16-vs-f32 drift, not a bug. The transcript compare is `dediarized` +(bracket metadata stripped to a space). Confirmed WER-neutral: on the first 100 +test-clean utterances the C++ ref-dtype WER (1.40%) is bit-identical to the +Oracle reference on the same subset (1.40%). Tolerances are pinned in +`tests/tolerances/moss.json` with a `_comment` block naming the precision +regime, the large-pre-normalization-activation maxes, and the encoder +padding-trim contract. | Field | Value | | --- | --- | @@ -158,26 +173,6 @@ padding-trim contract. Last validated at commit | Tolerances | `tests/tolerances/moss.json` | | Command | `uv run scripts/validate.py all --family moss --variant moss-transcribe-diarize` | -Selected tensors (observed on CPU, strict backend; see the tolerance file for -budgets and per-tensor notes): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.mel.in` | `9.872e-05` | `5.150e-06` | C++ MelFrontend vs reference feature extractor | -| `enc.pos_add.out` | `5.000e-02` | `2.200e-03` | Encoder input + positional embedding | -| `enc.block.0.out` | `1.200e-01` | `4.000e-03` | First Whisper encoder block | -| `enc.block.23.out` | `3.100e+03` | `8.000e-02` | Pre-final-LN residual; ref \|max\| ~3.6e3, bf16 rel error dominates max_abs (renormalized by `enc.ln_post`) | -| `enc.ln_post.out` | `1.300e+01` | `3.000e-03` | Encoder output LayerNorm | -| `enc.merge.out` | `1.500e+00` | `3.000e-03` | 4x temporal merge | -| `enc.adaptor.out` | `4.000e-01` | `1.200e-02` | VQAdaptor decoder handoff (rel_mean ~0.96%) | -| `dec.audio_injected` | `4.000e-01` | `7.000e-03` | Audio tokens scattered into the prompt | -| `dec.block.0.out` | `7.500e-01` | `9.500e-03` | First Qwen3 decoder block | -| `dec.block.27.out` | `3.200e+02` | `2.600e-01` | Pre-final-RMSNorm residual (bf16 accumulation over 28 layers) | -| `dec.out_before_head` | `8.500e+00` | `8.000e-02` | Pre-head hidden state | -| `dec.logits_raw` | `4.800e-01` | `7.200e-02` | Prefill logits; argmax preserved (transcript exact) | -| `dec.logits_raw.gen8` | `5.000e-01` | `6.500e-02` | Greedy step 8 logits (KV-cache decode coverage) | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | Pure embedding lookup (pinned exact) | - For the full porting writeup, see [`docs/porting/families/moss.md`](../porting/families/moss.md). diff --git a/docs/models/multitalker-parakeet-streaming-0.6b-v1.md b/docs/models/multitalker-parakeet-streaming-0.6b-v1.md index e3f0a38d..f315f7c6 100644 --- a/docs/models/multitalker-parakeet-streaming-0.6b-v1.md +++ b/docs/models/multitalker-parakeet-streaming-0.6b-v1.md @@ -1,9 +1,10 @@ # Multitalker Parakeet Streaming 0.6B v1 -NVIDIA's [`nvidia/multitalker-parakeet-streaming-0.6b-v1`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1) -ported to transcribe.cpp. A 0.6B-parameter cache-aware streaming -FastConformer encoder with an RNN-T transducer decoder, fine-tuned from -[`nvidia/nemotron-speech-streaming-en-0.6b`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b). + +Upstream: [`nvidia/multitalker-parakeet-streaming-0.6b-v1`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1) at [`8749fc7`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1/commit/8749fc7). + +Offline and cache-aware streaming English speech-to-text with punctuation and capitalization. A cache-aware streaming FastConformer encoder with an RNN-T transducer decoder, fine-tuned from nvidia/nemotron-speech-streaming-en-0.6b. Plain GGUFs run the single_speaker_mode ASR path, while bundle GGUFs under `bundle/` embed nvidia/diar_streaming_sortformer_4spk-v2.1 and, with `--diarize`, transcribe up to four overlapping speakers into a speaker-tagged transcript. The encoder preserves the upstream att_context_size=[70, 13] (1.12s) cache-aware attention mask; all four latency lookahead settings are selectable. + ## What it's for @@ -13,12 +14,13 @@ word-level timestamps are available. Upstream this is a **multitalker (speaker-attributed)** checkpoint: it can transcribe several overlapping speakers into per-speaker channels. This -port ships that path too, via **bundle GGUFs** that embed the +port ships that path too: every published GGUF is a **bundle** that embeds +the [`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1) -streaming diarizer alongside the ASR model. A plain (non-bundle) GGUF runs -the model's `single_speaker_mode` ASR path — a cache-aware streaming RNN-T -with the checkpoint's always-on layer-0 speaker-kernel injection. A bundle -GGUF with `--diarize` runs the full multitalker pipeline and emits a +streaming diarizer alongside the ASR model. Run it without `--diarize` and +you get the model's `single_speaker_mode` ASR path — a cache-aware streaming +RNN-T with the checkpoint's always-on layer-0 speaker-kernel injection. Run +it with `--diarize` and you get the full multitalker pipeline and a speaker-tagged transcript (see [Multitalker](#multitalker-speaker-attributed-asr)). @@ -29,43 +31,79 @@ See NVIDIA's [model card](https://huggingface.co/nvidia/multitalker-parakeet-str for training data, intended use, the multitalker methodology, and the full latency-vs-accuracy table. -Licensed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). -Ported from upstream commit -[`8749fc7`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1/commit/8749fc71fd6e2d88ef230159bbf2aea69b524ee1), -pinned 2026-07-12. + +Licensed [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). Ported from upstream commit [`8749fc7`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1/commit/8749fc7), pinned 2026-07-12. Validated against the NeMo reference at transcribe.cpp commit [`3083021`](https://github.com/handy-computer/transcribe.cpp/tree/3083021) on 2026-08-03. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) | | --- | --- | ---: | ---: | -| F32 | [multitalker-parakeet-streaming-0.6b-v1-F32.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-F32.gguf) | 2.49 GB | 2.19% | -| F16 | [multitalker-parakeet-streaming-0.6b-v1-F16.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-F16.gguf) | 1.25 GB | 2.19% | -| Q8_0 | [multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf) | 734 MB | 2.18% | -| Q6_K | [multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf) | 604 MB | 2.20% | -| Q5_K_M | [multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf) | 542 MB | 2.18% | -| Q4_K_M | [multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf) | 478 MB | 2.18% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -in `single_speaker_mode` with greedy RNN-T decoding, whisper-normalizer -scoring (PnC-stripped), and no external LM. F32 reference baseline: 2.19%. -The measured NeMo `single_speaker_mode` reference on the same split is -2.19%, and NVIDIA's self-reported number is 2.19% (from the -[HF model card](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)). - -### Multitalker bundles - -Bundle GGUFs embed the streaming Sortformer diarizer. The tier names the -ASR half's dtype; the embedded diarizer is F32 for the F32 bundle, F16 for -F16, and Q8_0 for all k-quant tiers. +| F32 | [bundle/multitalker-parakeet-streaming-0.6b-v1-F32.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-F32.gguf) | 2.96 GB | 2.19% | +| F16 | [bundle/multitalker-parakeet-streaming-0.6b-v1-F16.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-F16.gguf) | 1.48 GB | 2.19% | +| Q8_0 | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf) | 873 MB | 2.18% | +| Q6_K | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf) | 743 MB | 2.20% | +| Q5_K_M | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf) | 681 MB | 2.18% | +| Q4_K_M | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf) | 617 MB | 2.18% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Run in single_speaker_mode with greedy RNN-T decoding and whisper-normalizer +(PnC-stripped) scoring. F32 reference baseline: 2.19%. The measured NeMo +single_speaker_mode reference and NVIDIA's self-reported number on the same split +are both 2.19%. + +### Multitalker bundles (speaker-attributed ASR) + +Bundle GGUFs embed the streaming Sortformer diarizer alongside the ASR model. Run them with `--diarize` to get a speaker-tagged transcript with up to four speakers. The tier names the ASR half's dtype; the embedded diarizer is F32 for the F32 bundle, F16 for F16, and Q8_0 for all k-quant tiers. | Bundle | Download | Size | | --- | --- | ---: | -| F32 | [bundle/multitalker-parakeet-streaming-0.6b-v1-F32.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-F32.gguf) | 2.96 GB | -| F16 | [bundle/multitalker-parakeet-streaming-0.6b-v1-F16.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-F16.gguf) | 1.48 GB | -| Q8_0 | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf) | 873 MB | -| Q6_K | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf) | 743 MB | -| Q5_K_M | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf) | 681 MB | -| Q4_K_M | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf) | 617 MB | +| F32 | [bundle/multitalker-parakeet-streaming-0.6b-v1-F32.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-F32.gguf) | 2.96 GB | +| F16 | [bundle/multitalker-parakeet-streaming-0.6b-v1-F16.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-F16.gguf) | 1.48 GB | +| Q8_0 | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf) | 873 MB | +| Q6_K | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf) | 743 MB | +| Q5_K_M | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf) | 681 MB | +| Q4_K_M | [bundle/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/bundle/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf) | 617 MB | + +cpWER on AMI-IHM test (16 meetings, F32 bundle) is 19.35% in the default kernel mode and 23.73% in masked mode. The matched NeMo reference scores 21.39% and 24.00%, respectively; see the transcribe.cpp model page for the exactness accounting. + +```bash +build/bin/transcribe-cli --diarize \ + -m bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf \ + meeting.wav +``` + + + +**AMI IHM test, `kernel` mode** + +| Language | Metric | F32 | +| --- | --- | ---: | +| en | CPWER | 19.35% | + +**AMI IHM test, `masked` mode** + +| Language | Metric | F32 | +| --- | --- | ---: | +| en | CPWER | 23.73% | + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 6.52% | + + +### Bundle dtypes + +The tier names the ASR half's dtype; the embedded Sortformer diarizer is F32 +for the F32 bundle, F16 for F16, and Q8_0 for all k-quant tiers. ## Streaming parity @@ -118,43 +156,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 67 ms (164×) | 69 ms (159×) | -| Metal | dots (35.3s) | 184 ms (192×) | 185 ms (191×) | -| CPU | jfk (11.0s) | 310 ms (36×) | 307 ms (36×) | -| CPU | dots (35.3s) | 1.05 s (34×) | 1.03 s (34×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 53 ms (207.08×) | 54 ms (201.85×) | +| Metal | dots (35.3s) | 155 ms (227.85×) | 156 ms (226.47×) | +| CPU | jfk (11.0s) | 331 ms (33.25×) | 338 ms (32.58×) | +| CPU | dots (35.3s) | 1.10 s (32.18×) | 1.11 s (31.87×) | -macOS 26.5.1, transcribe.cpp `c55a09d`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 466 ms (24×) | 475 ms (23×) | -| Vulkan | dots (35.3s) | 1.36 s (26×) | 1.39 s (26×) | -| CPU | jfk (11.0s) | 751 ms (15×) | 816 ms (13×) | -| CPU | dots (35.3s) | 2.99 s (12×) | 3.12 s (11×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 439 ms (25.08×) | 438 ms (25.11×) | +| Vulkan | dots (35.3s) | 1.28 s (27.61×) | 1.30 s (27.12×) | +| CPU | jfk (11.0s) | 748 ms (14.70×) | 790 ms (13.92×) | +| CPU | dots (35.3s) | 2.91 s (12.13×) | 2.96 s (11.95×) | -Fedora 43, transcribe.cpp `c55a09d`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models multitalker-parakeet-streaming-0.6b-v1 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name multitalker-parakeet-streaming-0.6b-v1-publication +uv run scripts/bench/run.py --profile --models multitalker-parakeet-streaming-0.6b-v1 ``` ## Numerical Validation diff --git a/docs/models/nemotron-3.5-asr-streaming-0.6b.md b/docs/models/nemotron-3.5-asr-streaming-0.6b.md index 8dbaaedd..36e6c0bc 100644 --- a/docs/models/nemotron-3.5-asr-streaming-0.6b.md +++ b/docs/models/nemotron-3.5-asr-streaming-0.6b.md @@ -1,10 +1,10 @@ # Nemotron 3.5 ASR Streaming 0.6B -NVIDIA's [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b) -ported to transcribe.cpp. A 0.6B-parameter cache-aware streaming -FastConformer encoder with an RNN-T transducer decoder — the multilingual -successor to -[`nemotron-speech-streaming-en-0.6b`](nemotron-speech-streaming-en-0.6b.md). + +Upstream: [`nvidia/nemotron-3.5-asr-streaming-0.6b`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b) at [`24b151a`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b/commit/24b151a). + +Multilingual speech-to-text across 32 supported language-locales (the model's tokenizer recognizes 40, but 8 are adaptation-ready and need fine-tuning) with punctuation and capitalization. A cache-aware streaming FastConformer encoder with a prompt-conditioned RNN-T transducer decoder; the target language is selected per call (--language en-US, fr-FR, de-DE, ...) and an auto mode emits a tag. Ships both the offline path (att_context_size=[56, 13], 1.12s, headline accuracy) and runtime-selectable chunked streaming (--stream-chunk-ms 1120 --stream-att-right {0,3,6,13}). + ## What it's for @@ -35,10 +35,9 @@ See NVIDIA's [model card](https://huggingface.co/nvidia/nemotron-3.5-asr-streami for training data, the full language list, intended use, and the latency-vs-accuracy table. -Licensed under [OpenMDW-1.1](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b). -Ported from upstream commit -[`24b151a`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b/commit/24b151a851dd15909e1fc611b11bb2da52b9fc81), -pinned 2026-06-08. + +Licensed [OpenMDW-1.1](https://openmdw.ai/license/1-1/). Ported from upstream commit [`24b151a`](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b/commit/24b151a), pinned 2026-06-08. Validated against the NeMo reference at transcribe.cpp commit [`909e94e`](https://github.com/handy-computer/transcribe.cpp/tree/909e94e) on 2026-06-08. + ## Input limits @@ -53,37 +52,68 @@ stays unbounded for the same reason. See the ## Download -| Quantization | Download | Size | + +| Quantization | Download | Size | +| --- | --- | ---: | +| F32 | [nemotron-3.5-asr-streaming-0.6b-F32.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-F32.gguf) | 2.55 GB | +| F16 | [nemotron-3.5-asr-streaming-0.6b-F16.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-F16.gguf) | 1.28 GB | +| Q8_0 | [nemotron-3.5-asr-streaming-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q8_0.gguf) | 751 MB | +| Q6_K | [nemotron-3.5-asr-streaming-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q6_K.gguf) | 621 MB | +| Q5_K_M | [nemotron-3.5-asr-streaming-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q5_K_M.gguf) | 560 MB | +| Q4_K_M | [nemotron-3.5-asr-streaming-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q4_K_M.gguf) | 496 MB | + + + +WER on the full FLEURS en split (647 utterances), batch sizes 1 and 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy RNN-T decoding with whisper-normalizer scoring; the per-quant column is +FLEURS en. NeMo reference baseline on the same manifest: 7.99% (NVIDIA self-reports +7.91% en-US). On LibriSpeech test-clean (2620 utterances) the same presets score F32 +3.04 / F16 3.03 / Q8_0 3.06 / Q6_K 3.07 / Q5_K_M 3.10 / Q4_K_M 3.28, against a 3.03% +NeMo reference. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | | --- | --- | ---: | -| F32 | [nemotron-3.5-asr-streaming-0.6b-F32.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-F32.gguf) | 2.38 GB | -| F16 | [nemotron-3.5-asr-streaming-0.6b-F16.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-F16.gguf) | 1.19 GB | -| Q8_0 | [nemotron-3.5-asr-streaming-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q8_0.gguf) | 716 MB | -| Q6_K | [nemotron-3.5-asr-streaming-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q6_K.gguf) | 593 MB | -| Q5_K_M | [nemotron-3.5-asr-streaming-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q5_K_M.gguf) | 534 MB | -| Q4_K_M | [nemotron-3.5-asr-streaming-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf/resolve/main/nemotron-3.5-asr-streaming-0.6b-Q4_K_M.gguf) | 473 MB | - -**Accuracy.** Word error rate at the offline `att_context_size=[56,13]` -(1.12 s) setting, `--language en-US`, greedy RNN-T. C++ hypotheses were -generated on an L4 GPU and scored with the whisper-normalizer; the -reference column is NVIDIA NeMo measured on the same manifests. For -context, NVIDIA's self-reported FLEURS en-US WER is **7.91%** (and an -**8.84%** 19-locale macro-average) per the -[HF model card](https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b). - -| Preset | FLEURS test en (n=647) | LibriSpeech test-clean (n=2620) | -| --- | ---: | ---: | -| Reference (NeMo) | 7.99 | 3.03 | -| F32 | 7.97 | 3.04 | -| F16 | 7.97 | 3.03 | -| Q8_0 | 7.88 | 3.06 | -| Q6_K | 8.02 | 3.07 | -| Q5_K_M | 8.15 | 3.10 | -| Q4_K_M | 8.49 | 3.28 | - -The F32 reference dtype meets the measured-Oracle gate on both datasets. -F16/Q8_0/Q6_K/Q5_K_M land inside the reference 95% CI; Q4_K_M carries the -largest quantization loss (+0.50 on FLEURS, +0.25 on LibriSpeech) but is -accepted for shipping. +| ar | WER | 15.93% | +| bg | WER | 22.02% | +| cs | WER | 23.00% | +| da | WER | 28.51% | +| de | WER | 10.33% | +| es | WER | 6.30% | +| et | WER | 31.84% | +| fi | WER | 21.91% | +| fr | WER | 10.78% | +| hi | WER | 8.61% | +| hr | WER | 26.21% | +| hu | WER | 32.12% | +| it | WER | 5.78% | +| ja | CER | 13.52% | +| ko | CER | 8.89% | +| nb | WER | 19.24% | +| nl | WER | 13.61% | +| pl | WER | 17.54% | +| pt | WER | 8.52% | +| ro | WER | 28.28% | +| ru | WER | 12.61% | +| sk | WER | 23.25% | +| sv | WER | 24.32% | +| tr | WER | 15.40% | +| uk | WER | 14.88% | +| vi | WER | 13.96% | +| zh | CER | 18.87% | + +**LibriSpeech test-clean** + +| Language | Metric | F32 | F16 | Q8_0 | Q6_K | Q5_K_M | Q4_K_M | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| en | WER | 3.04% | 3.04% | 3.05% | 3.08% | 3.10% | 3.30% | + ## Quick Start @@ -105,10 +135,6 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - The decoder runs through a reused ggml graph for the joint output projection (the 13k-vocab RNN-T joint that dominates this variant's decode) and a thread-parallel predictor; both are the default, so these @@ -116,38 +142,38 @@ are out-of-the-box numbers with no tuning. ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 113 ms (98×) | 113 ms (98×) | -| Metal | dots (35.3s) | 361 ms (98×) | 368 ms (96×) | -| CPU | jfk (11.0s) | 367 ms (30×) | 362 ms (30×) | -| CPU | dots (35.3s) | 1.28 s (28×) | 1.25 s (28×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.5 (Darwin 25.5.0), transcribe.cpp `d9708f1`. Metal device: -Apple M4 Max (`MTLGPUFamilyApple9`). +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 76 ms (143.94×) | 77 ms (143.14×) | +| Metal | dots (35.3s) | 256 ms (138.26×) | 256 ms (137.88×) | +| CPU | jfk (11.0s) | 358 ms (30.76×) | 355 ms (31.01×) | +| CPU | dots (35.3s) | 1.19 s (29.73×) | 1.21 s (29.23×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 773 ms (14×) | 783 ms (14×) | -| Vulkan | dots (35.3s) | 2.37 s (15×) | 2.37 s (15×) | -| CPU | jfk (11.0s) | 1.37 s (8×) | 1.09 s (10×) | -| CPU | dots (35.3s) | 4.76 s (7×) | 4.17 s (8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 640 ms (17.18×) | 644 ms (17.09×) | +| Vulkan | dots (35.3s) | 2.07 s (17.09×) | 2.09 s (16.88×) | +| CPU | jfk (11.0s) | 951 ms (11.56×) | 993 ms (11.07×) | +| CPU | dots (35.3s) | 3.67 s (9.62×) | 3.74 s (9.45×) | -Fedora 43, transcribe.cpp `ef35659`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models nemotron-3.5-asr-streaming-0.6b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name nemotron-3.5-asr-streaming-0.6b-publication +uv run scripts/bench/run.py --profile --models nemotron-3.5-asr-streaming-0.6b ``` ## Numerical Validation @@ -194,9 +220,9 @@ on WER (Stage 7), not tensor tolerances. - The auxiliary CTC head present in the upstream checkpoint is dropped at conversion (the RNN-T head is the inference path); CTC-argmax timestamps are not available. -- WER is gated on English only (FLEURS test en + LibriSpeech test-clean - against the NeMo Oracle). The other 39 locales are exercised - functionally but not WER-scored here. Published latency numbers cover +- The measured-Oracle release gate uses English (FLEURS test en + + LibriSpeech test-clean). The publication catalog additionally carries a + Q8_0 FLEURS result for every supported language. Published latency numbers cover the offline `[56, 13]` path; the sub-1.12 s streaming settings are functionally validated (byte-equal at R=13) but not separately benchmarked. diff --git a/docs/models/nemotron-speech-streaming-en-0.6b.md b/docs/models/nemotron-speech-streaming-en-0.6b.md index 4d9c1b31..e925208b 100644 --- a/docs/models/nemotron-speech-streaming-en-0.6b.md +++ b/docs/models/nemotron-speech-streaming-en-0.6b.md @@ -1,8 +1,10 @@ # Nemotron Speech Streaming EN 0.6B -NVIDIA's [`nvidia/nemotron-speech-streaming-en-0.6b`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b) -ported to transcribe.cpp. A 0.6B-parameter cache-aware streaming -FastConformer encoder with an RNN-T transducer decoder. + +Upstream: [`nvidia/nemotron-speech-streaming-en-0.6b`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b) at [`ef3bf40`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b/commit/ef3bf40). + +English speech-to-text with punctuation and capitalization. A cache-aware streaming FastConformer encoder with an RNN-T transducer decoder. Runs in both offline and cache-aware streaming modes. The encoder preserves the upstream att_context_size=[70, 13] (1.12s) cache-aware attention mask end-to-end. + ## What it's for @@ -28,27 +30,39 @@ See NVIDIA's [model card](https://huggingface.co/nvidia/nemotron-speech-streamin for training data, intended use, streaming methodology, and the full latency-vs-accuracy table. -Licensed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). -Ported from upstream commit -[`ef3bf40`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b/commit/ef3bf40c90df5cd2de55cc07e06681e03d8e6ee4), -pinned 2026-05-11. + +Licensed [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/). Ported from upstream commit [`ef3bf40`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b/commit/ef3bf40), pinned 2026-05-11. Validated against the NeMo reference at transcribe.cpp commit [`12f1076`](https://github.com/handy-computer/transcribe.cpp/tree/12f1076) on 2026-05-11. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) | | --- | --- | ---: | ---: | -| F32 | [nemotron-speech-streaming-en-0.6b-F32.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-F32.gguf) | 2.30 GB | 2.31% | -| F16 | [nemotron-speech-streaming-en-0.6b-F16.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-F16.gguf) | 1.16 GB | 2.31% | -| Q8_0 | [nemotron-speech-streaming-en-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q8_0.gguf) | 696 MB | 2.31% | -| Q6_K | [nemotron-speech-streaming-en-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q6_K.gguf) | 573 MB | 2.29% | -| Q5_K_M | [nemotron-speech-streaming-en-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q5_K_M.gguf) | 514 MB | 2.34% | -| Q4_K_M | [nemotron-speech-streaming-en-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q4_K_M.gguf) | 453 MB | 2.38% | - -WER is measured on the full LibriSpeech test-clean split (2620 -utterances) with greedy RNN-T decoding. F32 reference baseline: 2.31%. -NVIDIA's self-reported number on the same split at -`att_context_size=[70, 13]` (1.12s chunk, w/o PnC) is 2.32% (from the -[HF model card](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b)). +| F32 | [nemotron-speech-streaming-en-0.6b-F32.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-F32.gguf) | 2.47 GB | 2.31% | +| F16 | [nemotron-speech-streaming-en-0.6b-F16.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-F16.gguf) | 1.24 GB | 2.31% | +| Q8_0 | [nemotron-speech-streaming-en-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q8_0.gguf) | 730 MB | 2.31% | +| Q6_K | [nemotron-speech-streaming-en-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q6_K.gguf) | 600 MB | 2.29% | +| Q5_K_M | [nemotron-speech-streaming-en-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q5_K_M.gguf) | 539 MB | 2.34% | +| Q4_K_M | [nemotron-speech-streaming-en-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/nemotron-speech-streaming-en-0.6b-gguf/resolve/main/nemotron-speech-streaming-en-0.6b-Q4_K_M.gguf) | 475 MB | 2.38% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy RNN-T decoding. F32 reference baseline: 2.31%. NVIDIA's self-reported number +on the same split at att_context_size=[70, 13] (1.12s chunk, w/o PnC) is 2.32%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 6.43% | + ## Streaming WER @@ -95,43 +109,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 73 ms (151×) | 73 ms (151×) | -| Metal | dots (35.3s) | 224 ms (158×) | 221 ms (160×) | -| CPU | jfk (11.0s) | 329 ms (33×) | 330 ms (33×) | -| CPU | dots (35.3s) | 1.12 s (31×) | 1.12 s (31×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `12f1076`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 53 ms (206.74×) | 55 ms (200.58×) | +| Metal | dots (35.3s) | 157 ms (225.20×) | 158 ms (223.33×) | +| CPU | jfk (11.0s) | 327 ms (33.64×) | 335 ms (32.81×) | +| CPU | dots (35.3s) | 1.10 s (32.03×) | 1.11 s (31.89×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 812 ms (14×) | 813 ms (14×) | -| Vulkan | dots (35.3s) | 2.93 s (12×) | 2.98 s (12×) | -| CPU | jfk (11.0s) | 1.39 s (8×) | 1.22 s (9×) | -| CPU | dots (35.3s) | 5.21 s (7×) | 4.76 s (7×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 424 ms (25.97×) | 426 ms (25.83×) | +| Vulkan | dots (35.3s) | 1.25 s (28.32×) | 1.26 s (28.01×) | +| CPU | jfk (11.0s) | 743 ms (14.81×) | 788 ms (13.96×) | +| CPU | dots (35.3s) | 2.86 s (12.33×) | 2.93 s (12.06×) | -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models nemotron-speech-streaming-en-0.6b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name nemotron-speech-streaming-en-0.6b-publication +uv run scripts/bench/run.py --profile --models nemotron-speech-streaming-en-0.6b ``` ## Numerical Validation diff --git a/docs/models/parakeet-ctc-0.6b.md b/docs/models/parakeet-ctc-0.6b.md index a5d1dcc7..445bf994 100644 --- a/docs/models/parakeet-ctc-0.6b.md +++ b/docs/models/parakeet-ctc-0.6b.md @@ -1,9 +1,10 @@ # Parakeet CTC 0.6B -NVIDIA's [`nvidia/parakeet-ctc-0.6b`](https://huggingface.co/nvidia/parakeet-ctc-0.6b) -ported to transcribe.cpp. A 0.6B-parameter FastConformer-Large encoder with a -linear CTC head — the simplest decoder in the parakeet family and therefore -the fastest. + +Upstream: [`nvidia/parakeet-ctc-0.6b`](https://huggingface.co/nvidia/parakeet-ctc-0.6b) at [`ad09ba1`](https://huggingface.co/nvidia/parakeet-ctc-0.6b/commit/ad09ba1). + +Offline English speech-to-text with greedy CTC decoding. A FastConformer-Large encoder with a linear CTC head — the simplest and fastest decoder in the parakeet family. Output is lowercase, no punctuation. Not a streaming model and does not translate. + ## What it's for @@ -19,22 +20,39 @@ on this codebase. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-ctc-0.6b) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`ad09ba1`](https://huggingface.co/nvidia/parakeet-ctc-0.6b/commit/ad09ba1cc62743fbc9814de5d2016fca9096485a), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`ad09ba1`](https://huggingface.co/nvidia/parakeet-ctc-0.6b/commit/ad09ba1), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-ctc-0.6b-F32.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-F32.gguf) | 2.44 GB | 1.87% | -| F16 | [parakeet-ctc-0.6b-F16.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-F16.gguf) | 1.22 GB | 1.87% | -| Q8_0 | [parakeet-ctc-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q8_0.gguf) | 722 MB | 1.87% | -| Q6_K | [parakeet-ctc-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q6_K.gguf) | 594 MB | 1.84% | -| Q5_K_M | [parakeet-ctc-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q5_K_M.gguf) | 533 MB | 1.87% | -| Q4_K_M | [parakeet-ctc-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q4_K_M.gguf) | 469 MB | 1.90% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy CTC decoding and no external LM. F32 reference baseline: 1.87%. NVIDIA's self-reported number on the same split is 1.87% (from the [HF model card](https://huggingface.co/nvidia/parakeet-ctc-0.6b)). +| F32 | [parakeet-ctc-0.6b-F32.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-F32.gguf) | 2.44 GB | 1.87% | +| F16 | [parakeet-ctc-0.6b-F16.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-F16.gguf) | 1.22 GB | 1.87% | +| Q8_0 | [parakeet-ctc-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q8_0.gguf) | 722 MB | 1.87% | +| Q6_K | [parakeet-ctc-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q6_K.gguf) | 594 MB | 1.84% | +| Q5_K_M | [parakeet-ctc-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q5_K_M.gguf) | 533 MB | 1.87% | +| Q4_K_M | [parakeet-ctc-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-0.6b-gguf/resolve/main/parakeet-ctc-0.6b-Q4_K_M.gguf) | 469 MB | 1.90% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy CTC decoding, no external LM. F32 reference baseline: 1.87%. NVIDIA's +self-reported number on the same split is 1.87%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 5.53% | + ## Quick Start @@ -55,43 +73,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 58 ms (191×) | 59 ms (185×) | -| Metal | dots (35.3s) | 143 ms (246×) | 145 ms (244×) | -| CPU | jfk (11.0s) | 356 ms (31×) | 298 ms (37×) | -| CPU | dots (35.3s) | 1.19 s (30×) | 1.00 s (35×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `a6c097e`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 48 ms (227.64×) | 48 ms (226.86×) | +| Metal | dots (35.3s) | 111 ms (317.21×) | 113 ms (311.61×) | +| CPU | jfk (11.0s) | 270 ms (40.66×) | 294 ms (37.48×) | +| CPU | dots (35.3s) | 921 ms (38.35×) | 987 ms (35.80×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 520 ms (21×) | 537 ms (20×) | -| Vulkan | dots (35.3s) | 1.50 s (24×) | 1.50 s (24×) | -| CPU | jfk (11.0s) | 1.07 s (10×) | 863 ms (13×) | -| CPU | dots (35.3s) | 3.67 s (10×) | 3.14 s (11×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 418 ms (26.29×) | 427 ms (25.77×) | +| Vulkan | dots (35.3s) | 1.14 s (30.91×) | 1.16 s (30.57×) | +| CPU | jfk (11.0s) | 664 ms (16.58×) | 723 ms (15.21×) | +| CPU | dots (35.3s) | 2.57 s (13.74×) | 2.64 s (13.36×) | -Fedora 43, transcribe.cpp `57997dc`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-ctc-0.6b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-ctc-0.6b-publication +uv run scripts/bench/run.py --profile --models parakeet-ctc-0.6b ``` ## Numerical Validation diff --git a/docs/models/parakeet-ctc-1.1b.md b/docs/models/parakeet-ctc-1.1b.md index 5a410e88..1306773b 100644 --- a/docs/models/parakeet-ctc-1.1b.md +++ b/docs/models/parakeet-ctc-1.1b.md @@ -1,8 +1,10 @@ # Parakeet CTC 1.1B -NVIDIA's [`nvidia/parakeet-ctc-1.1b`](https://huggingface.co/nvidia/parakeet-ctc-1.1b) -ported to transcribe.cpp. A 1.1B-parameter FastConformer-XL encoder with a -linear CTC head. + +Upstream: [`nvidia/parakeet-ctc-1.1b`](https://huggingface.co/nvidia/parakeet-ctc-1.1b) at [`a707e81`](https://huggingface.co/nvidia/parakeet-ctc-1.1b/commit/a707e81). + +Offline English speech-to-text with greedy CTC decoding. A FastConformer-XL encoder with a linear CTC head. Output is lowercase, no punctuation. Not a streaming model and does not translate. + ## What it's for @@ -18,22 +20,39 @@ LibriSpeech test-clean over the 0.6B sibling. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-ctc-1.1b) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`a707e81`](https://huggingface.co/nvidia/parakeet-ctc-1.1b/commit/a707e818195cb97c8f7da2fc36b221a29f69a5db), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`a707e81`](https://huggingface.co/nvidia/parakeet-ctc-1.1b/commit/a707e81), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-ctc-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-F32.gguf) | 4.25 GB | 1.85% | -| F16 | [parakeet-ctc-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-F16.gguf) | 2.13 GB | 1.85% | -| Q8_0 | [parakeet-ctc-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q8_0.gguf) | 1.26 GB | 1.85% | -| Q6_K | [parakeet-ctc-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q6_K.gguf) | 1.04 GB | 1.85% | -| Q5_K_M | [parakeet-ctc-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q5_K_M.gguf) | 929 MB | 1.84% | -| Q4_K_M | [parakeet-ctc-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q4_K_M.gguf) | 818 MB | 1.90% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy CTC decoding and no external LM. F32 reference baseline: 1.85%. NVIDIA's self-reported number on the same split is 1.83% (from the [HF model card](https://huggingface.co/nvidia/parakeet-ctc-1.1b)). +| F32 | [parakeet-ctc-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-F32.gguf) | 4.25 GB | 1.85% | +| F16 | [parakeet-ctc-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-F16.gguf) | 2.13 GB | 1.85% | +| Q8_0 | [parakeet-ctc-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q8_0.gguf) | 1.26 GB | 1.85% | +| Q6_K | [parakeet-ctc-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q6_K.gguf) | 1.04 GB | 1.85% | +| Q5_K_M | [parakeet-ctc-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q5_K_M.gguf) | 929 MB | 1.84% | +| Q4_K_M | [parakeet-ctc-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-ctc-1.1b-gguf/resolve/main/parakeet-ctc-1.1b-Q4_K_M.gguf) | 818 MB | 1.90% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy CTC decoding, no external LM. F32 reference baseline: 1.85%. NVIDIA's +self-reported number on the same split is 1.83%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 5.61% | + ## Quick Start @@ -54,43 +73,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 91 ms (121×) | 93 ms (118×) | -| Metal | dots (35.3s) | 224 ms (158×) | 224 ms (158×) | -| CPU | jfk (11.0s) | 602 ms (18×) | 501 ms (22×) | -| CPU | dots (35.3s) | 2.04 s (17×) | 1.70 s (21×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `a6c097e`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 79 ms (140.04×) | 80 ms (136.77×) | +| Metal | dots (35.3s) | 177 ms (200.12×) | 180 ms (195.92×) | +| CPU | jfk (11.0s) | 458 ms (24.00×) | 502 ms (21.92×) | +| CPU | dots (35.3s) | 1.58 s (22.32×) | 1.70 s (20.80×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 825 ms (13×) | 822 ms (13×) | -| Vulkan | dots (35.3s) | 2.34 s (15×) | 2.33 s (15×) | -| CPU | jfk (11.0s) | 1.75 s (6×) | 1.38 s (8×) | -| CPU | dots (35.3s) | 6.08 s (6×) | 5.12 s (7×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 691 ms (15.92×) | 711 ms (15.47×) | +| Vulkan | dots (35.3s) | 2.00 s (17.65×) | 2.02 s (17.49×) | +| CPU | jfk (11.0s) | 1.10 s (9.97×) | 1.21 s (9.06×) | +| CPU | dots (35.3s) | 4.48 s (7.88×) | 4.63 s (7.63×) | -Fedora 43, transcribe.cpp `57997dc`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-ctc-1.1b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-ctc-1.1b-publication +uv run scripts/bench/run.py --profile --models parakeet-ctc-1.1b ``` ## Numerical Validation diff --git a/docs/models/parakeet-primeline.md b/docs/models/parakeet-primeline.md index 4c3ae0f3..cccae6b0 100644 --- a/docs/models/parakeet-primeline.md +++ b/docs/models/parakeet-primeline.md @@ -1,9 +1,16 @@ # Parakeet primeLine (German-tuned) -primeLine's [`primeline/parakeet-primeline`](https://huggingface.co/primeline/parakeet-primeline) -ported to transcribe.cpp. A German fine-tune of NVIDIA's -[`parakeet-tdt-0.6b-v3`](parakeet-tdt-0.6b-v3.md): a 0.6B-parameter -FastConformer encoder with a TDT/RNNT transducer decoder. + +Upstream: [`primeline/parakeet-primeline`](https://huggingface.co/primeline/parakeet-primeline) at [`3f1a9bc`](https://huggingface.co/primeline/parakeet-primeline/commit/3f1a9bc). + +primeLine's German fine-tune of NVIDIA's parakeet-tdt-0.6b-v3. A +FastConformer encoder with a TDT/RNNT transducer decoder, taking 16 kHz +mono WAV and producing a punctuated, cased transcript with optional +token-level timestamps. Tuned for German, but the fine-tune did not +collapse the base model's multilingual ability: it still transcribes the +other 24 v3 languages with correct per-language casing and punctuation. +Not a streaming model and does not translate. + ## What it's for @@ -18,27 +25,83 @@ correct per-language casing and punctuation, so all 25 v3 languages remain usable. Pick this variant when German is your primary workload and `parakeet-tdt-0.6b-v3` when it is not. -Licensed CC-BY-4.0. Ported from upstream commit -[`3f1a9bc`](https://huggingface.co/primeline/parakeet-primeline/commit/3f1a9bcb611dfeda53fe74fe5f1a3d5701e8023e), -pinned 2026-08-16. + +Licensed CC-BY-4.0. Ported from upstream commit [`3f1a9bc`](https://huggingface.co/primeline/parakeet-primeline/commit/3f1a9bc), pinned 2026-08-16. Validated against the NeMo reference at transcribe.cpp commit [`856d7c1`](https://github.com/handy-computer/transcribe.cpp/tree/856d7c1) on 2026-08-16. + ## Download -| Quantization | Download | Size | WER (FLEURS de test) | + +| Quantization | Download | Size | WER (FLEURS de) | | --- | --- | ---: | ---: | -| F32 | [parakeet-primeline-F32.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-F32.gguf) | 2.51 GB | 6.00% | -| F16 | [parakeet-primeline-F16.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-F16.gguf) | 1.26 GB | 6.00% | -| Q8_0 | [parakeet-primeline-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q8_0.gguf) | 740 MB | 6.00% | -| Q6_K | [parakeet-primeline-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q6_K.gguf) | 610 MB | 5.96% | -| Q5_K_M | [parakeet-primeline-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q5_K_M.gguf) | 549 MB | 5.99% | -| Q4_K_M | [parakeet-primeline-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q4_K_M.gguf) | 485 MB | 5.98% | - -WER is measured on the full FLEURS German test split (862 utterances) -with greedy transducer decoding and no external LM. The reference -baseline, primeLine's own NeMo checkpoint over the identical manifest, -is 5.98%. The quant matrix spans 0.04pp with no monotonic degradation; -Q4_K_M scores marginally better than F32 (5.9845% vs 5.9952%), which is -noise, not an improvement. Any preset is safe to ship. +| F32 | [parakeet-primeline-F32.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-F32.gguf) | 2.51 GB | 6.00% | +| F16 | [parakeet-primeline-F16.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-F16.gguf) | 1.26 GB | 6.00% | +| Q8_0 | [parakeet-primeline-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q8_0.gguf) | 740 MB | 5.98% | +| Q6_K | [parakeet-primeline-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q6_K.gguf) | 610 MB | 5.96% | +| Q5_K_M | [parakeet-primeline-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q5_K_M.gguf) | 549 MB | 5.99% | +| Q4_K_M | [parakeet-primeline-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-primeline-gguf/resolve/main/parakeet-primeline-Q4_K_M.gguf) | 485 MB | 5.98% | + + + +WER on the full FLEURS de split (862 utterances), batch sizes 1 and 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy transducer decoding, no external LM. + +primeLine's published figures (2.95% average over Tuda-De, Multilingual +LibriSpeech, and Common Voice 19.0) are on different corpora and are not +comparable to these numbers. As a like-for-like baseline we ran primeLine's +own NeMo checkpoint over the identical manifest: **5.98% WER**. The C++ +numbers above match that reference within bootstrap-CI noise, and the quant +spread is 0.04pp end to end with no monotonic degradation. + +Orthography note: this checkpoint writes Swiss `ss` forms (`grosse`, +`heisst`) almost everywhere instead of `ß`, which appears just 5 times +across the 862 hypotheses. The upstream SentencePiece vocabulary carries +only 4 pieces containing `ß` against 58 containing `ss`, so this is a +property of the v3-family tokenizer, not of the port — the NeMo reference +produces the same spellings on the same utterances. FLEURS references use +`ß` throughout, which costs roughly 1.05pp: folding `ß`→`ss` on both sides +gives 4.92% for the reference and 4.94% for F32. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| bg | WER | 16.33% | +| cs | WER | 15.05% | +| da | WER | 20.52% | +| el | WER | 34.76% | +| en | WER | 4.82% | +| es | WER | 3.85% | +| et | WER | 17.17% | +| fi | WER | 13.39% | +| fr | WER | 6.35% | +| hr | WER | 13.68% | +| hu | WER | 17.52% | +| it | WER | 3.17% | +| lt | WER | 23.08% | +| lv | WER | 28.83% | +| mt | WER | 24.74% | +| nl | WER | 8.49% | +| pl | WER | 8.19% | +| pt | WER | 5.17% | +| ro | WER | 13.80% | +| ru | WER | 7.81% | +| sk | WER | 12.36% | +| sl | WER | 51.07% | +| sv | WER | 16.42% | +| uk | WER | 8.11% | + +**LibriSpeech test-clean** + +| Language | Metric | F32 | F16 | Q8_0 | Q6_K | Q5_K_M | Q4_K_M | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| en | WER | 2.20% | 2.19% | 2.20% | 2.19% | 2.20% | 2.26% | + primeLine's published 2.95% average is over Tuda-De, Multilingual LibriSpeech, and Common Voice 19.0. Those corpora are not in this repo's diff --git a/docs/models/parakeet-rnnt-0.6b.md b/docs/models/parakeet-rnnt-0.6b.md index a3cf5f17..ff1c3ac7 100644 --- a/docs/models/parakeet-rnnt-0.6b.md +++ b/docs/models/parakeet-rnnt-0.6b.md @@ -1,8 +1,10 @@ # Parakeet RNN-T 0.6B -NVIDIA's [`nvidia/parakeet-rnnt-0.6b`](https://huggingface.co/nvidia/parakeet-rnnt-0.6b) -ported to transcribe.cpp. A 0.6B-parameter FastConformer-Large encoder with a -classic RNN-T transducer decoder (predictor + joint, no duration head). + +Upstream: [`nvidia/parakeet-rnnt-0.6b`](https://huggingface.co/nvidia/parakeet-rnnt-0.6b) at [`c0c1f09`](https://huggingface.co/nvidia/parakeet-rnnt-0.6b/commit/c0c1f09). + +Offline English speech-to-text with greedy RNN-T decoding. A FastConformer-Large encoder with an RNN-T transducer decoder. Output is lowercase, no punctuation. Not a streaming model and does not translate. + ## What it's for @@ -20,22 +22,39 @@ than the CTC variant at the same encoder size. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-rnnt-0.6b) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`c0c1f09`](https://huggingface.co/nvidia/parakeet-rnnt-0.6b/commit/c0c1f09fdc3f18b0b2ddbeafd5d6684f1b38078f), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`c0c1f09`](https://huggingface.co/nvidia/parakeet-rnnt-0.6b/commit/c0c1f09), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-rnnt-0.6b-F32.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-F32.gguf) | 2.47 GB | 1.62% | -| F16 | [parakeet-rnnt-0.6b-F16.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-F16.gguf) | 1.24 GB | 1.62% | -| Q8_0 | [parakeet-rnnt-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q8_0.gguf) | 730 MB | 1.62% | -| Q6_K | [parakeet-rnnt-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q6_K.gguf) | 601 MB | 1.62% | -| Q5_K_M | [parakeet-rnnt-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q5_K_M.gguf) | 540 MB | 1.62% | -| Q4_K_M | [parakeet-rnnt-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q4_K_M.gguf) | 476 MB | 1.59% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy RNN-T decoding and no external LM. F32 reference baseline: 1.62%. NVIDIA's self-reported number on the same split is 1.63% (from the [HF model card](https://huggingface.co/nvidia/parakeet-rnnt-0.6b)). +| F32 | [parakeet-rnnt-0.6b-F32.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-F32.gguf) | 2.47 GB | 1.62% | +| F16 | [parakeet-rnnt-0.6b-F16.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-F16.gguf) | 1.24 GB | 1.62% | +| Q8_0 | [parakeet-rnnt-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q8_0.gguf) | 730 MB | 1.62% | +| Q6_K | [parakeet-rnnt-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q6_K.gguf) | 601 MB | 1.62% | +| Q5_K_M | [parakeet-rnnt-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q5_K_M.gguf) | 540 MB | 1.62% | +| Q4_K_M | [parakeet-rnnt-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-0.6b-gguf/resolve/main/parakeet-rnnt-0.6b-Q4_K_M.gguf) | 476 MB | 1.66% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy RNN-T decoding, no external LM. F32 reference baseline: 1.62%. NVIDIA's +self-reported number on the same split is 1.63%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.57% | + ## Quick Start @@ -56,43 +75,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 64 ms (173×) | 65 ms (170×) | -| Metal | dots (35.3s) | 178 ms (198×) | 181 ms (196×) | -| CPU | jfk (11.0s) | 360 ms (31×) | 302 ms (36×) | -| CPU | dots (35.3s) | 1.22 s (29×) | 1.03 s (34×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `12f1076`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 55 ms (201.01×) | 56 ms (198.17×) | +| Metal | dots (35.3s) | 138 ms (255.29×) | 142 ms (248.43×) | +| CPU | jfk (11.0s) | 281 ms (39.19×) | 301 ms (36.52×) | +| CPU | dots (35.3s) | 977 ms (36.16×) | 1.04 s (34.02×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 742 ms (15×) | 754 ms (15×) | -| Vulkan | dots (35.3s) | 2.55 s (14×) | 2.59 s (14×) | -| CPU | jfk (11.0s) | 1.24 s (9×) | 1.07 s (10×) | -| CPU | dots (35.3s) | 4.71 s (7×) | 4.14 s (9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 425 ms (25.87×) | 431 ms (25.51×) | +| Vulkan | dots (35.3s) | 1.25 s (28.33×) | 1.27 s (27.86×) | +| CPU | jfk (11.0s) | 675 ms (16.30×) | 720 ms (15.29×) | +| CPU | dots (35.3s) | 2.64 s (13.39×) | 2.71 s (13.05×) | -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-rnnt-0.6b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-rnnt-0.6b-publication +uv run scripts/bench/run.py --profile --models parakeet-rnnt-0.6b ``` ## Numerical Validation diff --git a/docs/models/parakeet-rnnt-1.1b.md b/docs/models/parakeet-rnnt-1.1b.md index db6c2048..51ebd5a6 100644 --- a/docs/models/parakeet-rnnt-1.1b.md +++ b/docs/models/parakeet-rnnt-1.1b.md @@ -1,8 +1,10 @@ # Parakeet RNN-T 1.1B -NVIDIA's [`nvidia/parakeet-rnnt-1.1b`](https://huggingface.co/nvidia/parakeet-rnnt-1.1b) -ported to transcribe.cpp. A 1.1B-parameter FastConformer-XL encoder with a -classic RNN-T transducer decoder (predictor + joint, no duration head). + +Upstream: [`nvidia/parakeet-rnnt-1.1b`](https://huggingface.co/nvidia/parakeet-rnnt-1.1b) at [`a07b19e`](https://huggingface.co/nvidia/parakeet-rnnt-1.1b/commit/a07b19e). + +Offline English speech-to-text with greedy RNN-T decoding. A FastConformer-XL encoder with an RNN-T transducer decoder. Output is lowercase, no punctuation. Not a streaming model and does not translate. + ## What it's for @@ -18,22 +20,39 @@ On our LibriSpeech test-clean runs `parakeet-tdt-1.1b` edges it out (1.38% vs See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-rnnt-1.1b) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`a07b19e`](https://huggingface.co/nvidia/parakeet-rnnt-1.1b/commit/a07b19e9), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`a07b19e`](https://huggingface.co/nvidia/parakeet-rnnt-1.1b/commit/a07b19e), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-rnnt-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-F32.gguf) | 4.28 GB | 1.45% | -| F16 | [parakeet-rnnt-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-F16.gguf) | 2.15 GB | 1.45% | -| Q8_0 | [parakeet-rnnt-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q8_0.gguf) | 1.27 GB | 1.46% | -| Q6_K | [parakeet-rnnt-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q6_K.gguf) | 1.04 GB | 1.43% | -| Q5_K_M | [parakeet-rnnt-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q5_K_M.gguf) | 936 MB | 1.43% | -| Q4_K_M | [parakeet-rnnt-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q4_K_M.gguf) | 825 MB | 1.41% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy RNN-T decoding and no external LM. F32 reference baseline: 1.45%. NVIDIA's self-reported number on the same split is 1.46% (from the [HF model card](https://huggingface.co/nvidia/parakeet-rnnt-1.1b)). +| F32 | [parakeet-rnnt-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-F32.gguf) | 4.28 GB | 1.45% | +| F16 | [parakeet-rnnt-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-F16.gguf) | 2.15 GB | 1.45% | +| Q8_0 | [parakeet-rnnt-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q8_0.gguf) | 1.27 GB | 1.46% | +| Q6_K | [parakeet-rnnt-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q6_K.gguf) | 1.04 GB | 1.43% | +| Q5_K_M | [parakeet-rnnt-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q5_K_M.gguf) | 936 MB | 1.43% | +| Q4_K_M | [parakeet-rnnt-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-rnnt-1.1b-gguf/resolve/main/parakeet-rnnt-1.1b-Q4_K_M.gguf) | 825 MB | 1.41% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy RNN-T decoding, no external LM. F32 reference baseline: 1.45%. NVIDIA's +self-reported number on the same split is 1.46%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.45% | + ## Quick Start @@ -54,43 +73,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 96 ms (114×) | 97 ms (114×) | -| Metal | dots (35.3s) | 258 ms (137×) | 265 ms (133×) | -| CPU | jfk (11.0s) | 606 ms (18×) | 506 ms (22×) | -| CPU | dots (35.3s) | 2.05 s (17×) | 1.72 s (20×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `12f1076`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 84 ms (131.63×) | 86 ms (128.14×) | +| Metal | dots (35.3s) | 201 ms (175.51×) | 207 ms (170.62×) | +| CPU | jfk (11.0s) | 492 ms (22.37×) | 516 ms (21.34×) | +| CPU | dots (35.3s) | 1.64 s (21.53×) | 1.86 s (19.04×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 1.02 s (11×) | 1.04 s (11×) | -| Vulkan | dots (35.3s) | 3.35 s (11×) | 3.31 s (11×) | -| CPU | jfk (11.0s) | 1.93 s (6×) | 1.58 s (7×) | -| CPU | dots (35.3s) | 7.12 s (5×) | 6.18 s (6×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 701 ms (15.70×) | 710 ms (15.50×) | +| Vulkan | dots (35.3s) | 2.15 s (16.47×) | 2.18 s (16.20×) | +| CPU | jfk (11.0s) | 1.12 s (9.80×) | 1.21 s (9.08×) | +| CPU | dots (35.3s) | 4.79 s (7.38×) | 4.73 s (7.48×) | -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-rnnt-1.1b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-rnnt-1.1b-publication +uv run scripts/bench/run.py --profile --models parakeet-rnnt-1.1b ``` ## Numerical Validation diff --git a/docs/models/parakeet-tdt-0.6b-v2.md b/docs/models/parakeet-tdt-0.6b-v2.md index 87ada6d3..399dcbc8 100644 --- a/docs/models/parakeet-tdt-0.6b-v2.md +++ b/docs/models/parakeet-tdt-0.6b-v2.md @@ -1,8 +1,13 @@ # Parakeet TDT 0.6B v2 -NVIDIA's [`nvidia/parakeet-tdt-0.6b-v2`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) -ported to transcribe.cpp. A 0.6B-parameter Conformer encoder with a TDT/RNNT -transducer decoder. + +Upstream: [`nvidia/parakeet-tdt-0.6b-v2`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) at [`1b149a3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2/commit/1b149a3). + +Offline English speech-to-text. A Conformer encoder with a TDT/RNNT transducer +decoder. Takes a 16 kHz mono WAV and produces a transcript with optional +token-level timestamps. Not a streaming model; no multilingual capability (see +v3 for that). + ## What it's for @@ -14,26 +19,40 @@ see v3. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`1b149a3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2/commit/1b149a3589351c96ddb101709fe7dd9c7069572f), -pinned 2026-04-15. + +Licensed CC-BY-4.0. Ported from upstream commit [`1b149a3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2/commit/1b149a3), pinned 2026-04-15. Validated against the NeMo reference at transcribe.cpp commit [`bf0d0b7`](https://github.com/handy-computer/transcribe.cpp/tree/bf0d0b7) on 2026-04-18. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-tdt-0.6b-v2-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-F32.gguf) | 2.47 GB | 1.68% | -| F16 | [parakeet-tdt-0.6b-v2-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-F16.gguf) | 1.24 GB | 1.68% | -| Q8_0 | [parakeet-tdt-0.6b-v2-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q8_0.gguf) | 730 MB | 1.69% | -| Q6_K | [parakeet-tdt-0.6b-v2-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q6_K.gguf) | 608 MB | 1.70% | -| Q5_K_M | [parakeet-tdt-0.6b-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q5_K_M.gguf) | 547 MB | 1.70% | -| Q4_K_M | [parakeet-tdt-0.6b-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q4_K_M.gguf) | 483 MB | 1.72% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy transducer decoding and no external LM. F32 reference baseline: -1.68%. NVIDIA's self-reported number on the same split is 1.69% (from the -[HF model card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2)), so our -F32 and Q8_0 ports match the upstream reference within rounding. +| F32 | [parakeet-tdt-0.6b-v2-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-F32.gguf) | 2.47 GB | 1.68% | +| F16 | [parakeet-tdt-0.6b-v2-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-F16.gguf) | 1.24 GB | 1.68% | +| Q8_0 | [parakeet-tdt-0.6b-v2-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q8_0.gguf) | 730 MB | 1.69% | +| Q6_K | [parakeet-tdt-0.6b-v2-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q6_K.gguf) | 600 MB | 1.70% | +| Q5_K_M | [parakeet-tdt-0.6b-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q5_K_M.gguf) | 539 MB | 1.70% | +| Q4_K_M | [parakeet-tdt-0.6b-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v2-gguf/resolve/main/parakeet-tdt-0.6b-v2-Q4_K_M.gguf) | 475 MB | 1.72% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy transducer decoding, no external LM. F32 reference baseline: 1.68%. NVIDIA's +self-reported number on the same split is 1.69%, so the F32 and Q8_0 ports match the +upstream reference within rounding. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.11% | + ## Quick Start @@ -54,51 +73,47 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 68 ms (163×) | 67 ms (163×) | -| Metal | dots (35.3s) | 189 ms (187×) | 193 ms (183×) | -| CPU | jfk (11.0s) | 371 ms (30×) | 312 ms (35×) | -| CPU | dots (35.3s) | 1.26 s (28×) | 1.07 s (33×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 55 ms (199.95×) | 56 ms (196.11×) | +| Metal | dots (35.3s) | 142 ms (248.28×) | 145 ms (243.28×) | +| CPU | jfk (11.0s) | 278 ms (39.64×) | 318 ms (34.64×) | +| CPU | dots (35.3s) | 987 ms (35.80×) | 1.09 s (32.53×) | -macOS 26.4.1, transcribe.cpp `12f1076`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 673 ms (16×) | 685 ms (16×) | -| Vulkan | dots (35.3s) | 2.45 s (14×) | 2.44 s (14×) | -| CPU | jfk (11.0s) | 1.23 s (9×) | 1.05 s (10×) | -| CPU | dots (35.3s) | 4.75 s (7×) | 4.14 s (9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 416 ms (26.45×) | 418 ms (26.34×) | +| Vulkan | dots (35.3s) | 1.24 s (28.42×) | 1.26 s (28.09×) | +| CPU | jfk (11.0s) | 696 ms (15.81×) | 749 ms (14.69×) | +| CPU | dots (35.3s) | 2.77 s (12.73×) | 2.83 s (12.48×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-tdt-0.6b-v2 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-tdt-0.6b-v2-publication +uv run scripts/bench/run.py --profile --models parakeet-tdt-0.6b-v2 ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against NeMo on `samples/jfk.wav`. All 18 checkpointed tensors fall within family tolerance, and the final -transcript matches the NeMo reference verbatim. Last validated at commit -[`bf0d0b7`](https://github.com/handy-computer/transcribe.cpp/tree/bf0d0b7). +transcript matches the NeMo reference verbatim. | Field | Value | | --- | --- | @@ -107,21 +122,6 @@ transcript matches the NeMo reference verbatim. Last validated at commit | Manifest | `tests/golden/parakeet/parakeet-tdt-0.6b-v2.manifest.json` | | Command | `uv run scripts/validate.py compare --family parakeet` | -Selected tensors: - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.mel.in` | `5.189e+00` | `1.639e-03` | fp64 vs fp32 STFT precision gap | -| `enc.pre_encode.out` | `2.011e+03` | `6.590e+01` | Mel gap amplified through pre-encoder | -| `enc.block.0.out` | `1.016e+03` | `1.396e+01` | Early encoder, still amplified | -| `enc.block.12.out` | `1.040e+03` | `1.330e+01` | Mid-encoder | -| `enc.block.23.out` | `4.392e-02` | `1.169e-03` | Converged by final block | -| `enc.final` | `4.392e-02` | `1.169e-03` | Final encoder output | -| `dec.enc_out` | `4.392e-02` | `1.169e-03` | Decoder input from encoder | -| `dec.embed.0` | `0.000e+00` | `0.000e+00` | Exact match | -| `dec.lstm.*` | `<= 1.788e-07` | near zero | fp32 round-off on first step | -| `dec.joint.0` | `7.033e+01` | `6.956e+01` | Joint projection over encoder drift | - The expected divergence is in the frontend: C++ runs the STFT in fp64 where NeMo runs fp32. The gap enters at the mel spectrogram, is amplified through the pre-encoder and early Conformer blocks, and attenuates to near-zero by the diff --git a/docs/models/parakeet-tdt-0.6b-v3.md b/docs/models/parakeet-tdt-0.6b-v3.md index 567fd42a..c81fd1e2 100644 --- a/docs/models/parakeet-tdt-0.6b-v3.md +++ b/docs/models/parakeet-tdt-0.6b-v3.md @@ -1,8 +1,13 @@ # Parakeet TDT 0.6B v3 -NVIDIA's [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) -ported to transcribe.cpp. A 0.6B-parameter Conformer encoder with a TDT/RNNT -transducer decoder. + +Upstream: [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) at [`6d590f7`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3/commit/6d590f7). + +Offline multilingual speech-to-text covering 25 European languages. A +Conformer encoder with a TDT/RNNT transducer decoder. Takes a 16 kHz mono +WAV and produces a transcript with optional token-level timestamps. Not a +streaming model and does not translate. + ## What it's for @@ -17,25 +22,63 @@ Spanish, Swedish, Ukrainian. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`6d590f7`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3/commit/6d590f77001d318fb17a0b5bf7ee329a91b52598), -pinned 2026-04-16. + +Licensed CC-BY-4.0. Ported from upstream commit [`6d590f7`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3/commit/6d590f7), pinned 2026-04-16. Validated against the NeMo reference at transcribe.cpp commit [`bf0d0b7`](https://github.com/handy-computer/transcribe.cpp/tree/bf0d0b7) on 2026-04-18. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-tdt-0.6b-v3-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-F32.gguf) | 2.51 GB | 1.95% | -| F16 | [parakeet-tdt-0.6b-v3-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-F16.gguf) | 1.26 GB | 1.95% | -| Q8_0 | [parakeet-tdt-0.6b-v3-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q8_0.gguf) | 740 MB | 1.94% | -| Q6_K | [parakeet-tdt-0.6b-v3-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q6_K.gguf) | 627 MB | 1.93% | -| Q5_K_M | [parakeet-tdt-0.6b-v3-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q5_K_M.gguf) | 565 MB | 1.92% | -| Q4_K_M | [parakeet-tdt-0.6b-v3-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q4_K_M.gguf) | 502 MB | 1.98% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy transducer decoding and no external LM. F32 reference baseline: -1.95%. NVIDIA's self-reported number on the same split is 1.93% (from the -[HF model card](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3)) +| F32 | [parakeet-tdt-0.6b-v3-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-F32.gguf) | 2.51 GB | 1.95% | +| F16 | [parakeet-tdt-0.6b-v3-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-F16.gguf) | 1.26 GB | 1.95% | +| Q8_0 | [parakeet-tdt-0.6b-v3-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q8_0.gguf) | 740 MB | 1.94% | +| Q6_K | [parakeet-tdt-0.6b-v3-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q6_K.gguf) | 610 MB | 1.93% | +| Q5_K_M | [parakeet-tdt-0.6b-v3-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q5_K_M.gguf) | 549 MB | 1.92% | +| Q4_K_M | [parakeet-tdt-0.6b-v3-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-0.6b-v3-gguf/resolve/main/parakeet-tdt-0.6b-v3-Q4_K_M.gguf) | 485 MB | 1.98% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy transducer decoding, no external LM. F32 reference baseline: 1.95%. NVIDIA's +self-reported number on the same split is 1.93%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| bg | WER | 12.81% | +| cs | WER | 12.31% | +| da | WER | 18.64% | +| de | WER | 5.24% | +| el | WER | 35.33% | +| en | WER | 4.83% | +| es | WER | 3.65% | +| et | WER | 17.96% | +| fi | WER | 13.30% | +| fr | WER | 5.30% | +| hr | WER | 12.59% | +| hu | WER | 16.06% | +| it | WER | 3.02% | +| lt | WER | 22.20% | +| lv | WER | 23.77% | +| mt | WER | 20.63% | +| nl | WER | 7.66% | +| pl | WER | 7.37% | +| pt | WER | 4.96% | +| ro | WER | 12.62% | +| ru | WER | 6.54% | +| sk | WER | 10.19% | +| sl | WER | 24.30% | +| sv | WER | 15.25% | +| uk | WER | 6.84% | + ## Quick Start @@ -56,51 +99,47 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 74 ms (149×) | 76 ms (146×) | -| Metal | dots (35.3s) | 231 ms (153×) | 230 ms (153×) | -| CPU | jfk (11.0s) | 381 ms (29×) | 323 ms (34×) | -| CPU | dots (35.3s) | 1.29 s (27×) | 1.11 s (32×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 60 ms (181.74×) | 62 ms (178.45×) | +| Metal | dots (35.3s) | 164 ms (215.45×) | 167 ms (212.12×) | +| CPU | jfk (11.0s) | 286 ms (38.48×) | 310 ms (35.53×) | +| CPU | dots (35.3s) | 1.00 s (35.16×) | 1.08 s (32.71×) | -macOS 26.4.1, transcribe.cpp `12f1076`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 858 ms (13×) | 868 ms (13×) | -| Vulkan | dots (35.3s) | 3.07 s (12×) | 3.10 s (11×) | -| CPU | jfk (11.0s) | 1.41 s (8×) | 1.22 s (9×) | -| CPU | dots (35.3s) | 5.34 s (7×) | 4.78 s (7×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 453 ms (24.29×) | 458 ms (24.01×) | +| Vulkan | dots (35.3s) | 1.37 s (25.84×) | 1.39 s (25.50×) | +| CPU | jfk (11.0s) | 729 ms (15.09×) | 794 ms (13.86×) | +| CPU | dots (35.3s) | 2.89 s (12.22×) | 2.97 s (11.89×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-tdt-0.6b-v3 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-tdt-0.6b-v3-publication +uv run scripts/bench/run.py --profile --models parakeet-tdt-0.6b-v3 ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against NeMo on `samples/jfk.wav`. All 18 checkpointed tensors fall within family tolerance, and the final -transcript matches the NeMo reference verbatim. Last validated at commit -[`bf0d0b7`](https://github.com/handy-computer/transcribe.cpp/tree/bf0d0b7). +transcript matches the NeMo reference verbatim. | Field | Value | | --- | --- | @@ -109,21 +148,6 @@ transcript matches the NeMo reference verbatim. Last validated at commit | Manifest | `tests/golden/parakeet/parakeet-tdt-0.6b-v3.manifest.json` | | Command | `uv run scripts/validate.py compare --family parakeet --variant parakeet-tdt-0.6b-v3` | -Selected tensors: - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.mel.in` | `5.189e+00` | `1.639e-03` | fp64 vs fp32 STFT precision gap | -| `enc.pre_encode.out` | `6.940e+03` | `2.438e+02` | Mel gap amplified through pre-encoder (v3 amplifies more aggressively than v2) | -| `enc.block.0.out` | `1.296e+03` | `1.468e+01` | Early encoder, still amplified | -| `enc.block.12.out` | `1.285e+03` | `1.433e+01` | Mid-encoder | -| `enc.block.23.out` | `3.055e-02` | `3.040e-04` | Converged by final block | -| `enc.final` | `3.055e-02` | `3.040e-04` | Final encoder output | -| `dec.enc_out` | `3.055e-02` | `3.040e-04` | Decoder input from encoder | -| `dec.embed.0` | `0.000e+00` | `0.000e+00` | Exact match | -| `dec.lstm.*` | `<= 1.192e-07` | near zero | fp32 round-off on first step | -| `dec.joint.0` | `1.190e+01` | `1.098e+01` | Joint projection over encoder drift | - Same divergence profile as v2: C++ runs the STFT in fp64 where NeMo runs fp32. The gap enters at the mel spectrogram, is amplified through the pre-encoder and early Conformer blocks, and attenuates to near-zero by the final encoder diff --git a/docs/models/parakeet-tdt-1.1b.md b/docs/models/parakeet-tdt-1.1b.md index 261c683d..494869c0 100644 --- a/docs/models/parakeet-tdt-1.1b.md +++ b/docs/models/parakeet-tdt-1.1b.md @@ -1,8 +1,10 @@ # Parakeet TDT 1.1B -NVIDIA's [`nvidia/parakeet-tdt-1.1b`](https://huggingface.co/nvidia/parakeet-tdt-1.1b) -ported to transcribe.cpp. A 1.1B-parameter FastConformer-XL encoder with a -TDT/RNN-T transducer decoder (predictor + joint with duration head). + +Upstream: [`nvidia/parakeet-tdt-1.1b`](https://huggingface.co/nvidia/parakeet-tdt-1.1b) at [`53276c6`](https://huggingface.co/nvidia/parakeet-tdt-1.1b/commit/53276c6). + +Offline English speech-to-text. A FastConformer-XL encoder with a TDT/RNNT transducer decoder. Takes a 16 kHz mono WAV and produces a transcript with optional token-level timestamps. Not a streaming model and does not translate. + ## What it's for @@ -19,22 +21,39 @@ each step can advance more than one frame. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-tdt-1.1b) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`53276c6`](https://huggingface.co/nvidia/parakeet-tdt-1.1b/commit/53276c64), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`53276c6`](https://huggingface.co/nvidia/parakeet-tdt-1.1b/commit/53276c6), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-tdt-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-F32.gguf) | 4.28 GB | 1.39% | -| F16 | [parakeet-tdt-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-F16.gguf) | 2.15 GB | 1.39% | -| Q8_0 | [parakeet-tdt-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q8_0.gguf) | 1.27 GB | 1.38% | -| Q6_K | [parakeet-tdt-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q6_K.gguf) | 1.04 GB | 1.40% | -| Q5_K_M | [parakeet-tdt-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q5_K_M.gguf) | 936 MB | 1.39% | -| Q4_K_M | [parakeet-tdt-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q4_K_M.gguf) | 825 MB | 1.42% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy TDT decoding and no external LM. F32 reference baseline: 1.39%. NVIDIA's self-reported number on the same split is 1.39% (from the [HF model card](https://huggingface.co/nvidia/parakeet-tdt-1.1b)). +| F32 | [parakeet-tdt-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-F32.gguf) | 4.28 GB | 1.39% | +| F16 | [parakeet-tdt-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-F16.gguf) | 2.15 GB | 1.39% | +| Q8_0 | [parakeet-tdt-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q8_0.gguf) | 1.27 GB | 1.38% | +| Q6_K | [parakeet-tdt-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q6_K.gguf) | 1.04 GB | 1.40% | +| Q5_K_M | [parakeet-tdt-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q5_K_M.gguf) | 936 MB | 1.39% | +| Q4_K_M | [parakeet-tdt-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt-1.1b-gguf/resolve/main/parakeet-tdt-1.1b-Q4_K_M.gguf) | 825 MB | 1.42% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy TDT/RNN-T transducer decoding, no external LM. F32 reference baseline: 1.39%. +NVIDIA's self-reported number on the same split is 1.39%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.24% | + ## Quick Start @@ -55,43 +74,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 98 ms (112×) | 97 ms (113×) | -| Metal | dots (35.3s) | 254 ms (139×) | 255 ms (139×) | -| CPU | jfk (11.0s) | 608 ms (18×) | 505 ms (22×) | -| CPU | dots (35.3s) | 2.04 s (17×) | 1.73 s (20×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `12f1076`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 81 ms (135.38×) | 84 ms (130.98×) | +| Metal | dots (35.3s) | 196 ms (180.48×) | 201 ms (175.57×) | +| CPU | jfk (11.0s) | 675 ms (16.31×) | 518 ms (21.25×) | +| CPU | dots (35.3s) | 1.76 s (20.06×) | 1.76 s (20.06×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 966 ms (11×) | 942 ms (12×) | -| Vulkan | dots (35.3s) | 3.03 s (12×) | 3.00 s (12×) | -| CPU | jfk (11.0s) | 1.92 s (6×) | 1.49 s (7×) | -| CPU | dots (35.3s) | 6.82 s (5×) | 5.94 s (6×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 694 ms (15.86×) | 706 ms (15.58×) | +| Vulkan | dots (35.3s) | 2.07 s (17.03×) | 2.12 s (16.65×) | +| CPU | jfk (11.0s) | 1.12 s (9.82×) | 1.21 s (9.08×) | +| CPU | dots (35.3s) | 4.54 s (7.79×) | 4.66 s (7.57×) | -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-tdt-1.1b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-tdt-1.1b-publication +uv run scripts/bench/run.py --profile --models parakeet-tdt-1.1b ``` ## Numerical Validation diff --git a/docs/models/parakeet-tdt_ctc-1.1b.md b/docs/models/parakeet-tdt_ctc-1.1b.md index 175da9bb..89d675ef 100644 --- a/docs/models/parakeet-tdt_ctc-1.1b.md +++ b/docs/models/parakeet-tdt_ctc-1.1b.md @@ -1,9 +1,10 @@ # Parakeet TDT-CTC 1.1B -NVIDIA's [`nvidia/parakeet-tdt_ctc-1.1b`](https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b) -ported to transcribe.cpp. A hybrid 1.1B-parameter FastConformer-XL encoder -with both TDT and CTC heads sharing the same encoder; transcribe.cpp uses -the TDT head by default. + +Upstream: [`nvidia/parakeet-tdt_ctc-1.1b`](https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b) at [`675e786`](https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b/commit/675e786). + +Offline English speech-to-text with punctuation and capitalization. A FastConformer-XL encoder with a TDT/RNNT transducer decoder (the auxiliary CTC head from the upstream hybrid checkpoint is dropped at convert time). Not a streaming model and does not translate. + ## What it's for @@ -18,22 +19,39 @@ to duration-aware frame skipping. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`675e786`](https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b/commit/675e786), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`675e786`](https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b/commit/675e786), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-tdt_ctc-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-F32.gguf) | 4.28 GB | 1.87% | -| F16 | [parakeet-tdt_ctc-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-F16.gguf) | 2.15 GB | 1.87% | -| Q8_0 | [parakeet-tdt_ctc-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q8_0.gguf) | 1.27 GB | 1.87% | -| Q6_K | [parakeet-tdt_ctc-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q6_K.gguf) | 1.04 GB | 1.87% | -| Q5_K_M | [parakeet-tdt_ctc-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q5_K_M.gguf) | 936 MB | 1.87% | -| Q4_K_M | [parakeet-tdt_ctc-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q4_K_M.gguf) | 825 MB | 1.91% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy TDT decoding and no external LM. F32 reference baseline: 1.87%. NVIDIA's self-reported number on the same split is 1.82% (from the [HF model card](https://huggingface.co/nvidia/parakeet-tdt_ctc-1.1b)). +| F32 | [parakeet-tdt_ctc-1.1b-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-F32.gguf) | 4.28 GB | 1.87% | +| F16 | [parakeet-tdt_ctc-1.1b-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-F16.gguf) | 2.15 GB | 1.87% | +| Q8_0 | [parakeet-tdt_ctc-1.1b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q8_0.gguf) | 1.27 GB | 1.87% | +| Q6_K | [parakeet-tdt_ctc-1.1b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q6_K.gguf) | 1.04 GB | 1.87% | +| Q5_K_M | [parakeet-tdt_ctc-1.1b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q5_K_M.gguf) | 936 MB | 1.87% | +| Q4_K_M | [parakeet-tdt_ctc-1.1b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-1.1b-gguf/resolve/main/parakeet-tdt_ctc-1.1b-Q4_K_M.gguf) | 825 MB | 1.91% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy TDT/RNN-T transducer decoding, no external LM. F32 reference baseline: 1.87%. +NVIDIA's self-reported number on the same split is 1.82%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.68% | + ## Quick Start @@ -54,43 +72,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 97 ms (113×) | 101 ms (109×) | -| Metal | dots (35.3s) | 256 ms (138×) | 258 ms (137×) | -| CPU | jfk (11.0s) | 603 ms (18×) | 510 ms (22×) | -| CPU | dots (35.3s) | 1.87 s (19×) | 1.55 s (23×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `12f1076`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 72 ms (153.68×) | 73 ms (150.60×) | +| Metal | dots (35.3s) | 184 ms (191.86×) | 188 ms (188.19×) | +| CPU | jfk (11.0s) | 458 ms (24.01×) | 509 ms (21.62×) | +| CPU | dots (35.3s) | 1.44 s (24.54×) | 1.57 s (22.56×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 979 ms (11×) | 993 ms (11×) | -| Vulkan | dots (35.3s) | 3.14 s (11×) | 3.14 s (11×) | -| CPU | jfk (11.0s) | 1.88 s (6×) | 1.53 s (7×) | -| CPU | dots (35.3s) | 6.54 s (5×) | 5.63 s (6×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 698 ms (15.76×) | 706 ms (15.59×) | +| Vulkan | dots (35.3s) | 1.93 s (18.32×) | 1.96 s (18.06×) | +| CPU | jfk (11.0s) | 1.10 s (10.03×) | 1.20 s (9.18×) | +| CPU | dots (35.3s) | 4.11 s (8.60×) | 4.23 s (8.35×) | -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-tdt_ctc-1.1b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-tdt_ctc-1.1b-publication +uv run scripts/bench/run.py --profile --models parakeet-tdt_ctc-1.1b ``` ## Numerical Validation diff --git a/docs/models/parakeet-tdt_ctc-110m.md b/docs/models/parakeet-tdt_ctc-110m.md index abf0dbb3..8ba9a69b 100644 --- a/docs/models/parakeet-tdt_ctc-110m.md +++ b/docs/models/parakeet-tdt_ctc-110m.md @@ -1,9 +1,10 @@ # Parakeet TDT-CTC 110M -NVIDIA's [`nvidia/parakeet-tdt_ctc-110m`](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m) -ported to transcribe.cpp. A hybrid 110M-parameter FastConformer encoder with -both TDT and CTC heads sharing the same encoder; transcribe.cpp uses the TDT -head by default. + +Upstream: [`nvidia/parakeet-tdt_ctc-110m`](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m) at [`431a349`](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m/commit/431a349). + +Offline English speech-to-text with punctuation and capitalization. A FastConformer encoder with a TDT/RNNT transducer decoder (the auxiliary CTC head from the upstream hybrid checkpoint is dropped at convert time). Not a streaming model and does not translate. + ## What it's for @@ -16,22 +17,39 @@ It is not a streaming model and does not translate. See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m) for training data, intended use, and upstream evaluation methodology. -Licensed CC-BY-4.0. Ported from upstream commit -[`431a349`](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m/commit/431a349f3051ab85c22b9b7a2741b5fe77065665), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`431a349`](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m/commit/431a349), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [parakeet-tdt_ctc-110m-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-F32.gguf) | 457 MB | 2.43% | -| F16 | [parakeet-tdt_ctc-110m-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-F16.gguf) | 229 MB | 2.43% | -| Q8_0 | [parakeet-tdt_ctc-110m-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q8_0.gguf) | 135 MB | 2.43% | -| Q6_K | [parakeet-tdt_ctc-110m-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q6_K.gguf) | 112 MB | 2.44% | -| Q5_K_M | [parakeet-tdt_ctc-110m-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q5_K_M.gguf) | 101 MB | 2.47% | -| Q4_K_M | [parakeet-tdt_ctc-110m-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q4_K_M.gguf) | 90 MB | 2.53% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy TDT decoding and no external LM. F32 reference baseline: 2.43%. NVIDIA's self-reported number on the same split is 2.40% (from the [HF model card](https://huggingface.co/nvidia/parakeet-tdt_ctc-110m)). +| F32 | [parakeet-tdt_ctc-110m-F32.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-F32.gguf) | 457 MB | 2.43% | +| F16 | [parakeet-tdt_ctc-110m-F16.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-F16.gguf) | 229 MB | 2.43% | +| Q8_0 | [parakeet-tdt_ctc-110m-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q8_0.gguf) | 135 MB | 2.43% | +| Q6_K | [parakeet-tdt_ctc-110m-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q6_K.gguf) | 112 MB | 2.44% | +| Q5_K_M | [parakeet-tdt_ctc-110m-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q5_K_M.gguf) | 101 MB | 2.47% | +| Q4_K_M | [parakeet-tdt_ctc-110m-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-tdt_ctc-110m-gguf/resolve/main/parakeet-tdt_ctc-110m-Q4_K_M.gguf) | 90 MB | 2.53% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy TDT/RNN-T transducer decoding, no external LM. F32 reference baseline: 2.43%. +NVIDIA's self-reported number on the same split is 2.40%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 6.11% | + ## Quick Start @@ -52,43 +70,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 34 ms (327×) | 35 ms (315×) | -| Metal | dots (35.3s) | 96 ms (368×) | 96 ms (367×) | -| CPU | jfk (11.0s) | 91 ms (121×) | 87 ms (126×) | -| CPU | dots (35.3s) | 318 ms (111×) | 306 ms (116×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `12f1076`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 27 ms (405.32×) | 27 ms (401.40×) | +| Metal | dots (35.3s) | 69 ms (509.69×) | 69 ms (510.03×) | +| CPU | jfk (11.0s) | 70 ms (156.94×) | 78 ms (140.61×) | +| CPU | dots (35.3s) | 252 ms (140.19×) | 279 ms (126.50×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 317 ms (35×) | 324 ms (34×) | -| Vulkan | dots (35.3s) | 1.18 s (30×) | 1.20 s (29×) | -| CPU | jfk (11.0s) | 422 ms (26×) | 395 ms (28×) | -| CPU | dots (35.3s) | 1.70 s (21×) | 1.64 s (22×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 142 ms (77.29×) | 144 ms (76.32×) | +| Vulkan | dots (35.3s) | 467 ms (75.62×) | 468 ms (75.48×) | +| CPU | jfk (11.0s) | 199 ms (55.36×) | 231 ms (47.53×) | +| CPU | dots (35.3s) | 790 ms (44.70×) | 870 ms (40.62×) | -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-tdt_ctc-110m \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-tdt_ctc-110m-publication +uv run scripts/bench/run.py --profile --models parakeet-tdt_ctc-110m ``` ## Numerical Validation diff --git a/docs/models/parakeet-unified-en-0.6b.md b/docs/models/parakeet-unified-en-0.6b.md index 39c6b10d..548095e0 100644 --- a/docs/models/parakeet-unified-en-0.6b.md +++ b/docs/models/parakeet-unified-en-0.6b.md @@ -1,8 +1,10 @@ # Parakeet Unified EN 0.6B -NVIDIA's [`nvidia/parakeet-unified-en-0.6b`](https://huggingface.co/nvidia/parakeet-unified-en-0.6b) -ported to transcribe.cpp. A 0.6B-parameter FastConformer encoder with an -RNN-T transducer decoder, trained as a "unified" streaming/offline model. + +Upstream: [`nvidia/parakeet-unified-en-0.6b`](https://huggingface.co/nvidia/parakeet-unified-en-0.6b) at [`d4ac992`](https://huggingface.co/nvidia/parakeet-unified-en-0.6b/commit/d4ac992). + +English speech-to-text with punctuation and capitalization. A FastConformer encoder with an RNN-T transducer decoder, trained as a 'unified' streaming/offline model. This port runs the model in both offline and buffered streaming modes. + ## What it's for @@ -32,22 +34,39 @@ See NVIDIA's [model card](https://huggingface.co/nvidia/parakeet-unified-en-0.6b for training data, intended use, streaming methodology, and upstream evaluation results. -Licensed CC-BY-4.0. Ported from upstream commit -[`d4ac992`](https://huggingface.co/nvidia/parakeet-unified-en-0.6b/commit/d4ac9928), -pinned 2026-05-10. + +Licensed CC-BY-4.0. Ported from upstream commit [`d4ac992`](https://huggingface.co/nvidia/parakeet-unified-en-0.6b/commit/d4ac992), pinned 2026-05-10. Validated against the NeMo reference at transcribe.cpp commit [`42528dd`](https://github.com/handy-computer/transcribe.cpp/tree/42528dd) on 2026-05-10. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) | | --- | --- | ---: | ---: | -| F32 | [parakeet-unified-en-0.6b-F32.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-F32.gguf) | 2.47 GB | 1.59% | -| F16 | [parakeet-unified-en-0.6b-F16.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-F16.gguf) | 1.24 GB | 1.59% | -| Q8_0 | [parakeet-unified-en-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q8_0.gguf) | 731 MB | 1.60% | -| Q6_K | [parakeet-unified-en-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q6_K.gguf) | 602 MB | 1.61% | -| Q5_K_M | [parakeet-unified-en-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q5_K_M.gguf) | 541 MB | 1.58% | -| Q4_K_M | [parakeet-unified-en-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q4_K_M.gguf) | 477 MB | 1.62% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) with greedy RNN-T decoding and no external LM. F32 reference baseline: 1.59%. NVIDIA's self-reported number on the same split is 1.63% (from the [HF model card](https://huggingface.co/nvidia/parakeet-unified-en-0.6b)). +| F32 | [parakeet-unified-en-0.6b-F32.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-F32.gguf) | 2.47 GB | 1.59% | +| F16 | [parakeet-unified-en-0.6b-F16.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-F16.gguf) | 1.24 GB | 1.59% | +| Q8_0 | [parakeet-unified-en-0.6b-Q8_0.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q8_0.gguf) | 731 MB | 1.60% | +| Q6_K | [parakeet-unified-en-0.6b-Q6_K.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q6_K.gguf) | 602 MB | 1.61% | +| Q5_K_M | [parakeet-unified-en-0.6b-Q5_K_M.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q5_K_M.gguf) | 541 MB | 1.58% | +| Q4_K_M | [parakeet-unified-en-0.6b-Q4_K_M.gguf](https://huggingface.co/handy-computer/parakeet-unified-en-0.6b-gguf/resolve/main/parakeet-unified-en-0.6b-Q4_K_M.gguf) | 477 MB | 1.62% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy RNN-T decoding, no external LM. F32 reference baseline: 1.59%. NVIDIA's +self-reported number on the same split is 1.63%. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 3.99% | + ## Quick Start @@ -68,43 +87,40 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). Cells gated on `Tctl < 55°C` per backend. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 69 ms (158×) | 71 ms (155×) | -| Metal | dots (35.3s) | 210 ms (168×) | 209 ms (169×) | -| CPU | jfk (11.0s) | 375 ms (29×) | 318 ms (35×) | -| CPU | dots (35.3s) | 1.27 s (28×) | 1.09 s (32×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `12f1076`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 59 ms (187.72×) | 60 ms (183.34×) | +| Metal | dots (35.3s) | 155 ms (228.39×) | 161 ms (219.30×) | +| CPU | jfk (11.0s) | 285 ms (38.55×) | 307 ms (35.86×) | +| CPU | dots (35.3s) | 979 ms (36.09×) | 1.05 s (33.77×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 839 ms (13×) | 848 ms (13×) | -| Vulkan | dots (35.3s) | 3.03 s (12×) | 3.05 s (12×) | -| CPU | jfk (11.0s) | 1.35 s (8×) | 1.18 s (9×) | -| CPU | dots (35.3s) | 5.22 s (7×) | 4.66 s (8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 448 ms (24.55×) | 455 ms (24.15×) | +| Vulkan | dots (35.3s) | 1.36 s (26.01×) | 1.37 s (25.74×) | +| CPU | jfk (11.0s) | 712 ms (15.45×) | 788 ms (13.96×) | +| CPU | dots (35.3s) | 2.84 s (12.46×) | 2.95 s (11.97×) | -Fedora 43, transcribe.cpp `12f1076`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models parakeet-unified-en-0.6b \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name parakeet-unified-en-0.6b-publication +uv run scripts/bench/run.py --profile --models parakeet-unified-en-0.6b ``` ## Numerical Validation diff --git a/docs/models/parakeet.md b/docs/models/parakeet.md index b476cc73..1cf84d44 100644 --- a/docs/models/parakeet.md +++ b/docs/models/parakeet.md @@ -42,8 +42,7 @@ If you specifically need the lowest WER or a different decoder: (`parakeet-ctc-0.6b` / `parakeet-ctc-1.1b`). Single-pass greedy alignment, no transducer loop — at a ~0.2pp WER cost vs the same-size RNN-T. -- **Tiny footprint.** `parakeet-tdt_ctc-110m` (135 MB at Q8_0) is the - smallest Parakeet. The 1.1B `tdt_ctc` ships both heads but is +- **Tiny footprint.** `parakeet-tdt_ctc-110m` is the smallest Parakeet. The 1.1B `tdt_ctc` ships both heads but is primarily useful when you want TDT speed with CTC as a fallback at runtime. @@ -53,19 +52,21 @@ WER is on LibriSpeech test-clean for the **Q8_0** preset, measured by transcribe.cpp's WER pipeline. See each per-variant doc for the full quant matrix and the comparison to NVIDIA's self-reported numbers. -| Variant | Decoder | Params | Q8_0 size | WER (Q8_0) | Languages | Doc | -| --- | --- | ---: | ---: | ---: | --- | --- | -| `parakeet-tdt-0.6b-v2` | TDT | 0.6B | 730 MB | 1.69% | English | [parakeet-tdt-0.6b-v2.md](parakeet-tdt-0.6b-v2.md) | -| `parakeet-tdt-0.6b-v3` | TDT | 0.6B | 740 MB | 1.94% | 25 European | [parakeet-tdt-0.6b-v3.md](parakeet-tdt-0.6b-v3.md) | -| `parakeet-primeline` | TDT | 0.6B | 740 MB | 6.00%* | 25 European, German-tuned | [parakeet-primeline.md](parakeet-primeline.md) | -| `parakeet-tdt-1.1b` | TDT | 1.1B | 1.27 GB | 1.38% | English | [parakeet-tdt-1.1b.md](parakeet-tdt-1.1b.md) | -| `parakeet-tdt_ctc-110m` | TDT+CTC | 110M | 135 MB | 2.43% | English | [parakeet-tdt_ctc-110m.md](parakeet-tdt_ctc-110m.md) | -| `parakeet-tdt_ctc-1.1b` | TDT+CTC | 1.1B | 1.27 GB | 1.87% | English | [parakeet-tdt_ctc-1.1b.md](parakeet-tdt_ctc-1.1b.md) | -| `parakeet-rnnt-0.6b` | RNN-T | 0.6B | 730 MB | 1.62% | English | [parakeet-rnnt-0.6b.md](parakeet-rnnt-0.6b.md) | -| `parakeet-rnnt-1.1b` | RNN-T | 1.1B | 1.27 GB | 1.46% | English | [parakeet-rnnt-1.1b.md](parakeet-rnnt-1.1b.md) | -| `parakeet-ctc-0.6b` | CTC | 0.6B | 722 MB | 1.87% | English | [parakeet-ctc-0.6b.md](parakeet-ctc-0.6b.md) | -| `parakeet-ctc-1.1b` | CTC | 1.1B | 1.26 GB | 1.85% | English | [parakeet-ctc-1.1b.md](parakeet-ctc-1.1b.md) | -| `parakeet-unified-en-0.6b` | RNN-T | 0.6B | 731 MB | 1.60% | English | [parakeet-unified-en-0.6b.md](parakeet-unified-en-0.6b.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `parakeet-tdt-0.6b-v2` | 618M | en | 730 MB | LibriSpeech test-clean (WER) | 1.69% | token timestamps | [parakeet-tdt-0.6b-v2.md](parakeet-tdt-0.6b-v2.md) | +| `parakeet-tdt-0.6b-v3` | 627M | 25 languages + auto-detect | 740 MB | LibriSpeech test-clean (WER) | 1.94% | token timestamps | [parakeet-tdt-0.6b-v3.md](parakeet-tdt-0.6b-v3.md) | +| `parakeet-primeline` | 627M | 25 languages + auto-detect | 740 MB | FLEURS de (WER) | 5.98% | token timestamps | [parakeet-primeline.md](parakeet-primeline.md) | +| `parakeet-tdt-1.1b` | 1.1B | en | 1.27 GB | LibriSpeech test-clean (WER) | 1.38% | token timestamps | [parakeet-tdt-1.1b.md](parakeet-tdt-1.1b.md) | +| `parakeet-tdt_ctc-110m` | 114M | en | 135 MB | LibriSpeech test-clean (WER) | 2.43% | token timestamps | [parakeet-tdt_ctc-110m.md](parakeet-tdt_ctc-110m.md) | +| `parakeet-tdt_ctc-1.1b` | 1.1B | en | 1.27 GB | LibriSpeech test-clean (WER) | 1.87% | token timestamps | [parakeet-tdt_ctc-1.1b.md](parakeet-tdt_ctc-1.1b.md) | +| `parakeet-rnnt-0.6b` | 617M | en | 730 MB | LibriSpeech test-clean (WER) | 1.62% | token timestamps | [parakeet-rnnt-0.6b.md](parakeet-rnnt-0.6b.md) | +| `parakeet-rnnt-1.1b` | 1.1B | en | 1.27 GB | LibriSpeech test-clean (WER) | 1.46% | token timestamps | [parakeet-rnnt-1.1b.md](parakeet-rnnt-1.1b.md) | +| `parakeet-ctc-0.6b` | 609M | en | 722 MB | LibriSpeech test-clean (WER) | 1.87% | token timestamps | [parakeet-ctc-0.6b.md](parakeet-ctc-0.6b.md) | +| `parakeet-ctc-1.1b` | 1.1B | en | 1.26 GB | LibriSpeech test-clean (WER) | 1.85% | token timestamps | [parakeet-ctc-1.1b.md](parakeet-ctc-1.1b.md) | +| `parakeet-unified-en-0.6b` | 618M | en | 731 MB | LibriSpeech test-clean (WER) | 1.60% | streaming, token timestamps | [parakeet-unified-en-0.6b.md](parakeet-unified-en-0.6b.md) | + \* `parakeet-primeline` is scored on FLEURS German (862 utterances), not LibriSpeech test-clean, so its number is not comparable to the rest diff --git a/docs/models/qwen3-asr-0.6b.md b/docs/models/qwen3-asr-0.6b.md index d36a91e2..93e2c4e3 100644 --- a/docs/models/qwen3-asr-0.6b.md +++ b/docs/models/qwen3-asr-0.6b.md @@ -1,9 +1,14 @@ # Qwen3-ASR 0.6B -Alibaba's [`Qwen/Qwen3-ASR-0.6B`](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) -ported to transcribe.cpp. An 18-layer bidirectional audio encoder feeds a -28-layer Qwen3 causal LM with audio-token injection (no cross-attention — -the LM processes a fused audio+text sequence through a chat template). + +Upstream: [`Qwen/Qwen3-ASR-0.6B`](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) at [`5eb1441`](https://huggingface.co/Qwen/Qwen3-ASR-0.6B/commit/5eb1441). + +Offline multilingual speech-to-text. An 18-layer bidirectional audio encoder +feeds a 28-layer Qwen3 causal LM with audio-token injection (fused +audio+text sequence, no cross-attention). Auto-detects the audio's language +across 30 languages and emits the transcript in that language. Takes a +16 kHz mono WAV; explicit language hints are not supported at this time. + ## What it's for @@ -17,24 +22,73 @@ See the [Qwen3-ASR model card](https://huggingface.co/Qwen/Qwen3-ASR-0.6B) for training data, intended use, and upstream evaluation methodology. -Licensed Apache-2.0 (weights) / Apache-2.0 (author `qwen_asr` package). -Ported from upstream commit -[`5eb1441`](https://huggingface.co/Qwen/Qwen3-ASR-0.6B/commit/5eb144179a02acc5e5ba31e748d22b0cf3e303b0). + +Licensed Apache-2.0. Ported from upstream commit [`5eb1441`](https://huggingface.co/Qwen/Qwen3-ASR-0.6B/commit/5eb1441), pinned 2026-04-19. Validated against the qwen_asr 0.0.6 reference at transcribe.cpp commit [`3f61df7`](https://github.com/handy-computer/transcribe.cpp/tree/3f61df7) on 2026-04-20. + + +The author's `qwen_asr` package is likewise Apache-2.0. ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [Qwen3-ASR-0.6B-BF16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-BF16.gguf) | 1499 MB | 2.11% | -| F16 | [Qwen3-ASR-0.6B-F16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-F16.gguf) | 1507 MB | 2.12% | -| Q8_0 | [Qwen3-ASR-0.6B-Q8_0.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q8_0.gguf) | 811 MB | 2.11% | -| Q6_K | [Qwen3-ASR-0.6B-Q6_K.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q6_K.gguf) | 763 MB | 2.10% | -| Q5_K_M | [Qwen3-ASR-0.6B-Q5_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q5_K_M.gguf) | 707 MB | 2.21% | -| Q4_K_M | [Qwen3-ASR-0.6B-Q4_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q4_K_M.gguf) | 654 MB | 2.26% | - -WER measured on LibriSpeech `test-clean` (2620 utterances), Whisper-style -English text normalizer, jiwer 3.x, metal backend on Apple M4. Reproduce -with `scripts/wer/run.py` + `scripts/wer/score.py`. +| BF16 | [Qwen3-ASR-0.6B-BF16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-BF16.gguf) | 1.57 GB | 2.12% | +| F16 | [Qwen3-ASR-0.6B-F16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-F16.gguf) | 1.58 GB | 2.12% | +| Q8_0 | [Qwen3-ASR-0.6B-Q8_0.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q8_0.gguf) | 850 MB | 2.11% | +| Q6_K | [Qwen3-ASR-0.6B-Q6_K.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q6_K.gguf) | 690 MB | 2.11% | +| Q5_K_M | [Qwen3-ASR-0.6B-Q5_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q5_K_M.gguf) | 645 MB | 2.21% | +| Q4_K_M | [Qwen3-ASR-0.6B-Q4_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-0.6B-gguf/resolve/main/Qwen3-ASR-0.6B-Q4_K_M.gguf) | 590 MB | 2.26% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Scored with the Whisper-style English text normalizer and jiwer 3.x on an Apple M4. +Qwen3-ASR is a multilingual model — this number characterizes the English case only. +BF16 / F16 / Q8_0 / Q6_K are all within bootstrap CI of each other; Q5_K_M and +Q4_K_M show a small but real regression driven by the tied token-embedding / head. +Reproduce with `scripts/wer/run.py` + `scripts/wer/score.py`. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| ar | WER | 24.51% | +| cs | WER | 44.50% | +| da | WER | 36.07% | +| de | WER | 6.80% | +| el | WER | 49.12% | +| en | WER | 4.23% | +| es | WER | 4.88% | +| fa | WER | 50.30% | +| fi | WER | 46.49% | +| fil | WER | 35.43% | +| fr | WER | 7.76% | +| hi | WER | 12.68% | +| hu | WER | 56.24% | +| id | WER | 8.49% | +| it | WER | 5.19% | +| ja | CER | 8.61% | +| ko | CER | 5.82% | +| mk | WER | 35.09% | +| ms | WER | 17.18% | +| nl | WER | 13.90% | +| pl | WER | 25.06% | +| pt | WER | 6.57% | +| ro | WER | 40.65% | +| ru | WER | 10.30% | +| sv | WER | 35.72% | +| th | CER | 8.81% | +| tr | WER | 16.74% | +| vi | WER | 9.32% | +| yue | CER | 7.91% | +| zh | CER | 7.57% | + **FLEURS-zh** (945 utterances) CER: 7.6% on the upstream `qwen_asr` reference, 7.64% on the Q8_0 port (95% CI [6.74%, 8.51%]); within @@ -71,53 +125,48 @@ chat template is tracked as follow-up work; see the family note at ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 155 ms (71×) | 142 ms (77×) | -| Metal | dots (35.3s) | 597 ms (59×) | 527 ms (67×) | -| CPU | jfk (11.0s) | 660 ms (17×) | 588 ms (19×) | -| CPU | dots (35.3s) | 2.26 s (16×) | 2.10 s (17×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.3.1, transcribe.cpp `0c88a71`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 147 ms (75.02×) | 134 ms (81.78×) | +| Metal | dots (35.3s) | 556 ms (63.50×) | 511 ms (69.19×) | +| CPU | jfk (11.0s) | 545 ms (20.20×) | 547 ms (20.10×) | +| CPU | dots (35.3s) | 2.06 s (17.13×) | 1.97 s (17.95×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 1.27 s (8.7×) | 1.08 s (10.1×) | -| Vulkan | dots (35.3s) | 4.87 s (7.3×) | 3.99 s (8.9×) | -| CPU | jfk (11.0s) | 2.37 s (4.6×) | 1.92 s (5.7×) | -| CPU | dots (35.3s) | 8.61 s (4.1×) | 7.34 s (4.8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 1.03 s (10.73×) | 908 ms (12.12×) | +| Vulkan | dots (35.3s) | 4.07 s (8.68×) | 3.55 s (9.96×) | +| CPU | jfk (11.0s) | 1.86 s (5.91×) | 1.67 s (6.58×) | +| CPU | dots (35.3s) | 7.55 s (4.68×) | 6.75 s (5.23×) | -Fedora 43, transcribe.cpp `3d16f74`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models Qwen3-ASR-0.6B \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name qwen3-asr-0.6b-publication +uv run scripts/bench/run.py --profile --models qwen3-asr-0.6b ``` ## Numerical Validation -transcribe.cpp is validated tensor-by-tensor against the author -reference implementation (`qwen_asr` 0.0.6 / transformers 4.57.6) on -`samples/jfk.wav`. All 13 checkpointed tensors fall within family -tolerance on CPU / Metal / Vulkan, and the transcript matches the -reference verbatim. Last validated at commit -[`3f61df7`](https://github.com/handy-computer/transcribe.cpp/tree/3f61df7). +transcribe.cpp is validated tensor-by-tensor against the author reference +implementation (`qwen_asr` 0.0.6 / transformers 4.57.6) on +`samples/jfk.wav`. All 13 checkpointed tensors fall within family tolerance +on CPU / Metal / Vulkan, and the transcript matches the reference verbatim. | Field | Value | | --- | --- | @@ -126,22 +175,6 @@ reference verbatim. Last validated at commit | Manifest | `tests/golden/qwen3_asr/qwen3-asr-0.6b.manifest.json` | | Command | `uv run scripts/validate.py all --family qwen3_asr --variant qwen3-asr-0.6b` | -Selected tensors (observed on CPU; see tolerance file for budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.mel.in` | `3.906e-03` | `6.608e-04` | fp64 vs fp32 STFT precision gap | -| `enc.subsample.out` | `1.814e-02` | `1.891e-03` | After the 4× conv subsampler | -| `enc.block.0.out` | `5.645e-02` | `2.768e-03` | Early encoder | -| `enc.block.17.out` | `8.221e-01` | `9.720e-03` | Final encoder block | -| `enc.proj.out` | `1.484e-02` | `2.324e-04` | Audio→LM width projection | -| `dec.audio_injected` | `1.484e-02` | `2.103e-04` | Fused audio+text sequence | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | Exact match | -| `dec.block.0.out` | `1.719e-01` | `3.127e-03` | Early LM | -| `dec.block.27.out` | `2.912e+01` | `3.824e-01` | Final LM block (accumulated) | -| `dec.out_before_head`| `2.844e+01` | `1.786e-01` | Pre-head hidden state | -| `dec.logits_raw` | `1.054e+00` | `1.749e-01` | Raw logits | - ## Reproduction ### Convert diff --git a/docs/models/qwen3-asr-1.7b.md b/docs/models/qwen3-asr-1.7b.md index e32f5ca4..c092028c 100644 --- a/docs/models/qwen3-asr-1.7b.md +++ b/docs/models/qwen3-asr-1.7b.md @@ -1,10 +1,16 @@ # Qwen3-ASR 1.7B -Alibaba's [`Qwen/Qwen3-ASR-1.7B`](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) -ported to transcribe.cpp. Architecture is the same audio-LLM pattern as -the 0.6B variant (24-layer bidirectional audio encoder + Qwen3 causal LM -with audio-token injection); the 1.7B is wider: encoder `d_model=1024` -(16 heads), LM `hidden_size=2048`, `intermediate_size=6144`. + +Upstream: [`Qwen/Qwen3-ASR-1.7B`](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) at [`7278e1e`](https://huggingface.co/Qwen/Qwen3-ASR-1.7B/commit/7278e1e). + +Offline multilingual speech-to-text. Same audio-LLM architecture as the +0.6B variant (bidirectional audio encoder feeding a Qwen3 causal LM with +audio-token injection), wider: encoder `d_model=1024` (16 heads), LM +`hidden_size=2048`, `intermediate_size=6144`. Auto-detects the audio's +language across 30 languages and emits the transcript in that language. +Takes a 16 kHz mono WAV; explicit language hints are not supported at +this time. + ## What it's for @@ -16,24 +22,73 @@ See the [Qwen3-ASR-1.7B model card](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) for training data and upstream evaluation. -Licensed Apache-2.0 (weights) / Apache-2.0 (author `qwen_asr` package). -Ported from upstream commit -[`7278e1e`](https://huggingface.co/Qwen/Qwen3-ASR-1.7B/commit/7278e1e70fe206f11671096ffdd38061171dd6e5). + +Licensed Apache-2.0. Ported from upstream commit [`7278e1e`](https://huggingface.co/Qwen/Qwen3-ASR-1.7B/commit/7278e1e), pinned 2026-04-19. Validated against the qwen_asr 0.0.6 reference at transcribe.cpp commit [`3f61df7`](https://github.com/handy-computer/transcribe.cpp/tree/3f61df7) on 2026-04-20. + + +The author's `qwen_asr` package is likewise Apache-2.0. ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [Qwen3-ASR-1.7B-BF16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-BF16.gguf) | 3894 MB | 1.62% | -| F16 | [Qwen3-ASR-1.7B-F16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-F16.gguf) | 3902 MB | 1.62% | -| Q8_0 | [Qwen3-ASR-1.7B-Q8_0.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q8_0.gguf) | 2084 MB | 1.61% | -| Q6_K | [Qwen3-ASR-1.7B-Q6_K.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q6_K.gguf) | 1614 MB | 1.65% | -| Q5_K_M | [Qwen3-ASR-1.7B-Q5_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q5_K_M.gguf) | 1447 MB | 1.65% | -| Q4_K_M | [Qwen3-ASR-1.7B-Q4_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q4_K_M.gguf) | 1259 MB | 1.81% | - -WER measured on LibriSpeech `test-clean` (2620 utterances), Whisper-style -English text normalizer, jiwer 3.x, metal backend on Apple M4. Reproduce -with `scripts/wer/run.py` + `scripts/wer/score.py`. +| BF16 | [Qwen3-ASR-1.7B-BF16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-BF16.gguf) | 4.08 GB | 1.62% | +| F16 | [Qwen3-ASR-1.7B-F16.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-F16.gguf) | 4.09 GB | 1.62% | +| Q8_0 | [Qwen3-ASR-1.7B-Q8_0.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q8_0.gguf) | 2.19 GB | 1.62% | +| Q6_K | [Qwen3-ASR-1.7B-Q6_K.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q6_K.gguf) | 1.69 GB | 1.65% | +| Q5_K_M | [Qwen3-ASR-1.7B-Q5_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q5_K_M.gguf) | 1.52 GB | 1.65% | +| Q4_K_M | [Qwen3-ASR-1.7B-Q4_K_M.gguf](https://huggingface.co/handy-computer/Qwen3-ASR-1.7B-gguf/resolve/main/Qwen3-ASR-1.7B-Q4_K_M.gguf) | 1.32 GB | 1.81% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Scored with the Whisper-style English text normalizer and jiwer 3.x on an Apple M4. +Qwen3-ASR is a multilingual model — this number characterizes the English case only. +The larger decoder gives 1.7B more quantization headroom than the 0.6B; BF16 / F16 / +Q8_0 / Q6_K / Q5_K_M are all within bootstrap CI of each other, and Q4_K_M regresses +only ~0.2 WER points. Reproduce with `scripts/wer/run.py` + `scripts/wer/score.py`. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| ar | WER | 14.91% | +| cs | WER | 22.97% | +| da | WER | 21.23% | +| de | WER | 4.25% | +| el | WER | 29.22% | +| en | WER | 3.23% | +| es | WER | 3.31% | +| fa | WER | 28.29% | +| fi | WER | 25.48% | +| fil | WER | 24.29% | +| fr | WER | 4.52% | +| hi | WER | 7.84% | +| hu | WER | 32.84% | +| id | WER | 5.37% | +| it | WER | 2.68% | +| ja | CER | 5.29% | +| ko | CER | 4.60% | +| mk | WER | 18.22% | +| ms | WER | 10.42% | +| nl | WER | 7.43% | +| pl | WER | 12.50% | +| pt | WER | 4.37% | +| ro | WER | 20.46% | +| ru | WER | 6.25% | +| sv | WER | 19.68% | +| th | CER | 6.89% | +| tr | WER | 9.46% | +| vi | WER | 6.15% | +| yue | CER | 6.13% | +| zh | CER | 7.14% | + ## Quick Start @@ -61,43 +116,40 @@ the rationale and the planned follow-up. ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 256 ms (43×) | 213 ms (52×) | -| Metal | dots (35.3s) | 978 ms (36×) | 803 ms (44×) | -| CPU | jfk (11.0s) | 1.40 s (8×) | 1.10 s (10×) | -| CPU | dots (35.3s) | 4.46 s (8×) | 4.04 s (9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.3.1, transcribe.cpp `0c88a71`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 243 ms (45.33×) | 209 ms (52.65×) | +| Metal | dots (35.3s) | 959 ms (36.83×) | 804 ms (43.95×) | +| CPU | jfk (11.0s) | 1.06 s (10.35×) | 1.23 s (8.96×) | +| CPU | dots (35.3s) | 4.00 s (8.83×) | 3.80 s (9.31×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 2.66 s (4.1×) | 2.29 s (4.8×) | -| Vulkan | dots (35.3s) | 9.87 s (3.6×) | 8.37 s (4.2×) | -| CPU | jfk (11.0s) | 5.19 s (2.1×) | 3.58 s (3.1×) | -| CPU | dots (35.3s) | 18.53 s (1.9×) | 12.93 s (2.7×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 2.47 s (4.45×) | 2.02 s (5.45×) | +| Vulkan | dots (35.3s) | 9.49 s (3.72×) | 7.89 s (4.48×) | +| CPU | jfk (11.0s) | 4.10 s (2.68×) | 3.51 s (3.14×) | +| CPU | dots (35.3s) | 15.63 s (2.26×) | 13.07 s (2.70×) | -Fedora 43, transcribe.cpp `3d16f74`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models Qwen3-ASR-1.7B \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name qwen3-asr-1.7b-publication +uv run scripts/bench/run.py --profile --models qwen3-asr-1.7b ``` ## Numerical Validation @@ -117,22 +169,6 @@ validated at commit | Manifest | `tests/golden/qwen3_asr/qwen3-asr-1.7b.manifest.json` | | Command | `uv run scripts/validate.py all --family qwen3_asr --variant qwen3-asr-1.7b` | -Selected tensors (observed on CPU; see tolerance file for budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `enc.mel.in` | `3.906e-03` | `6.608e-04` | fp64 vs fp32 STFT precision gap | -| `enc.subsample.out` | `2.574e-02` | `2.473e-03` | After the 4× conv subsampler | -| `enc.block.0.out` | `4.970e-02` | `3.068e-03` | Early encoder | -| `enc.block.23.out` | `6.846e-01` | `1.125e-02` | Final encoder block | -| `enc.proj.out` | `2.351e-03` | `1.595e-04` | Audio→LM width projection | -| `dec.audio_injected` | `2.351e-03` | `1.443e-04` | Fused audio+text sequence | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | Exact match | -| `dec.block.0.out` | `1.439e-01` | `3.223e-03` | Early LM | -| `dec.block.27.out` | `2.052e+02` | `9.783e-01` | Final LM block (accumulated) | -| `dec.out_before_head`| `1.828e+01` | `7.569e-02` | Pre-head hidden state | -| `dec.logits_raw` | `1.266e+00` | `2.131e-01` | Raw logits | - ## Reproduction ### Convert diff --git a/docs/models/qwen3-asr.md b/docs/models/qwen3-asr.md index a5553c73..ed816492 100644 --- a/docs/models/qwen3-asr.md +++ b/docs/models/qwen3-asr.md @@ -13,9 +13,8 @@ see the family doc at ## Choosing a variant -- **Smaller, faster, near-realtime CPU.** `qwen3-asr-0.6b` — 600M - parameters, 811 MB at Q8_0. 18-layer encoder + Qwen3 LM with - `hidden_size=1024`. ~2.1% WER on LibriSpeech test-clean. +- **Smaller, faster, near-realtime CPU.** `qwen3-asr-0.6b` pairs an + 18-layer encoder with a Qwen3 LM at `hidden_size=1024`. - **Accuracy headroom.** `qwen3-asr-1.7b` widens both halves of the model (24-layer encoder, LM `hidden_size=2048`, `intermediate_size=6144`) for ~0.5pp WER improvement at ~2.5× the @@ -29,10 +28,12 @@ WER is on LibriSpeech test-clean for the **Q8_0** preset, measured by transcribe.cpp's WER pipeline. See each per-variant doc for the full quant matrix. -| Variant | Params | Q8_0 size | WER (Q8_0) | Languages | Doc | -| --- | ---: | ---: | ---: | --- | --- | -| `qwen3-asr-0.6b` | ~600M | 811 MB | 2.11% | 30 (auto-detect) | [qwen3-asr-0.6b.md](qwen3-asr-0.6b.md) | -| `qwen3-asr-1.7b` | ~1.7B | 2.08 GB | 1.61% | 30 (auto-detect) | [qwen3-asr-1.7b.md](qwen3-asr-1.7b.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `qwen3-asr-0.6b` | 782M | 30 languages + auto-detect | 850 MB | LibriSpeech test-clean (WER) | 2.11% | - | [qwen3-asr-0.6b.md](qwen3-asr-0.6b.md) | +| `qwen3-asr-1.7b` | 2B | 30 languages + auto-detect | 2.19 GB | LibriSpeech test-clean (WER) | 1.62% | - | [qwen3-asr-1.7b.md](qwen3-asr-1.7b.md) | + Pre-built GGUFs for every variant and quant are hosted under [`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); diff --git a/docs/models/sensevoice-small.md b/docs/models/sensevoice-small.md index f2acdf50..0d4d28cd 100644 --- a/docs/models/sensevoice-small.md +++ b/docs/models/sensevoice-small.md @@ -1,9 +1,18 @@ # SenseVoice Small -Alibaba / FunAudioLLM's [`FunAudioLLM/SenseVoiceSmall`](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) -ported to transcribe.cpp. A 234M-parameter SAN-M encoder with a single CTC -head over a 25,055-token SentencePiece vocabulary covering Chinese, Cantonese, -English, Japanese, and Korean. + +Upstream: [`FunAudioLLM/SenseVoiceSmall`](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) at [`3eb3b4e`](https://huggingface.co/FunAudioLLM/SenseVoiceSmall/commit/3eb3b4e). + +Offline multilingual speech-to-text in Chinese, Cantonese, English, Japanese, +and Korean. A SAN-M encoder with a single CTC head over a 25,055-token +SentencePiece vocabulary. Takes a 16 kHz mono WAV (capped at 30 seconds per +call, per upstream's direct-inference contract) and produces a transcript. Not +a streaming model, no translation, no built-in long-form chunking. The same CTC +head also emits language-ID, simple emotion labels, audio-event tags, and +inverse-text-normalization control tags. These tags are hidden unless +`--raw-tokens` is passed. ITN is on by default for readable casing, +punctuation, and digits; pass `--no-itn` for upstream's spoken-form output. + ## What it's for @@ -46,12 +55,11 @@ numbers below are measured with ITN **off**, matching the reference runs; see See FunAudioLLM's [model card](https://huggingface.co/FunAudioLLM/SenseVoiceSmall) for training data, intended use, and upstream evaluation methodology. -Licensed under the **FunASR Model Open Source License Agreement** — -the legacy "model-license" form -([MODEL_LICENSE](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE)). -Ported from upstream commit -[`3eb3b4e`](https://huggingface.co/FunAudioLLM/SenseVoiceSmall/commit/3eb3b4eeffc2f2dde6051b853983753db33e35c3), -pinned 2026-05-06. + +Licensed [model-license (FunASR MODEL_LICENSE)](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE). Ported from upstream commit [`3eb3b4e`](https://huggingface.co/FunAudioLLM/SenseVoiceSmall/commit/3eb3b4e), pinned 2026-05-06. Validated against the FunASR reference at transcribe.cpp commit [`f094d28`](https://github.com/handy-computer/transcribe.cpp/tree/f094d28) on 2026-05-06. + + +This is the legacy `model-license` form of the agreement. ## Input limits @@ -63,23 +71,43 @@ recordings (e.g. with VAD) for best results. See the ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [SenseVoiceSmall-F32.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-F32.gguf) | 893 MB | 3.13% | -| F16 | [SenseVoiceSmall-F16.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-F16.gguf) | 449 MB | 3.13% | -| Q8_0 | [SenseVoiceSmall-Q8_0.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q8_0.gguf) | 241 MB | 3.13% | -| Q6_K | [SenseVoiceSmall-Q6_K.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q6_K.gguf) | 187 MB | 3.14% | -| Q5_K_M | [SenseVoiceSmall-Q5_K_M.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q5_K_M.gguf) | 164 MB | 3.18% | -| Q4_K_M | [SenseVoiceSmall-Q4_K_M.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q4_K_M.gguf) | 139 MB | 3.45% | - -WER is measured on the full LibriSpeech test-clean split (2620 utterances) -with greedy CTC decoding. The publisher does not report a numerical -LibriSpeech WER, so the -gate baseline is **our own FunASR 1.3.1 reference run** on the same manifest: -3.13% (95% CI [2.93%, 3.34%]). transcribe.cpp's F32 port matches that -baseline within +0.002 percentage-points. Q4_K_M is the only quant with a -visible regression (+0.32 pp); F16 / Q8_0 / Q6_K / Q5_K_M are within -bootstrap noise of F32. +| F32 | [SenseVoiceSmall-F32.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-F32.gguf) | 937 MB | 3.13% | +| F16 | [SenseVoiceSmall-F16.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-F16.gguf) | 470 MB | 3.13% | +| Q8_0 | [SenseVoiceSmall-Q8_0.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q8_0.gguf) | 253 MB | 3.13% | +| Q6_K | [SenseVoiceSmall-Q6_K.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q6_K.gguf) | 196 MB | 3.14% | +| Q5_K_M | [SenseVoiceSmall-Q5_K_M.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q5_K_M.gguf) | 172 MB | 3.18% | +| Q4_K_M | [SenseVoiceSmall-Q4_K_M.gguf](https://huggingface.co/handy-computer/SenseVoiceSmall-gguf/resolve/main/SenseVoiceSmall-Q4_K_M.gguf) | 146 MB | 3.45% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances). Figures without a commit were published before provenance was recorded. + + + +Greedy CTC decoding. The publisher does not report a numerical LibriSpeech WER (the +model card publishes scores only as PNG figures), so the gate baseline is our own +FunASR 1.3.1 reference run on the same manifest: 3.13% (95% CI [2.93%, 3.34%]). +transcribe.cpp's F32 port matches that baseline within +0.002 percentage-points. +LibriSpeech is an English benchmark; SenseVoice's strongest case is Mandarin, and +AISHELL-1 (CER) is the recommended complementary check. These table values were +measured with ITN off, matching the FunASR reference; `scripts/wer/run.py` pins +`--no-itn` so the benchmark does not inherit the runtime default. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 7.14% | +| ja | CER | 7.63% | +| ko | CER | 8.27% | +| yue | CER | 37.44% | +| zh | CER | 10.12% | + LibriSpeech is an English benchmark; SenseVoice's strongest case is Mandarin. **FLEURS-zh** (945 utterances) CER: 10.20% on our FunASR 1.3.1 @@ -126,52 +154,49 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), -with speedup over realtime in parentheses. Units: `ms` below 1 s, `s` -above (2 decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Metal | jfk (11.0s) | 42 ms (260×) | 44 ms (250×) | -| Metal | dots (35.3s) | 111 ms (319×) | 137 ms (258×) | -| CPU | jfk (11.0s) | 208 ms (53×) | 213 ms (52×) | -| CPU | dots (35.3s) | 700 ms (50×) | 727 ms (49×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ---------------: | ---------------: | +| Metal | jfk (11.0s) | 45 ms (245.28×) | 46 ms (238.72×) | +| Metal | dots (35.3s) | 122 ms (288.69×) | 123 ms (286.71×) | +| CPU | jfk (11.0s) | 215 ms (51.15×) | 220 ms (49.92×) | +| CPU | dots (35.3s) | 737 ms (47.92×) | 757 ms (46.69×) | -macOS 26.4.1, transcribe.cpp `811fe2a`. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ------------: | ------------: | -| Vulkan | jfk (11.0s) | 329 ms (33×) | 332 ms (33×) | -| Vulkan | dots (35.3s) | 1.11 s (32×) | 1.12 s (31×) | -| CPU | jfk (11.0s) | 687 ms (16×) | 590 ms (19×) | -| CPU | dots (35.3s) | 2.31 s (15×) | 2.03 s (17×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -Fedora 43, transcribe.cpp `8635bd1`. Vulkan device: `AMD Radeon Graphics (RADV RENOIR)`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 395 ms (27.85×) | 398 ms (27.63×) | +| Vulkan | dots (35.3s) | 1.28 s (27.53×) | 1.30 s (27.27×) | +| CPU | jfk (11.0s) | 531 ms (20.71×) | 627 ms (17.55×) | +| CPU | dots (35.3s) | 1.88 s (18.76×) | 2.15 s (16.40×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models SenseVoiceSmall \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name sensevoice-small-publication +uv run scripts/bench/run.py --profile --models sensevoice-small ``` ## Numerical Validation -transcribe.cpp is validated tensor-by-tensor against FunASR 1.3.1 -on `samples/jfk.wav`. All 16 checkpointed tensors fall within family +transcribe.cpp is validated tensor-by-tensor against FunASR 1.3.1 on +`samples/jfk.wav`. All 16 checkpointed tensors fall within family tolerance, and the final transcript matches the FunASR reference verbatim (both spelled `… laled out …` on token `1089-134686-0000` — a quirk of -SenseVoice, not a port defect). Last validated at commit -[`f094d28`](https://github.com/handy-computer/transcribe.cpp/tree/f094d28). +SenseVoice, not a port defect). | Field | Value | | --- | --- | @@ -180,23 +205,6 @@ SenseVoice, not a port defect). Last validated at commit | Manifest | `tests/golden/sensevoice/sensevoice-small.manifest.json` | | Command | `uv run scripts/validate.py compare --family sensevoice --variant sensevoice-small` | -Selected tensors: - -| Tensor | Max abs diff | Mean abs diff | Notes | -| --- | ---: | ---: | --- | -| `frontend.fbank.lfr.cmvn.out` | `3.13e-03` | `6.34e-04` | fp32 FFT vs C++ fp64 STFT round-off | -| `enc.input.with_prefix` | `3.13e-03` | `6.20e-04` | frontend drift carried by concat (no compute) | -| `enc.embed.out` | `7.08e-02` | `1.40e-02` | frontend drift × √d_model after sinusoidal PE | -| `enc.encoders0.0.out` | `1.53e+02` | `2.52e+00` | first SAN-M block (560→512 projection) | -| `enc.encoders.0.out` | `9.63e+01` | `2.78e+00` | main-tier block 0 | -| `enc.encoders.24.out` | `4.60e+02` | `1.41e+01` | mid-tier (block 24); reference values ~4.5k | -| `enc.encoders.48.out` | `4.52e+04` | `7.74e+01` | last main block; reference values ~46k | -| `enc.after_norm.out` | `6.66e+00` | `4.71e-01` | tier-boundary LayerNorm renormalises | -| `enc.tp_encoders.{0,10,19}.out` | `≤ 1.10e+04` | `≤ 9.07e+00` | tp-tier 20-block stack | -| `enc.tp_norm.out` | `1.53e+01` | `5.86e-01` | final encoder output, post-LN | -| `ctc.logits.raw` | `3.23e+01` | `1.87e+00` | CTC logits — argmax positions identical | -| `ctc.log_probs` | `3.07e+01` | `2.90e+00` | log-softmax CTC distribution | - The expected divergence is fp32 reduction-order drift accumulated through 70 SAN-M blocks. SenseVoice's encoder has **no inter-layer normalization** (only `after_norm` between the two tiers and `tp_norm` at the end), so diff --git a/docs/models/voxtral-mini-3b-2507.md b/docs/models/voxtral-mini-3b-2507.md index 3b6b3cd1..ce1f0329 100644 --- a/docs/models/voxtral-mini-3b-2507.md +++ b/docs/models/voxtral-mini-3b-2507.md @@ -1,12 +1,17 @@ # Voxtral Mini 3B (2507) -Mistral's [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) -ported to transcribe.cpp. An offline audio-LLM: a Whisper-large-v3 -bidirectional audio encoder (32 layers, `d_model=1280`, 20 heads) feeds -a 4-frame-group projector (375 audio tokens per 30 s chunk) into a -Ministral-3B causal LM (30 layers, `hidden_size=3072`, -`intermediate_size=8192`, GQA 32 q / 8 kv heads, NEOX RoPE, SwiGLU) via -audio-token injection at the `audio_token_id=24` positions in the prompt. + +Upstream: [`mistralai/Voxtral-Mini-3B-2507`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) at [`3060fe3`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507/commit/3060fe3). + +Offline audio-LLM speech-to-text and speech translation. A Whisper-large-v3 +bidirectional audio encoder feeds a 4-frame-group projector (375 audio tokens +per 30 s chunk) into a Ministral-3B causal LM (30 layers, GQA 32/8, NEOX RoPE, +SwiGLU) via audio-token injection. Takes a 16 kHz mono WAV and produces a +transcript via greedy decoding; speech translation runs through the +mistral-common instruct template. The smaller sibling of Voxtral Small 24B — +same encoder, projector, log-mel frontend, and tekken tokenizer, with a 3B +decoder in place of Mistral-Small-24B. + ## What it's for @@ -23,27 +28,49 @@ mono WAV and produces a transcript via greedy decoding. See Mistral's [model card](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507) for training data, intended use, and upstream evaluation. -Licensed Apache-2.0. Ported from upstream commit -[`3060fe3`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507/commit/3060fe34b35ba5d44202ce9ff3c097642914f8f3), -pinned 2026-06-06. + +Licensed Apache-2.0. Ported from upstream commit [`3060fe3`](https://huggingface.co/mistralai/Voxtral-Mini-3B-2507/commit/3060fe3), pinned 2026-06-06. Validated against the Transformers reference at transcribe.cpp commit [`483c122`](https://github.com/handy-computer/transcribe.cpp/tree/483c122) on 2026-06-06. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [Voxtral-Mini-3B-2507-BF16.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-BF16.gguf) | 9.37 GB | 1.88% | -| F16 | [Voxtral-Mini-3B-2507-F16.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-F16.gguf) | 9.38 GB | 1.89% | -| Q8_0 | [Voxtral-Mini-3B-2507-Q8_0.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q8_0.gguf) | 5.00 GB | 1.87% | -| Q6_K | [Voxtral-Mini-3B-2507-Q6_K.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q6_K.gguf) | 3.87 GB | 1.87% | -| Q5_K_M | [Voxtral-Mini-3B-2507-Q5_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q5_K_M.gguf) | 3.46 GB | 1.91% | -| Q4_K_M | [Voxtral-Mini-3B-2507-Q4_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q4_K_M.gguf) | 2.98 GB | 1.94% | - -WER measured on the full LibriSpeech `test-clean` split (2620 utterances) -with the Whisper-style English text normalizer, batch size 8 on an NVIDIA -L40S. The same-machine HuggingFace `transformers` reference run -(`VoxtralForConditionalGeneration`, BF16, `attn_implementation=eager`, -greedy) lands at **1.87%**, and the BF16 GGUF matches it (1.87% at batch -1). +| BF16 | [Voxtral-Mini-3B-2507-BF16.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-BF16.gguf) | 9.37 GB | 1.88% | +| F16 | [Voxtral-Mini-3B-2507-F16.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-F16.gguf) | 9.38 GB | 1.89% | +| Q8_0 | [Voxtral-Mini-3B-2507-Q8_0.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q8_0.gguf) | 5.00 GB | 1.87% | +| Q6_K | [Voxtral-Mini-3B-2507-Q6_K.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q6_K.gguf) | 3.87 GB | 1.87% | +| Q5_K_M | [Voxtral-Mini-3B-2507-Q5_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q5_K_M.gguf) | 3.46 GB | 1.91% | +| Q4_K_M | [Voxtral-Mini-3B-2507-Q4_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-gguf/resolve/main/Voxtral-Mini-3B-2507-Q4_K_M.gguf) | 2.98 GB | 1.94% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Scored with the Whisper English text normalizer on an NVIDIA L40S. Same-machine +HuggingFace transformers reference (VoxtralForConditionalGeneration, BF16, +attn_implementation=eager, greedy): 1.87%; the BF16 GGUF matches within rounding. +The BF16-vs-reference parity is the family's tensor-level numerical gate — 43 +checkpointed tensors within tolerance, transcript byte-exact. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 4.71% | +| en | WER | 3.89% | +| es | WER | 3.52% | +| fr | WER | 4.51% | +| hi | WER | 8.93% | +| it | WER | 2.56% | +| nl | WER | 6.57% | +| pt | WER | 3.84% | + ## Quick Start @@ -81,41 +108,40 @@ CLI flags: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 727.3 ms (15.1×) | 656.8 ms (16.7×) | -| Metal | dots (35.3s) | 2.40 s (14.7×) | 1.90 s (18.6×) | -| CPU | jfk (11.0s) | 6.06 s (1.8×) | 6.76 s (1.6×) | -| CPU | dots (35.3s) | 16.60 s (2.1×) | 15.31 s (2.3×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 862.3 ms (12.76×) | 769.4 ms (14.30×) | +| Metal | dots (35.3s) | 2.51 s (14.08×) | 2.09 s (16.90×) | +| CPU | jfk (11.0s) | 5.83 s (1.89×) | 6.12 s (1.80×) | +| CPU | dots (35.3s) | 14.94 s (2.36×) | 14.18 s (2.49×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Vulkan | jfk (11.0s) | 9.57 s (1.2×) | 9.37 s (1.2×) | -| Vulkan | dots (35.3s) | 26.49 s (1.3×) | 23.82 s (1.5×) | -| CPU | jfk (11.0s) | 26.15 s (0.4×) | 19.39 s (0.6×) | -| CPU | dots (35.3s) | 63.96 s (0.6×) | 45.92 s (0.8×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -Fedora Linux 43, transcribe.cpp `91af262`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 11.11 s (0.99×) | 10.58 s (1.04×) | +| Vulkan | dots (35.3s) | 30.04 s (1.18×) | 27.46 s (1.29×) | +| CPU | jfk (11.0s) | 22.37 s (0.49×) | 20.88 s (0.53×) | +| CPU | dots (35.3s) | 55.99 s (0.63×) | 49.49 s (0.71×) | + +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models Voxtral-Mini-3B-2507 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name voxtral-mini-3b-2507-publication +uv run scripts/bench/run.py --profile --models voxtral-mini-3b-2507 ``` ## Numerical Validation @@ -148,11 +174,3 @@ are pinned in `tests/tolerances/voxtral.json`. | Tolerances | `tests/tolerances/voxtral.json` | | Command | `uv run scripts/validate.py all --family voxtral --variant voxtral-mini-3b-2507` | -Selected tensors (observed on CPU, strict backend; see tolerance file -for budgets): - -| Tensor | Shape | Max abs diff | Mean abs diff | Notes | -| --- | --- | ---: | ---: | --- | -| `enc.mel.in` | `[128,3000]` | `2.229e-05` | `4.060e-08` | In-process log-mel vs reference `WhisperFeatureExtractor` — the frontend-parity gate | -| `enc.out` | `[1500,1280]` | `1.414e+01` | `1.121e-02` | Final encoder LayerNorm; the drift here is the reference's BF16 activation rounding (cpp-vs-F32-ref is ~4× tighter) | -| `proj.out` | `[375,3072]` | `3.493e-01` | `3.158e-03` | Projector output (the audio embeddings injected into the LM) | diff --git a/docs/models/voxtral-realtime.md b/docs/models/voxtral-realtime.md index d1bdce8a..a8b4a88f 100644 --- a/docs/models/voxtral-realtime.md +++ b/docs/models/voxtral-realtime.md @@ -12,9 +12,11 @@ delay-token latency conditioning emits one text token per 80 ms audio slot Architecturally distinct from the offline [Voxtral 2507](voxtral.md) family (own arch, streaming frontend with a fixed global log-mel max, causal encoder, additive fusion, ada-norm FFN scaling) — it shares only the projector shape -and the tekken tokenizer. Licensed Apache-2.0. Ported from upstream commit -[`2769294`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602/commit/2769294), -pinned 2026-06-06. +and the tekken tokenizer. + + +Licensed Apache-2.0. Ported from upstream commit [`2769294`](https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602/commit/2769294), pinned 2026-06-06. Validated against the Transformers reference at transcribe.cpp commit [`483c122`](https://github.com/handy-computer/transcribe.cpp/tree/483c122) on 2026-06-06. + ## What it's for @@ -90,44 +92,40 @@ CLI flags: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). Measured on the offline path at the family-default `K=1` -speculative decoding. - ### Apple M4 Max + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Metal | jfk (11.0s) | 1.22 s (9.0×) | 1.14 s (9.7×) | -| Metal | dots (35.3s) | 4.34 s (8.1×) | 3.91 s (9.0×) | -| CPU | jfk (11.0s) | 4.43 s (2.5×) | 4.69 s (2.3×) | -| CPU | dots (35.3s) | 13.65 s (2.6×) | 13.12 s (2.7×) | +| Metal | jfk (11.0s) | 1.76 s (6.24×) | 1.49 s (7.36×) | +| Metal | dots (35.3s) | 5.03 s (7.03×) | 4.44 s (7.97×) | +| CPU | jfk (11.0s) | 4.88 s (2.25×) | 5.04 s (2.18×) | +| CPU | dots (35.3s) | 13.80 s (2.56×) | 13.20 s (2.68×) | -macOS 15, transcribe.cpp `483c122`. Metal device: Apple M4 Max. +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + ### AMD Ryzen 7 4750U Pro + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 12.62 s (0.87×) | 10.97 s (1.00×) | -| Vulkan | dots (35.3s) | 39.29 s (0.90×) | 33.51 s (1.05×) | -| CPU | jfk (11.0s) | 19.54 s (0.56×) | 13.80 s (0.80×) | -| CPU | dots (35.3s) | 58.00 s (0.61×) | 41.54 s (0.85×) | +| Vulkan | jfk (11.0s) | 14.95 s (0.74×) | 13.17 s (0.84×) | +| Vulkan | dots (35.3s) | 45.03 s (0.78×) | 39.16 s (0.90×) | +| CPU | jfk (11.0s) | 19.76 s (0.56×) | 16.39 s (0.67×) | +| CPU | dots (35.3s) | 57.92 s (0.61×) | 46.12 s (0.77×) | -Fedora 43, transcribe.cpp `483c122`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models Voxtral-Mini-4B-Realtime-2602 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu,vulkan \ - --iters 3 --warmup 1 \ - --name voxtral-mini-4b-realtime-2602-publication +uv run scripts/bench/run.py --profile --models voxtral-mini-4b-realtime-2602 ``` ## Speculative decoding diff --git a/docs/models/voxtral-small-24b-2507.md b/docs/models/voxtral-small-24b-2507.md index e23feb17..63c5e8b0 100644 --- a/docs/models/voxtral-small-24b-2507.md +++ b/docs/models/voxtral-small-24b-2507.md @@ -1,17 +1,15 @@ # Voxtral Small 24B (2507) -Mistral's [`mistralai/Voxtral-Small-24B-2507`](https://huggingface.co/mistralai/Voxtral-Small-24B-2507) -ported to transcribe.cpp. An offline audio-LLM: a Whisper-large-v3 -bidirectional audio encoder (32 layers, `d_model=1280`, 20 heads) feeds -a 4-frame-group projector (375 audio tokens per 30 s chunk) into a -Mistral-Small-24B causal LM (40 layers, `hidden_size=5120`, -`intermediate_size=32768`, GQA 32 q / 8 kv heads, NEOX RoPE, SwiGLU) via -audio-token injection at the `audio_token_id=24` positions in the prompt. - -It is the larger sibling of [Voxtral Mini 3B](voxtral-mini-3b-2507.md): -the audio encoder, projector pattern, log-mel frontend, and tekken -tokenizer are identical — only the text decoder is scaled up (Mistral-Small-24B -in place of Ministral-3B). + +Upstream: [`mistralai/Voxtral-Small-24B-2507`](https://huggingface.co/mistralai/Voxtral-Small-24B-2507) at [`da5b424`](https://huggingface.co/mistralai/Voxtral-Small-24B-2507/commit/da5b424). + +Offline audio-LLM speech-to-text and speech translation. A Whisper-large-v3 +bidirectional audio encoder feeds a 4-frame-group projector (375 audio tokens +per 30 s chunk) into a Mistral-Small-24B causal LM (40 layers, GQA 32/8, NEOX +RoPE, SwiGLU) via audio-token injection. Takes a 16 kHz mono WAV and produces a +transcript via greedy decoding. The larger sibling of Voxtral Mini 3B — same +encoder, projector, frontend, and tokenizer, with a scaled-up decoder. + ## What it's for @@ -28,26 +26,50 @@ mono WAV and produces a transcript via greedy decoding. See Mistral's [model card](https://huggingface.co/mistralai/Voxtral-Small-24B-2507) for training data, intended use, and upstream evaluation. -Licensed Apache-2.0. Ported from upstream commit -[`da5b424`](https://huggingface.co/mistralai/Voxtral-Small-24B-2507/commit/da5b42409f279fdd92febee0511a6c32828569c1), -pinned 2026-06-05. + +Licensed Apache-2.0. Ported from upstream commit [`da5b424`](https://huggingface.co/mistralai/Voxtral-Small-24B-2507/commit/da5b424), pinned 2026-06-05. Validated against the Transformers reference at transcribe.cpp commit [`dac22fa`](https://github.com/handy-computer/transcribe.cpp/tree/dac22fa) on 2026-06-05. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| BF16 | [Voxtral-Small-24B-2507-BF16.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-BF16.gguf) | 48.54 GB | 1.56% | -| F16 | [Voxtral-Small-24B-2507-F16.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-F16.gguf) | 48.55 GB | 1.57% | -| Q8_0 | [Voxtral-Small-24B-2507-Q8_0.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q8_0.gguf) | 25.81 GB | 1.56% | -| Q6_K | [Voxtral-Small-24B-2507-Q6_K.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q6_K.gguf) | 19.94 GB | 1.58% | -| Q5_K_M | [Voxtral-Small-24B-2507-Q5_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q5_K_M.gguf) | 17.14 GB | 1.60% | -| Q4_K_M | [Voxtral-Small-24B-2507-Q4_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q4_K_M.gguf) | 14.30 GB | 2.11% | - -WER measured on the full LibriSpeech `test-clean` split (2620 utterances) -with the Whisper-style English text normalizer, batch size 8 on an NVIDIA -A100 80 GB. The same-split HuggingFace `transformers` reference run -(`VoxtralForConditionalGeneration`, BF16, greedy) lands at **1.57%**, and -the BF16 GGUF matches it at **1.56%**. +| BF16 | [Voxtral-Small-24B-2507-BF16.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-BF16.gguf) | 48.54 GB | 1.56% | +| F16 | [Voxtral-Small-24B-2507-F16.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-F16.gguf) | 48.55 GB | 1.57% | +| Q8_0 | [Voxtral-Small-24B-2507-Q8_0.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q8_0.gguf) | 25.81 GB | 1.56% | +| Q6_K | [Voxtral-Small-24B-2507-Q6_K.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q6_K.gguf) | 19.94 GB | 1.58% | +| Q5_K_M | [Voxtral-Small-24B-2507-Q5_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q5_K_M.gguf) | 17.14 GB | 1.60% | +| Q4_K_M | [Voxtral-Small-24B-2507-Q4_K_M.gguf](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-gguf/resolve/main/Voxtral-Small-24B-2507-Q4_K_M.gguf) | 14.30 GB | 2.11% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 8, timestamps none. Figures without a commit were published before provenance was recorded. + + + +Greedy decoding scored with the Whisper English text normalizer on an NVIDIA A100 80 +GB. Same-machine HuggingFace transformers reference +(VoxtralForConditionalGeneration, BF16, greedy): 1.57%; the BF16 GGUF matches at +1.56%. Validation for this variant is end-to-end by WER — the family's tensor-level +numerical parity is established by the Voxtral Mini 3B sibling (identical +architecture). + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| de | WER | 3.29% | +| en | WER | 3.55% | +| es | WER | 2.86% | +| fr | WER | 3.86% | +| hi | WER | 7.40% | +| it | WER | 2.69% | +| nl | WER | 5.12% | +| pt | WER | 3.74% | + ## Quick Start @@ -87,16 +109,18 @@ CLI flags: ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 3.36 s (3.3×) | 2.62 s (4.2×) | -| Metal | dots (35.3s) | 11.20 s (3.2×) | 8.95 s (3.9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Metal | jfk (11.0s) | 4.45 s (2.47×) | 3.96 s (2.77×) | +| Metal | dots (35.3s) | 14.68 s (2.41×) | 12.93 s (2.73×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + A 24B is a GPU-class model; on Apple Silicon it runs at **~3–4× realtime** on Metal (the 3B sibling is ~15–18×). CPU is impractical at this size and is @@ -105,13 +129,7 @@ not benchmarked. transcribe.cpp `96adddb`. Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models Voxtral-Small-24B-2507 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal \ - --iters 3 --warmup 1 \ - --name voxtral-small-24b-2507-publication +uv run scripts/bench/run.py --profile --models voxtral-small-24b-2507 ``` ## Notes diff --git a/docs/models/voxtral.md b/docs/models/voxtral.md index eb8e4435..e7530683 100644 --- a/docs/models/voxtral.md +++ b/docs/models/voxtral.md @@ -17,14 +17,16 @@ For Mistral's **streaming** sibling, see ## Variants -| Variant | Text decoder | BF16 WER (test-clean) | Card | GGUF | -| --- | --- | ---: | --- | --- | -| `voxtral-mini-3b-2507` | Ministral-3B (30L, `d=3072`) | 1.88% | [card](voxtral-mini-3b-2507.md) | [HF](https://huggingface.co/handy-computer/Voxtral-Mini-3B-2507-GGUF) | -| `voxtral-small-24b-2507` | Mistral-Small-24B (40L, `d=5120`) | 1.56% | [card](voxtral-small-24b-2507.md) | [HF](https://huggingface.co/handy-computer/Voxtral-Small-24B-2507-GGUF) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `voxtral-mini-3b-2507` | 4.7B | 8 languages + auto-detect | 5.00 GB | LibriSpeech test-clean (WER) | 1.87% | translate | [voxtral-mini-3b-2507.md](voxtral-mini-3b-2507.md) | +| `voxtral-small-24b-2507` | 24.3B | 8 languages + auto-detect | 25.81 GB | LibriSpeech test-clean (WER) | 1.56% | translate | [voxtral-small-24b-2507.md](voxtral-small-24b-2507.md) | + WER on the full LibriSpeech `test-clean` split (2620 utterances), Whisper English normalizer. Both match the HuggingFace `transformers` reference -within rounding (3B 1.87%, 24B 1.57%). See each variant's card for the full +within rounding. See each variant's card for the full quant matrix, per-quant WER, and quick-start commands. ## Input limits diff --git a/docs/models/whisper-base.en.md b/docs/models/whisper-base.en.md index af23eece..1173ac0a 100644 --- a/docs/models/whisper-base.en.md +++ b/docs/models/whisper-base.en.md @@ -1,8 +1,10 @@ # Whisper base.en -OpenAI's [`openai/whisper-base.en`](https://huggingface.co/openai/whisper-base.en) ported to transcribe.cpp. A 74M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-base.en`](https://huggingface.co/openai/whisper-base.en) at [`911407f`](https://huggingface.co/openai/whisper-base.en/commit/911407f). + +OpenAI Whisper base.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,42 @@ Offline English speech-to-text. The model takes a 16 kHz mono WAV and returns a See the [upstream model card](https://huggingface.co/openai/whisper-base.en) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`911407f`](https://huggingface.co/openai/whisper-base.en/commit/911407f), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`911407f`](https://huggingface.co/openai/whisper-base.en/commit/911407f), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-base.en-F32.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-F32.gguf) | 279 MB | 4.14% | -| F16 | [whisper-base.en-F16.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-F16.gguf) | 144 MB | 4.13% | -| Q8_0 | [whisper-base.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q8_0.gguf) | 81 MB | 4.16% | -| Q6_K | [whisper-base.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q6_K.gguf) | 65 MB | 4.15% | -| Q5_K_M | [whisper-base.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q5_K_M.gguf) | 61 MB | 4.16% | -| Q4_K_M | [whisper-base.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q4_K_M.gguf) | 56 MB | 4.29% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-base.en-F32.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-F32.gguf) | 292 MB | 4.30% | +| F16 | [whisper-base.en-F16.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-F16.gguf) | 151 MB | 4.13% | +| Q8_0 | [whisper-base.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q8_0.gguf) | 85 MB | 4.16% | +| Q6_K | [whisper-base.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q6_K.gguf) | 68 MB | 4.15% | +| Q5_K_M | [whisper-base.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q5_K_M.gguf) | 64 MB | 4.16% | +| Q4_K_M | [whisper-base.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-base.en-gguf/resolve/main/whisper-base.en-Q4_K_M.gguf) | 59 MB | 4.29% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 4.25%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 7.60% | + ## Quick Start @@ -50,62 +69,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). Decode latency dominates as model size grows; the encoder -is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 50.3 ms (218.9×) | 49.5 ms (222.3×) | -| Metal | dots (35.3s) | 172.4 ms (204.9×) | 161.4 ms (218.9×) | -| CPU | jfk (11.0s) | 352.5 ms (31.2×) | 325.6 ms (33.8×) | -| CPU | dots (35.3s) | 780.9 ms (45.2×) | 733.6 ms (48.2×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -----------------: | -----------------: | +| Metal | jfk (11.0s) | 51.8 ms (212.45×) | 51.8 ms (212.42×) | +| Metal | dots (35.3s) | 185.2 ms (190.73×) | 181.2 ms (194.94×) | +| CPU | jfk (11.0s) | 173.6 ms (63.35×) | 195.3 ms (56.31×) | +| CPU | dots (35.3s) | 423.6 ms (83.40×) | 463.9 ms (76.16×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-base.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-base.en-publication +uv run scripts/bench/run.py --profile --models whisper-base.en ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 327 ms (33.6×) | 337 ms (32.7×) | -| Vulkan | dots (35.3s) | 909 ms (38.9×) | 912 ms (38.7×) | -| CPU | jfk (11.0s) | 1.01 s (10.8×) | 836 ms (13.2×) | -| CPU | dots (35.3s) | 2.41 s (14.7×) | 2.19 s (16.1×) | +| Vulkan | jfk (11.0s) | 376 ms (29.22×) | 385 ms (28.56×) | +| Vulkan | dots (35.3s) | 1.01 s (34.81×) | 1.06 s (33.38×) | +| CPU | jfk (11.0s) | 589 ms (18.68×) | 575 ms (19.14×) | +| CPU | dots (35.3s) | 1.57 s (22.52×) | 1.50 s (23.60×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-base.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-base.en-publication +uv run scripts/bench/run.py --profile --models whisper-base.en ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance. Tolerance budget lives at -[`tests/tolerances/whisper-base.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-base.en.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-base.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-base.en.json). | Field | Value | | --- | --- | @@ -114,24 +123,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-base.en.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-base.en` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `5.811e-06` | `6.342e-08` | fp32 conv stem | -| `enc.conv2.out` | `1.681e-05` | `2.860e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `2.146e-05` | `6.671e-07` | first encoder block | -| `enc.block.5.out` | `4.608e-02` | `2.940e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `6.485e-05` | `1.328e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `6.855e-06` | `2.809e-07` | first decoder block, prompt pass | -| `dec.block.5.out` | `2.861e-05` | `1.841e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `1.984e-04` | `1.569e-05` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `6.676e-05` | `9.382e-06` | vocab projection (raw logits) | -| `dec.logits` | `6.294e-05` | `2.236e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `7.248e-05` | `4.264e-05` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-base.md b/docs/models/whisper-base.md index a01e31ec..aed1660b 100644 --- a/docs/models/whisper-base.md +++ b/docs/models/whisper-base.md @@ -1,8 +1,10 @@ # Whisper base -OpenAI's [`openai/whisper-base`](https://huggingface.co/openai/whisper-base) ported to transcribe.cpp. A 74M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-base`](https://huggingface.co/openai/whisper-base) at [`e37978b`](https://huggingface.co/openai/whisper-base/commit/e37978b). + +OpenAI Whisper base — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-base) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`e37978b`](https://huggingface.co/openai/whisper-base/commit/e37978b), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`e37978b`](https://huggingface.co/openai/whisper-base/commit/e37978b), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-base-F32.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-F32.gguf) | 279 MB | 5.10% | -| F16 | [whisper-base-F16.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-F16.gguf) | 144 MB | 5.10% | -| Q8_0 | [whisper-base-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q8_0.gguf) | 81 MB | 5.12% | -| Q6_K | [whisper-base-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q6_K.gguf) | 65 MB | 5.12% | -| Q5_K_M | [whisper-base-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q5_K_M.gguf) | 61 MB | 5.19% | -| Q4_K_M | [whisper-base-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q4_K_M.gguf) | 56 MB | 5.36% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-base-F32.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-F32.gguf) | 292 MB | 5.11% | +| F16 | [whisper-base-F16.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-F16.gguf) | 151 MB | 5.10% | +| Q8_0 | [whisper-base-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q8_0.gguf) | 85 MB | 5.12% | +| Q6_K | [whisper-base-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q6_K.gguf) | 68 MB | 5.11% | +| Q5_K_M | [whisper-base-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q5_K_M.gguf) | 64 MB | 5.19% | +| Q4_K_M | [whisper-base-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-base-gguf/resolve/main/whisper-base-Q4_K_M.gguf) | 59 MB | 5.36% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 5.009%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 83.05% | +| am | WER | 150.97% | +| ar | WER | 52.74% | +| as | WER | 100.60% | +| az | WER | 81.22% | +| be | WER | 92.72% | +| bg | WER | 70.53% | +| bn | WER | 100.73% | +| bs | WER | 71.53% | +| ca | WER | 29.48% | +| cs | WER | 70.14% | +| cy | WER | 98.19% | +| da | WER | 63.85% | +| de | WER | 19.69% | +| el | WER | 59.14% | +| en | WER | 9.88% | +| es | WER | 11.15% | +| et | WER | 81.71% | +| fa | WER | 87.72% | +| fi | WER | 49.46% | +| fil | WER | 49.32% | +| fr | WER | 27.91% | +| gl | WER | 50.06% | +| gu | WER | 100.40% | +| ha | WER | 108.15% | +| he | WER | 65.56% | +| hi | WER | 100.01% | +| hr | WER | 64.23% | +| hu | WER | 72.26% | +| hy | WER | 127.56% | +| id | WER | 38.02% | +| is | WER | 99.32% | +| it | WER | 17.26% | +| ja | CER | 25.28% | +| jv | WER | 93.07% | +| ka | WER | 117.78% | +| kk | WER | 99.79% | +| km | CER | 134.48% | +| kn | WER | 102.88% | +| ko | CER | 12.98% | +| lb | WER | 107.78% | +| ln | WER | 102.73% | +| lo | CER | 104.35% | +| lt | WER | 91.78% | +| lv | WER | 84.60% | +| mi | WER | 81.65% | +| mk | WER | 63.95% | +| ml | WER | 102.84% | +| mn | WER | 124.42% | +| mr | WER | 100.42% | +| ms | WER | 40.87% | +| mt | WER | 103.46% | +| my | CER | 130.63% | +| nb | WER | 49.26% | +| ne | WER | 101.15% | +| nl | WER | 36.75% | +| oc | WER | 88.62% | +| pa | WER | 101.13% | +| pl | WER | 35.68% | +| ps | WER | 101.19% | +| pt | WER | 13.91% | +| ro | WER | 62.16% | +| ru | WER | 22.92% | +| sd | WER | 103.23% | +| sk | WER | 65.77% | +| sl | WER | 77.90% | +| sn | WER | 134.76% | +| so | WER | 107.06% | +| sr | WER | 69.25% | +| sv | WER | 42.40% | +| sw | WER | 100.69% | +| ta | WER | 58.84% | +| te | WER | 101.77% | +| tg | WER | 108.30% | +| th | CER | 38.10% | +| tr | WER | 31.09% | +| uk | WER | 42.03% | +| ur | WER | 55.42% | +| uz | WER | 111.42% | +| vi | WER | 42.60% | +| yo | WER | 103.28% | +| zh | CER | 36.21% | + ## Quick Start @@ -50,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ----------------: | ----------------: | -| Metal | jfk (11.0s) | 52.1 ms (211.0×) | 53.6 ms (205.2×) | -| Metal | dots (35.3s) | 170.0 ms (207.8×) | 168.3 ms (209.9×) | -| CPU | jfk (11.0s) | 374.0 ms (29.4×) | 347.6 ms (31.6×) | -| CPU | dots (35.3s) | 806.1 ms (43.8×) | 750.3 ms (47.1×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -----------------: | -----------------: | +| Metal | jfk (11.0s) | 54.5 ms (201.79×) | 54.6 ms (201.42×) | +| Metal | dots (35.3s) | 191.6 ms (184.36×) | 187.2 ms (188.77×) | +| CPU | jfk (11.0s) | 185.3 ms (59.35×) | 210.8 ms (52.18×) | +| CPU | dots (35.3s) | 436.6 ms (80.93×) | 482.5 ms (73.23×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-base \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-base-publication +uv run scripts/bench/run.py --profile --models whisper-base ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 351 ms (31.3×) | 356 ms (30.9×) | -| Vulkan | dots (35.3s) | 922 ms (38.3×) | 946 ms (37.4×) | -| CPU | jfk (11.0s) | 1.11 s (9.9×) | 913 ms (12.1×) | -| CPU | dots (35.3s) | 2.54 s (13.9×) | 2.27 s (15.5×) | +| Vulkan | jfk (11.0s) | 382 ms (28.80×) | 408 ms (27.00×) | +| Vulkan | dots (35.3s) | 1.21 s (29.23×) | 1.21 s (29.14×) | +| CPU | jfk (11.0s) | 646 ms (17.03×) | 635 ms (17.32×) | +| CPU | dots (35.3s) | 1.63 s (21.63×) | 1.56 s (22.63×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-base \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-base-publication +uv run scripts/bench/run.py --profile --models whisper-base ``` ## Numerical Validation -transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's cases (`samples/jfk.wav` and `samples/german.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcripts match the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-base.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-base.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcripts match the HF reference verbatim. Tolerance budget lives at +[`tests/tolerances/whisper-base.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-base.json). | Field | Value | | --- | --- | @@ -114,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-base.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-base` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `5.960e-06` | `6.348e-08` | fp32 conv stem | -| `enc.conv2.out` | `1.466e-05` | `2.544e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `1.794e-05` | `6.097e-07` | first encoder block | -| `enc.block.5.out` | `1.513e-02` | `5.212e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `1.699e-03` | `2.588e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `5.603e-06` | `2.874e-07` | first decoder block, prompt pass | -| `dec.block.5.out` | `1.373e-04` | `1.643e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `1.526e-04` | `9.441e-06` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `6.890e-05` | `1.462e-05` | vocab projection (raw logits) | -| `dec.logits` | `4.768e-05` | `9.442e-06` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `8.774e-05` | `5.206e-05` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-large-v2.md b/docs/models/whisper-large-v2.md index 974d4027..2c1ca677 100644 --- a/docs/models/whisper-large-v2.md +++ b/docs/models/whisper-large-v2.md @@ -1,8 +1,10 @@ # Whisper large-v2 -OpenAI's [`openai/whisper-large-v2`](https://huggingface.co/openai/whisper-large-v2) ported to transcribe.cpp. A 1.55B-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-large-v2`](https://huggingface.co/openai/whisper-large-v2) at [`ae46427`](https://huggingface.co/openai/whisper-large-v2/commit/ae46427). + +OpenAI Whisper large-v2 — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-large-v2) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`ae46427`](https://huggingface.co/openai/whisper-large-v2/commit/ae46427), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`ae46427`](https://huggingface.co/openai/whisper-large-v2/commit/ae46427), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-large-v2-F32.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-F32.gguf) | 5.75 GB | 2.68% | -| F16 | [whisper-large-v2-F16.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-F16.gguf) | 2.89 GB | 2.94% | -| Q8_0 | [whisper-large-v2-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q8_0.gguf) | 1.55 GB | 2.65% | -| Q6_K | [whisper-large-v2-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q6_K.gguf) | 1.21 GB | 2.83% | -| Q5_K_M | [whisper-large-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q5_K_M.gguf) | 1.08 GB | 2.72% | -| Q4_K_M | [whisper-large-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q4_K_M.gguf) | 950 MB | 2.46% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-large-v2-F32.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-F32.gguf) | 6.18 GB | 2.67% | +| F16 | [whisper-large-v2-F16.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-F16.gguf) | 3.11 GB | 2.68% | +| Q8_0 | [whisper-large-v2-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q8_0.gguf) | 1.67 GB | 2.97% | +| Q6_K | [whisper-large-v2-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q6_K.gguf) | 1.30 GB | 2.83% | +| Q5_K_M | [whisper-large-v2-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q5_K_M.gguf) | 1.16 GB | 2.71% | +| Q4_K_M | [whisper-large-v2-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v2-gguf/resolve/main/whisper-large-v2-Q4_K_M.gguf) | 997 MB | 2.46% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 2.83%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 38.45% | +| am | WER | 140.81% | +| ar | WER | 17.06% | +| as | WER | 104.58% | +| az | WER | 24.13% | +| be | WER | 46.96% | +| bg | WER | 15.81% | +| bn | WER | 103.42% | +| bs | WER | 17.02% | +| ca | WER | 5.56% | +| cs | WER | 14.42% | +| cy | WER | 30.55% | +| da | WER | 14.92% | +| de | WER | 4.53% | +| el | WER | 13.51% | +| en | WER | 4.21% | +| es | WER | 3.30% | +| et | WER | 23.25% | +| fa | WER | 34.25% | +| fi | WER | 9.58% | +| fil | WER | 13.17% | +| fr | WER | 5.81% | +| gl | WER | 16.57% | +| gu | WER | 103.37% | +| ha | WER | 92.22% | +| he | WER | 27.78% | +| hi | WER | 23.27% | +| hr | WER | 14.18% | +| hu | WER | 17.84% | +| hy | WER | 46.93% | +| id | WER | 7.43% | +| is | WER | 39.59% | +| it | WER | 3.59% | +| ja | CER | 5.56% | +| jv | WER | 69.69% | +| ka | WER | 115.24% | +| kk | WER | 40.13% | +| km | CER | 150.84% | +| kn | WER | 47.64% | +| ko | CER | 4.99% | +| lb | WER | 92.83% | +| ln | WER | 79.40% | +| lo | CER | 101.65% | +| lt | WER | 30.37% | +| lv | WER | 24.49% | +| mi | WER | 39.72% | +| mk | WER | 18.76% | +| ml | WER | 101.85% | +| mn | WER | 115.70% | +| mr | WER | 39.81% | +| ms | WER | 9.38% | +| mt | WER | 73.79% | +| my | CER | 149.11% | +| nb | WER | 9.73% | +| ne | WER | 47.74% | +| nl | WER | 6.76% | +| oc | WER | 75.41% | +| pa | WER | 102.11% | +| pl | WER | 5.87% | +| ps | WER | 94.98% | +| pt | WER | 4.40% | +| ro | WER | 17.10% | +| ru | WER | 5.61% | +| sd | WER | 148.34% | +| sk | WER | 12.49% | +| sl | WER | 24.73% | +| sn | WER | 127.84% | +| so | WER | 106.80% | +| sr | WER | 38.06% | +| sv | WER | 9.25% | +| sw | WER | 41.85% | +| ta | WER | 20.12% | +| te | WER | 100.76% | +| tg | WER | 89.42% | +| th | CER | 12.42% | +| tr | WER | 7.63% | +| uk | WER | 8.17% | +| ur | WER | 23.96% | +| uz | WER | 91.97% | +| vi | WER | 11.25% | +| yo | WER | 96.00% | +| zh | CER | 15.39% | + ## Quick Start @@ -50,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 508.0 ms (21.7×) | 498.3 ms (22.1×) | -| Metal | dots (35.3s) | 1.37 s (25.7×) | 1.33 s (26.5×) | -| CPU | jfk (11.0s) | 9.66 s (1.1×) | 7.46 s (1.5×) | -| CPU | dots (35.3s) | 19.72 s (1.8×) | 15.43 s (2.3×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 597.7 ms (18.40×) | 590.0 ms (18.65×) | +| Metal | dots (35.3s) | 1.60 s (22.08×) | 1.54 s (22.89×) | +| CPU | jfk (11.0s) | 3.73 s (2.95×) | 4.15 s (2.65×) | +| CPU | dots (35.3s) | 8.09 s (4.36×) | 8.64 s (4.09×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large-v2 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-large-v2-publication +uv run scripts/bench/run.py --profile --models whisper-large-v2 ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 6.27 s (1.8×) | 6.07 s (1.8×) | -| Vulkan | dots (35.3s) | 14.29 s (2.5×) | 13.68 s (2.6×) | -| CPU | jfk (11.0s) | 25.73 s (0.4×) | 19.46 s (0.6×) | -| CPU | dots (35.3s) | 53.75 s (0.7×) | 43.11 s (0.8×) | +| Vulkan | jfk (11.0s) | 6.48 s (1.70×) | 6.51 s (1.69×) | +| Vulkan | dots (35.3s) | 15.24 s (2.32×) | 15.05 s (2.35×) | +| CPU | jfk (11.0s) | 13.96 s (0.79×) | 13.69 s (0.80×) | +| CPU | dots (35.3s) | 31.19 s (1.13×) | 29.47 s (1.20×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large-v2 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-large-v2-publication +uv run scripts/bench/run.py --profile --models whisper-large-v2 ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcript matches the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-large-v2.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large-v2.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-large-v2.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large-v2.json). | Field | Value | | --- | --- | @@ -114,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-large-v2.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-large-v2` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `3.725e-06` | `2.701e-08` | fp32 conv stem | -| `enc.conv2.out` | `1.800e-05` | `4.611e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `5.424e-05` | `1.111e-06` | first encoder block | -| `enc.block.31.out` | `1.822e-02` | `3.259e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `1.213e-03` | `2.945e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `4.530e-06` | `2.545e-07` | first decoder block, prompt pass | -| `dec.block.31.out` | `6.104e-05` | `1.777e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `4.387e-05` | `2.792e-06` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `3.910e-05` | `9.921e-06` | vocab projection (raw logits) | -| `dec.logits` | `7.486e-05` | `1.652e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `9.537e-06` | `2.033e-06` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-large-v3-turbo.md b/docs/models/whisper-large-v3-turbo.md index 797a9d02..266d2029 100644 --- a/docs/models/whisper-large-v3-turbo.md +++ b/docs/models/whisper-large-v3-turbo.md @@ -1,8 +1,10 @@ # Whisper large-v3-turbo -OpenAI's [`openai/whisper-large-v3-turbo`](https://huggingface.co/openai/whisper-large-v3-turbo) ported to transcribe.cpp. A 809M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-large-v3-turbo`](https://huggingface.co/openai/whisper-large-v3-turbo) at [`41f01f3`](https://huggingface.co/openai/whisper-large-v3-turbo/commit/41f01f3). + +OpenAI Whisper large-v3-turbo — converted to GGUF for transcribe.cpp. Multilingual transcription and language detection; unlike the full large-v3 model, this turbo variant does not support speech translation. The v3 family adds Cantonese (yue) and uses a 128-bin mel input. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,24 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-large-v3-turbo) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`41f01f3`](https://huggingface.co/openai/whisper-large-v3-turbo/commit/41f01f3), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`41f01f3`](https://huggingface.co/openai/whisper-large-v3-turbo/commit/41f01f3), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F16 | [whisper-large-v3-turbo-F16.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-F16.gguf) | 1.51 GB | 2.01% | -| Q8_0 | [whisper-large-v3-turbo-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q8_0.gguf) | 845 MB | 2.01% | -| Q6_K | [whisper-large-v3-turbo-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q6_K.gguf) | 660 MB | 2.01% | -| Q5_K_M | [whisper-large-v3-turbo-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q5_K_M.gguf) | 591 MB | 2.03% | -| Q4_K_M | [whisper-large-v3-turbo-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q4_K_M.gguf) | 511 MB | 2.04% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F16 | [whisper-large-v3-turbo-F16.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-F16.gguf) | 1.63 GB | 2.01% | +| Q8_0 | [whisper-large-v3-turbo-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q8_0.gguf) | 886 MB | 2.01% | +| Q6_K | [whisper-large-v3-turbo-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q6_K.gguf) | 693 MB | 2.01% | +| Q5_K_M | [whisper-large-v3-turbo-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q5_K_M.gguf) | 620 MB | 2.03% | +| Q4_K_M | [whisper-large-v3-turbo-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-turbo-gguf/resolve/main/whisper-large-v3-turbo-Q4_K_M.gguf) | 536 MB | 2.04% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 2.10%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 36.06% | +| am | WER | 146.29% | +| ar | WER | 15.48% | +| as | WER | 101.22% | +| az | WER | 23.15% | +| be | WER | 50.65% | +| bg | WER | 13.58% | +| bn | WER | 67.53% | +| bs | WER | 14.77% | +| ca | WER | 5.42% | +| cs | WER | 11.81% | +| cy | WER | 36.42% | +| da | WER | 13.60% | +| de | WER | 4.54% | +| el | WER | 13.26% | +| en | WER | 4.38% | +| es | WER | 3.12% | +| et | WER | 18.44% | +| fa | WER | 30.56% | +| fi | WER | 8.29% | +| fil | WER | 12.08% | +| fr | WER | 5.51% | +| gl | WER | 12.76% | +| gu | WER | 78.95% | +| ha | WER | 97.24% | +| he | WER | 29.71% | +| hi | WER | 18.85% | +| hr | WER | 12.54% | +| hu | WER | 15.07% | +| hy | WER | 45.62% | +| id | WER | 7.20% | +| is | WER | 21.39% | +| it | WER | 2.77% | +| ja | CER | 4.82% | +| jv | WER | 53.80% | +| ka | WER | 109.21% | +| kk | WER | 21.27% | +| km | CER | 95.20% | +| kn | WER | 32.57% | +| ko | CER | 5.24% | +| lb | WER | 87.21% | +| ln | WER | 75.39% | +| lo | CER | 115.41% | +| lt | WER | 25.11% | +| lv | WER | 19.53% | +| mi | WER | 48.91% | +| mk | WER | 17.85% | +| ml | WER | 98.75% | +| mn | WER | 101.49% | +| mr | WER | 36.12% | +| ms | WER | 8.64% | +| mt | WER | 70.92% | +| my | CER | 121.67% | +| nb | WER | 9.10% | +| ne | WER | 43.15% | +| nl | WER | 5.98% | +| oc | WER | 70.94% | +| pa | WER | 99.53% | +| pl | WER | 5.81% | +| ps | WER | 91.81% | +| pt | WER | 4.17% | +| ro | WER | 10.90% | +| ru | WER | 5.93% | +| sd | WER | 122.10% | +| sk | WER | 10.21% | +| sl | WER | 20.56% | +| sn | WER | 110.94% | +| so | WER | 101.29% | +| sr | WER | 32.36% | +| sv | WER | 8.72% | +| sw | WER | 33.96% | +| ta | WER | 27.41% | +| te | WER | 63.03% | +| tg | WER | 106.06% | +| th | CER | 13.15% | +| tr | WER | 6.97% | +| uk | WER | 7.31% | +| ur | WER | 23.19% | +| uz | WER | 102.52% | +| vi | WER | 9.48% | +| yo | WER | 99.38% | +| yue | CER | 34.62% | +| zh | CER | 8.50% | + ## Quick Start @@ -49,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 286.1 ms (38.4×) | 288.7 ms (38.1×) | -| Metal | dots (35.3s) | 649.5 ms (54.4×) | 641.1 ms (55.1×) | -| CPU | jfk (11.0s) | 7.60 s (1.4×) | 5.89 s (1.9×) | -| CPU | dots (35.3s) | 15.34 s (2.3×) | 11.87 s (3.0×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 303.8 ms (36.21×) | 291.1 ms (37.79×) | +| Metal | dots (35.3s) | 691.2 ms (51.12×) | 666.7 ms (53.00×) | +| CPU | jfk (11.0s) | 2.85 s (3.85×) | 3.11 s (3.54×) | +| CPU | dots (35.3s) | 5.80 s (6.09×) | 6.30 s (5.61×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large-v3-turbo \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-large-v3-turbo-publication +uv run scripts/bench/run.py --profile --models whisper-large-v3-turbo ``` ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | -------------: | ------------: | -| Vulkan | jfk (11.0s) | 4.14 s (2.7×) | 4.15 s (2.7×) | -| Vulkan | dots (35.3s) | 8.70 s (4.1×) | 8.88 s (4.0×) | -| CPU | jfk (11.0s) | 19.85 s (0.6×) | 15.74 s (0.7×) | -| CPU | dots (35.3s) | 40.18 s (0.9×) | 32.22 s (1.1×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 4.40 s (2.50×) | 4.45 s (2.47×) | +| Vulkan | dots (35.3s) | 9.59 s (3.69×) | 9.67 s (3.65×) | +| CPU | jfk (11.0s) | 9.85 s (1.12×) | 9.58 s (1.15×) | +| CPU | dots (35.3s) | 20.36 s (1.74×) | 19.72 s (1.79×) | -Fedora 43, transcribe.cpp `2ab01b8`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `cd0ea568` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large-v3-turbo \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-large-v3-turbo-publication +uv run scripts/bench/run.py --profile --models whisper-large-v3-turbo ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 22 checkpointed tensors fall within per-variant tolerance, and the transcript matches the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-large-v3-turbo.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large-v3-turbo.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-large-v3-turbo.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large-v3-turbo.json). | Field | Value | | --- | --- | @@ -113,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-large-v3-turbo.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-large-v3-turbo` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `4.055e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `1.258e-03` | `4.264e-05` | fp32 conv stem | -| `enc.conv2.out` | `2.696e-03` | `5.033e-05` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `1.439e-02` | `1.334e-03` | first encoder block | -| `enc.block.31.out` | `2.335e+00` | `3.895e-03` | final encoder block (peak signal grows with depth) | -| `enc.final` | `1.343e+00` | `7.386e-03` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `8.217e-03` | `1.484e-04` | first decoder block, prompt pass | -| `dec.block.3.out` | `1.453e-02` | `9.290e-04` | final decoder block (accumulated) | -| `dec.out_before_head` | `2.122e-01` | `2.041e-02` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `1.462e-01` | `2.968e-02` | vocab projection (raw logits) | -| `dec.logits` | `1.771e-01` | `3.328e-02` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `3.806e-02` | `6.097e-03` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-large-v3.md b/docs/models/whisper-large-v3.md index 5fc3fd02..538cc43a 100644 --- a/docs/models/whisper-large-v3.md +++ b/docs/models/whisper-large-v3.md @@ -1,8 +1,10 @@ # Whisper large-v3 -OpenAI's [`openai/whisper-large-v3`](https://huggingface.co/openai/whisper-large-v3) ported to transcribe.cpp. A 1.55B-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-large-v3`](https://huggingface.co/openai/whisper-large-v3) at [`06f233f`](https://huggingface.co/openai/whisper-large-v3/commit/06f233f). + +OpenAI Whisper large-v3 — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). v3 family adds Cantonese (yue) and uses a 128-bin mel input. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,24 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-large-v3) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`06f233f`](https://huggingface.co/openai/whisper-large-v3/commit/06f233f), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`06f233f`](https://huggingface.co/openai/whisper-large-v3/commit/06f233f), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F16 | [whisper-large-v3-F16.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-F16.gguf) | 2.88 GB | 1.81% | -| Q8_0 | [whisper-large-v3-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q8_0.gguf) | 1.55 GB | 1.82% | -| Q6_K | [whisper-large-v3-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q6_K.gguf) | 1.21 GB | 1.83% | -| Q5_K_M | [whisper-large-v3-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q5_K_M.gguf) | 1.08 GB | 1.84% | -| Q4_K_M | [whisper-large-v3-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q4_K_M.gguf) | 951 MB | 1.86% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F16 | [whisper-large-v3-F16.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-F16.gguf) | 3.11 GB | 1.81% | +| Q8_0 | [whisper-large-v3-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q8_0.gguf) | 1.67 GB | 1.82% | +| Q6_K | [whisper-large-v3-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q6_K.gguf) | 1.30 GB | 1.83% | +| Q5_K_M | [whisper-large-v3-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q5_K_M.gguf) | 1.16 GB | 1.84% | +| Q4_K_M | [whisper-large-v3-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-v3-gguf/resolve/main/whisper-large-v3-Q4_K_M.gguf) | 997 MB | 1.86% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 2.01%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 32.43% | +| am | WER | 132.70% | +| ar | WER | 14.92% | +| as | WER | 104.77% | +| az | WER | 21.10% | +| be | WER | 43.78% | +| bg | WER | 12.82% | +| bn | WER | 55.03% | +| bs | WER | 13.50% | +| ca | WER | 4.97% | +| cs | WER | 10.50% | +| cy | WER | 30.67% | +| da | WER | 12.48% | +| de | WER | 4.13% | +| el | WER | 11.53% | +| en | WER | 4.03% | +| es | WER | 2.70% | +| et | WER | 18.30% | +| fa | WER | 30.11% | +| fi | WER | 7.73% | +| fil | WER | 11.82% | +| fr | WER | 5.39% | +| gl | WER | 13.27% | +| gu | WER | 66.38% | +| ha | WER | 85.65% | +| he | WER | 26.73% | +| hi | WER | 17.06% | +| hr | WER | 10.94% | +| hu | WER | 13.40% | +| hy | WER | 43.64% | +| id | WER | 6.08% | +| is | WER | 31.85% | +| it | WER | 2.54% | +| ja | CER | 4.81% | +| jv | WER | 64.72% | +| ka | WER | 93.97% | +| kk | WER | 33.07% | +| km | CER | 101.09% | +| kn | WER | 31.99% | +| ko | CER | 4.89% | +| lb | WER | 85.67% | +| ln | WER | 72.22% | +| lo | CER | 100.28% | +| lt | WER | 24.55% | +| lv | WER | 19.21% | +| mi | WER | 38.78% | +| mk | WER | 15.09% | +| ml | WER | 100.17% | +| mn | WER | 85.60% | +| mr | WER | 34.30% | +| ms | WER | 7.59% | +| mt | WER | 68.79% | +| my | CER | 143.39% | +| nb | WER | 8.19% | +| ne | WER | 40.44% | +| nl | WER | 5.42% | +| oc | WER | 69.15% | +| pa | WER | 57.39% | +| pl | WER | 4.69% | +| ps | WER | 89.14% | +| pt | WER | 3.88% | +| ro | WER | 9.20% | +| ru | WER | 4.96% | +| sd | WER | 184.20% | +| sk | WER | 9.25% | +| sl | WER | 19.15% | +| sn | WER | 115.38% | +| so | WER | 91.10% | +| sr | WER | 28.49% | +| sv | WER | 7.80% | +| sw | WER | 34.57% | +| ta | WER | 20.04% | +| te | WER | 65.84% | +| tg | WER | 80.86% | +| th | CER | 8.78% | +| tr | WER | 6.51% | +| uk | WER | 6.28% | +| ur | WER | 21.75% | +| uz | WER | 86.28% | +| vi | WER | 8.74% | +| yo | WER | 97.26% | +| yue | CER | 22.06% | +| zh | CER | 7.98% | + ## Quick Start @@ -49,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 508.6 ms (21.6×) | 517.0 ms (21.3×) | -| Metal | dots (35.3s) | 1.38 s (25.7×) | 1.35 s (26.1×) | -| CPU | jfk (11.0s) | 9.68 s (1.1×) | 7.48 s (1.5×) | -| CPU | dots (35.3s) | 19.86 s (1.8×) | 15.45 s (2.3×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 744.8 ms (14.77×) | 616.5 ms (17.84×) | +| Metal | dots (35.3s) | 1.79 s (19.74×) | 1.58 s (22.41×) | +| CPU | jfk (11.0s) | 3.73 s (2.95×) | 4.14 s (2.66×) | +| CPU | dots (35.3s) | 8.07 s (4.38×) | 8.72 s (4.05×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large-v3 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-large-v3-publication +uv run scripts/bench/run.py --profile --models whisper-large-v3 ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 6.30 s (1.7×) | 6.07 s (1.8×) | -| Vulkan | dots (35.3s) | 14.42 s (2.5×) | 13.75 s (2.6×) | -| CPU | jfk (11.0s) | 25.59 s (0.4×) | 19.96 s (0.6×) | -| CPU | dots (35.3s) | 53.80 s (0.7×) | 43.18 s (0.8×) | +| Vulkan | jfk (11.0s) | 6.60 s (1.67×) | 6.63 s (1.66×) | +| Vulkan | dots (35.3s) | 15.20 s (2.32×) | 15.05 s (2.35×) | +| CPU | jfk (11.0s) | 13.60 s (0.81×) | 12.95 s (0.85×) | +| CPU | dots (35.3s) | 31.24 s (1.13×) | 28.10 s (1.26×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large-v3 \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-large-v3-publication +uv run scripts/bench/run.py --profile --models whisper-large-v3 ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcript matches the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-large-v3.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large-v3.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-large-v3.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large-v3.json). | Field | Value | | --- | --- | @@ -113,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-large-v3.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-large-v3` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `4.055e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `1.574e-03` | `4.796e-05` | fp32 conv stem | -| `enc.conv2.out` | `4.495e-03` | `6.128e-05` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `1.093e-02` | `1.163e-03` | first encoder block | -| `enc.block.31.out` | `1.439e+00` | `1.996e-03` | final encoder block (peak signal grows with depth) | -| `enc.final` | `1.332e+00` | `2.135e-03` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `1.465e-02` | `6.079e-04` | first decoder block, prompt pass | -| `dec.block.31.out` | `4.006e-01` | `1.022e-02` | final decoder block (accumulated) | -| `dec.out_before_head` | `1.770e-01` | `1.271e-02` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `1.371e-01` | `2.566e-02` | vocab projection (raw logits) | -| `dec.logits` | `2.347e-01` | `1.833e-02` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `2.925e-02` | `5.238e-03` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-large.md b/docs/models/whisper-large.md index 8306a0a3..ba75f67b 100644 --- a/docs/models/whisper-large.md +++ b/docs/models/whisper-large.md @@ -1,8 +1,10 @@ # Whisper large -OpenAI's [`openai/whisper-large`](https://huggingface.co/openai/whisper-large) ported to transcribe.cpp. A 1.55B-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-large`](https://huggingface.co/openai/whisper-large) at [`4ef9b41`](https://huggingface.co/openai/whisper-large/commit/4ef9b41). + +OpenAI Whisper large — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-large) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`4ef9b41`](https://huggingface.co/openai/whisper-large/commit/4ef9b41), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`4ef9b41`](https://huggingface.co/openai/whisper-large/commit/4ef9b41), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-large-F32.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-F32.gguf) | 5.75 GB | 2.72% | -| F16 | [whisper-large-F16.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-F16.gguf) | 2.89 GB | 2.74% | -| Q8_0 | [whisper-large-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q8_0.gguf) | 1.55 GB | 2.74% | -| Q6_K | [whisper-large-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q6_K.gguf) | 1.21 GB | 2.62% | -| Q5_K_M | [whisper-large-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q5_K_M.gguf) | 1.08 GB | 2.70% | -| Q4_K_M | [whisper-large-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q4_K_M.gguf) | 950 MB | 2.67% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-large-F32.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-F32.gguf) | 6.18 GB | 2.72% | +| F16 | [whisper-large-F16.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-F16.gguf) | 3.11 GB | 2.72% | +| Q8_0 | [whisper-large-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q8_0.gguf) | 1.67 GB | 2.71% | +| Q6_K | [whisper-large-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q6_K.gguf) | 1.30 GB | 2.62% | +| Q5_K_M | [whisper-large-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q5_K_M.gguf) | 1.16 GB | 2.84% | +| Q4_K_M | [whisper-large-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-large-gguf/resolve/main/whisper-large-Q4_K_M.gguf) | 997 MB | 2.67% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 2.73%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 44.92% | +| am | WER | 133.86% | +| ar | WER | 19.47% | +| as | WER | 105.09% | +| az | WER | 30.38% | +| be | WER | 58.26% | +| bg | WER | 19.89% | +| bn | WER | 105.08% | +| bs | WER | 22.20% | +| ca | WER | 7.01% | +| cs | WER | 18.77% | +| cy | WER | 36.33% | +| da | WER | 17.82% | +| de | WER | 5.04% | +| el | WER | 18.79% | +| en | WER | 4.46% | +| es | WER | 3.55% | +| et | WER | 27.31% | +| fa | WER | 37.88% | +| fi | WER | 12.73% | +| fil | WER | 15.44% | +| fr | WER | 6.95% | +| gl | WER | 20.10% | +| gu | WER | 104.29% | +| ha | WER | 90.12% | +| he | WER | 30.18% | +| hi | WER | 29.10% | +| hr | WER | 18.32% | +| hu | WER | 21.97% | +| hy | WER | 54.73% | +| id | WER | 8.93% | +| is | WER | 49.76% | +| it | WER | 3.67% | +| ja | CER | 6.95% | +| jv | WER | 86.31% | +| ka | WER | 119.65% | +| kk | WER | 47.29% | +| km | CER | 118.15% | +| kn | WER | 74.18% | +| ko | CER | 5.25% | +| lb | WER | 91.22% | +| ln | WER | 81.13% | +| lo | CER | 102.35% | +| lt | WER | 37.58% | +| lv | WER | 30.18% | +| mi | WER | 54.38% | +| mk | WER | 22.81% | +| ml | WER | 101.20% | +| mn | WER | 115.92% | +| mr | WER | 48.22% | +| ms | WER | 11.28% | +| mt | WER | 84.11% | +| my | CER | 128.00% | +| nb | WER | 11.98% | +| ne | WER | 55.71% | +| nl | WER | 8.48% | +| oc | WER | 76.72% | +| pa | WER | 102.98% | +| pl | WER | 7.44% | +| ps | WER | 98.52% | +| pt | WER | 4.49% | +| ro | WER | 17.52% | +| ru | WER | 6.49% | +| sd | WER | 176.25% | +| sk | WER | 17.17% | +| sl | WER | 29.94% | +| sn | WER | 142.46% | +| so | WER | 105.45% | +| sr | WER | 35.57% | +| sv | WER | 11.50% | +| sw | WER | 52.38% | +| ta | WER | 21.70% | +| te | WER | 99.27% | +| tg | WER | 79.55% | +| th | CER | 13.88% | +| tr | WER | 8.73% | +| uk | WER | 9.87% | +| ur | WER | 26.28% | +| uz | WER | 96.56% | +| vi | WER | 11.51% | +| yo | WER | 113.72% | +| zh | CER | 19.32% | + ## Quick Start @@ -50,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 476.5 ms (23.1×) | 465.1 ms (23.6×) | -| Metal | dots (35.3s) | 1.33 s (26.6×) | 1.26 s (28.0×) | -| CPU | jfk (11.0s) | 9.63 s (1.1×) | 7.43 s (1.5×) | -| CPU | dots (35.3s) | 19.88 s (1.8×) | 15.49 s (2.3×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 596.0 ms (18.46×) | 562.3 ms (19.56×) | +| Metal | dots (35.3s) | 1.63 s (21.63×) | 1.42 s (24.91×) | +| CPU | jfk (11.0s) | 3.70 s (2.97×) | 4.07 s (2.71×) | +| CPU | dots (35.3s) | 8.05 s (4.39×) | 8.90 s (3.97×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-large-publication +uv run scripts/bench/run.py --profile --models whisper-large ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 6.26 s (1.8×) | 6.13 s (1.8×) | -| Vulkan | dots (35.3s) | 14.41 s (2.5×) | 13.72 s (2.6×) | -| CPU | jfk (11.0s) | 26.18 s (0.4×) | 19.83 s (0.6×) | -| CPU | dots (35.3s) | 55.64 s (0.6×) | 43.98 s (0.8×) | +| Vulkan | jfk (11.0s) | 6.48 s (1.70×) | 6.51 s (1.69×) | +| Vulkan | dots (35.3s) | 15.31 s (2.31×) | 15.11 s (2.34×) | +| CPU | jfk (11.0s) | 14.00 s (0.79×) | 13.57 s (0.81×) | +| CPU | dots (35.3s) | 31.33 s (1.13×) | 29.55 s (1.20×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-large \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-large-publication +uv run scripts/bench/run.py --profile --models whisper-large ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcript matches the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-large.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-large.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-large.json). | Field | Value | | --- | --- | @@ -114,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-large.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-large` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `2.146e-06` | `3.695e-08` | fp32 conv stem | -| `enc.conv2.out` | `2.098e-05` | `3.419e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `2.718e-05` | `8.861e-07` | first encoder block | -| `enc.block.31.out` | `3.262e-01` | `7.382e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `1.916e-03` | `5.219e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `1.907e-05` | `3.328e-07` | first decoder block, prompt pass | -| `dec.block.31.out` | `2.441e-04` | `6.015e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `2.522e-04` | `1.035e-05` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `9.346e-05` | `1.508e-05` | vocab projection (raw logits) | -| `dec.logits` | `8.965e-05` | `2.757e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `3.052e-05` | `1.148e-05` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-medium.en.md b/docs/models/whisper-medium.en.md index 3842e1d1..b9b78d06 100644 --- a/docs/models/whisper-medium.en.md +++ b/docs/models/whisper-medium.en.md @@ -1,8 +1,10 @@ # Whisper medium.en -OpenAI's [`openai/whisper-medium.en`](https://huggingface.co/openai/whisper-medium.en) ported to transcribe.cpp. A 769M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-medium.en`](https://huggingface.co/openai/whisper-medium.en) at [`2e98eb6`](https://huggingface.co/openai/whisper-medium.en/commit/2e98eb6). + +OpenAI Whisper medium.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,42 @@ Offline English speech-to-text. The model takes a 16 kHz mono WAV and returns a See the [upstream model card](https://huggingface.co/openai/whisper-medium.en) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`2e98eb6`](https://huggingface.co/openai/whisper-medium.en/commit/2e98eb6), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`2e98eb6`](https://huggingface.co/openai/whisper-medium.en/commit/2e98eb6), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-medium.en-F32.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-F32.gguf) | 2.85 GB | 2.74% | -| F16 | [whisper-medium.en-F16.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-F16.gguf) | 1.44 GB | 2.73% | -| Q8_0 | [whisper-medium.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q8_0.gguf) | 793 MB | 2.72% | -| Q6_K | [whisper-medium.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q6_K.gguf) | 618 MB | 2.83% | -| Q5_K_M | [whisper-medium.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q5_K_M.gguf) | 556 MB | 2.74% | -| Q4_K_M | [whisper-medium.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q4_K_M.gguf) | 481 MB | 2.91% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-medium.en-F32.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-F32.gguf) | 3.06 GB | 2.74% | +| F16 | [whisper-medium.en-F16.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-F16.gguf) | 1.54 GB | 2.73% | +| Q8_0 | [whisper-medium.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q8_0.gguf) | 831 MB | 2.72% | +| Q6_K | [whisper-medium.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q6_K.gguf) | 648 MB | 2.82% | +| Q5_K_M | [whisper-medium.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q5_K_M.gguf) | 583 MB | 2.75% | +| Q4_K_M | [whisper-medium.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium.en-gguf/resolve/main/whisper-medium.en-Q4_K_M.gguf) | 504 MB | 2.91% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 3.02%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 4.88% | + ## Quick Start @@ -50,62 +69,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 249.7 ms (44.0×) | 243.3 ms (45.2×) | -| Metal | dots (35.3s) | 762.9 ms (46.3×) | 725.9 ms (48.7×) | -| CPU | jfk (11.0s) | 4.29 s (2.6×) | 3.37 s (3.3×) | -| CPU | dots (35.3s) | 9.07 s (3.9×) | 7.23 s (4.9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 291.6 ms (37.73×) | 270.2 ms (40.72×) | +| Metal | dots (35.3s) | 912.8 ms (38.71×) | 1.27 s (27.90×) | +| CPU | jfk (11.0s) | 1.75 s (6.29×) | 1.89 s (5.83×) | +| CPU | dots (35.3s) | 3.89 s (9.09×) | 4.15 s (8.51×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-medium.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-medium.en-publication +uv run scripts/bench/run.py --profile --models whisper-medium.en ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 2.74 s (4.0×) | 2.55 s (4.3×) | -| Vulkan | dots (35.3s) | 6.76 s (5.2×) | 6.44 s (5.5×) | -| CPU | jfk (11.0s) | 11.53 s (1.0×) | 9.36 s (1.2×) | -| CPU | dots (35.3s) | 26.63 s (1.3×) | 21.07 s (1.7×) | +| Vulkan | jfk (11.0s) | 2.71 s (4.07×) | 2.65 s (4.16×) | +| Vulkan | dots (35.3s) | 7.46 s (4.74×) | 7.06 s (5.01×) | +| CPU | jfk (11.0s) | 6.20 s (1.77×) | 5.94 s (1.85×) | +| CPU | dots (35.3s) | 14.99 s (2.36×) | 14.01 s (2.52×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-medium.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-medium.en-publication +uv run scripts/bench/run.py --profile --models whisper-medium.en ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance. Tolerance budget lives at -[`tests/tolerances/whisper-medium.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-medium.en.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-medium.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-medium.en.json). | Field | Value | | --- | --- | @@ -114,24 +123,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-medium.en.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-medium.en` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `4.679e-06` | `3.732e-08` | fp32 conv stem | -| `enc.conv2.out` | `1.380e-05` | `3.135e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `1.717e-05` | `8.644e-07` | first encoder block | -| `enc.block.23.out` | `4.858e-02` | `5.993e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `2.327e-03` | `4.378e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `5.245e-06` | `2.235e-07` | first decoder block, prompt pass | -| `dec.block.23.out` | `1.793e-04` | `5.787e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `2.990e-04` | `1.923e-05` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `4.959e-05` | `6.956e-06` | vocab projection (raw logits) | -| `dec.logits` | `8.297e-05` | `2.911e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `3.433e-05` | `8.558e-06` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-medium.md b/docs/models/whisper-medium.md index 62b2c725..878b2d44 100644 --- a/docs/models/whisper-medium.md +++ b/docs/models/whisper-medium.md @@ -1,8 +1,10 @@ # Whisper medium -OpenAI's [`openai/whisper-medium`](https://huggingface.co/openai/whisper-medium) ported to transcribe.cpp. A 769M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-medium`](https://huggingface.co/openai/whisper-medium) at [`abdf7c3`](https://huggingface.co/openai/whisper-medium/commit/abdf7c3). + +OpenAI Whisper medium — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-medium) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`abdf7c3`](https://huggingface.co/openai/whisper-medium/commit/abdf7c3), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`abdf7c3`](https://huggingface.co/openai/whisper-medium/commit/abdf7c3), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-medium-F32.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-F32.gguf) | 2.85 GB | 2.63% | -| F16 | [whisper-medium-F16.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-F16.gguf) | 1.44 GB | 2.63% | -| Q8_0 | [whisper-medium-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q8_0.gguf) | 793 MB | 2.64% | -| Q6_K | [whisper-medium-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q6_K.gguf) | 618 MB | 2.59% | -| Q5_K_M | [whisper-medium-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q5_K_M.gguf) | 556 MB | 2.62% | -| Q4_K_M | [whisper-medium-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q4_K_M.gguf) | 481 MB | 2.59% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-medium-F32.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-F32.gguf) | 3.06 GB | 2.64% | +| F16 | [whisper-medium-F16.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-F16.gguf) | 1.54 GB | 2.63% | +| Q8_0 | [whisper-medium-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q8_0.gguf) | 832 MB | 2.64% | +| Q6_K | [whisper-medium-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q6_K.gguf) | 648 MB | 2.59% | +| Q5_K_M | [whisper-medium-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q5_K_M.gguf) | 583 MB | 2.62% | +| Q4_K_M | [whisper-medium-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-medium-gguf/resolve/main/whisper-medium-Q4_K_M.gguf) | 504 MB | 2.59% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 2.90%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 47.33% | +| am | WER | 116.27% | +| ar | WER | 21.90% | +| as | WER | 101.63% | +| az | WER | 34.57% | +| be | WER | 61.87% | +| bg | WER | 23.00% | +| bn | WER | 102.52% | +| bs | WER | 26.69% | +| ca | WER | 8.02% | +| cs | WER | 22.85% | +| cy | WER | 40.08% | +| da | WER | 21.17% | +| de | WER | 6.23% | +| el | WER | 20.06% | +| en | WER | 4.64% | +| es | WER | 3.80% | +| et | WER | 31.52% | +| fa | WER | 42.57% | +| fi | WER | 14.67% | +| fil | WER | 18.36% | +| fr | WER | 8.07% | +| gl | WER | 22.24% | +| gu | WER | 104.11% | +| ha | WER | 95.12% | +| he | WER | 33.69% | +| hi | WER | 26.09% | +| hr | WER | 21.19% | +| hu | WER | 26.07% | +| hy | WER | 58.42% | +| id | WER | 10.79% | +| is | WER | 51.60% | +| it | WER | 4.17% | +| ja | CER | 7.35% | +| jv | WER | 73.01% | +| ka | WER | 128.01% | +| kk | WER | 53.09% | +| km | CER | 108.92% | +| kn | WER | 87.30% | +| ko | CER | 5.46% | +| lb | WER | 98.33% | +| ln | WER | 92.09% | +| lo | CER | 101.16% | +| lt | WER | 43.34% | +| lv | WER | 33.58% | +| mi | WER | 95.79% | +| mk | WER | 24.75% | +| ml | WER | 101.00% | +| mn | WER | 110.55% | +| mr | WER | 58.43% | +| ms | WER | 13.23% | +| mt | WER | 85.06% | +| my | CER | 117.51% | +| nb | WER | 13.66% | +| ne | WER | 54.08% | +| nl | WER | 10.40% | +| oc | WER | 81.55% | +| pa | WER | 103.19% | +| pl | WER | 8.59% | +| ps | WER | 105.77% | +| pt | WER | 5.07% | +| ro | WER | 24.17% | +| ru | WER | 7.30% | +| sd | WER | 132.27% | +| sk | WER | 18.73% | +| sl | WER | 33.98% | +| sn | WER | 134.28% | +| so | WER | 102.63% | +| sr | WER | 55.14% | +| sv | WER | 12.47% | +| sw | WER | 57.09% | +| ta | WER | 23.72% | +| te | WER | 102.69% | +| tg | WER | 76.32% | +| th | CER | 16.08% | +| tr | WER | 9.35% | +| uk | WER | 11.59% | +| ur | WER | 28.67% | +| uz | WER | 115.02% | +| vi | WER | 13.74% | +| yo | WER | 109.00% | +| zh | CER | 13.13% | + ## Quick Start @@ -50,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). Decode latency dominates as model size grows; the encoder -is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 280.0 ms (39.3×) | 270.9 ms (40.6×) | -| Metal | dots (35.3s) | 802.5 ms (44.0×) | 759.2 ms (46.5×) | -| CPU | jfk (11.0s) | 4.75 s (2.3×) | 3.90 s (2.8×) | -| CPU | dots (35.3s) | 9.62 s (3.7×) | 7.93 s (4.5×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 343.4 ms (32.03×) | 325.9 ms (33.76×) | +| Metal | dots (35.3s) | 1.02 s (34.62×) | 891.0 ms (39.65×) | +| CPU | jfk (11.0s) | 1.88 s (5.84×) | 2.08 s (5.28×) | +| CPU | dots (35.3s) | 4.08 s (8.66×) | 4.40 s (8.02×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-medium \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-medium-publication +uv run scripts/bench/run.py --profile --models whisper-medium ``` ### AMD Ryzen 7 PRO 4750U -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | -------------: | ------------: | -| Vulkan | jfk (11.0s) | 3.00 s (3.7×) | 2.83 s (3.9×) | -| Vulkan | dots (35.3s) | 7.03 s (5.0×) | 6.77 s (5.2×) | -| CPU | jfk (11.0s) | 13.14 s (0.8×) | 10.47 s (1.1×) | -| CPU | dots (35.3s) | 27.87 s (1.3×) | 22.57 s (1.6×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | --------------: | --------------: | +| Vulkan | jfk (11.0s) | 3.04 s (3.62×) | 3.01 s (3.65×) | +| Vulkan | dots (35.3s) | 7.77 s (4.55×) | 7.40 s (4.77×) | +| CPU | jfk (11.0s) | 6.93 s (1.59×) | 6.65 s (1.65×) | +| CPU | dots (35.3s) | 15.60 s (2.26×) | 14.67 s (2.41×) | -Fedora 43, transcribe.cpp `2ab01b8`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-medium \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-medium-publication +uv run scripts/bench/run.py --profile --models whisper-medium ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's cases (`samples/jfk.wav` and `samples/german.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcripts match the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-medium.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-medium.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-medium.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-medium.json). | Field | Value | | --- | --- | @@ -114,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-medium.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-medium` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `3.946e-05` | `1.396e-07` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `6.765e-06` | `6.724e-08` | fp32 conv stem | -| `enc.conv2.out` | `2.050e-05` | `3.044e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `2.295e-05` | `7.375e-07` | first encoder block | -| `enc.block.23.out` | `1.220e-01` | `1.999e-05` | final encoder block (peak signal grows with depth) | -| `enc.final` | `1.661e-02` | `1.164e-05` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `3.910e-05` | `1.036e-06` | first decoder block, prompt pass | -| `dec.block.23.out` | `2.548e-03` | `1.848e-05` | final decoder block (accumulated) | -| `dec.out_before_head` | `8.469e-04` | `3.497e-05` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `5.851e-04` | `6.196e-05` | vocab projection (raw logits) | -| `dec.logits` | `7.286e-04` | `8.941e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `7.176e-05` | `4.527e-05` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-small.en.md b/docs/models/whisper-small.en.md index 8297555c..0c9b9ef3 100644 --- a/docs/models/whisper-small.en.md +++ b/docs/models/whisper-small.en.md @@ -1,8 +1,10 @@ # Whisper small.en -OpenAI's [`openai/whisper-small.en`](https://huggingface.co/openai/whisper-small.en) ported to transcribe.cpp. A 244M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-small.en`](https://huggingface.co/openai/whisper-small.en) at [`e872752`](https://huggingface.co/openai/whisper-small.en/commit/e872752). + +OpenAI Whisper small.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,42 @@ Offline English speech-to-text. The model takes a 16 kHz mono WAV and returns a See the [upstream model card](https://huggingface.co/openai/whisper-small.en) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`e872752`](https://huggingface.co/openai/whisper-small.en/commit/e872752), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`e872752`](https://huggingface.co/openai/whisper-small.en/commit/e872752), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-small.en-F32.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-F32.gguf) | 924 MB | 3.09% | -| F16 | [whisper-small.en-F16.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-F16.gguf) | 470 MB | 2.97% | -| Q8_0 | [whisper-small.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q8_0.gguf) | 257 MB | 3.09% | -| Q6_K | [whisper-small.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q6_K.gguf) | 202 MB | 2.97% | -| Q5_K_M | [whisper-small.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q5_K_M.gguf) | 185 MB | 3.12% | -| Q4_K_M | [whisper-small.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q4_K_M.gguf) | 164 MB | 3.08% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-small.en-F32.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-F32.gguf) | 969 MB | 3.11% | +| F16 | [whisper-small.en-F16.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-F16.gguf) | 493 MB | 2.97% | +| Q8_0 | [whisper-small.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q8_0.gguf) | 270 MB | 3.09% | +| Q6_K | [whisper-small.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q6_K.gguf) | 212 MB | 2.97% | +| Q5_K_M | [whisper-small.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q5_K_M.gguf) | 194 MB | 3.11% | +| Q4_K_M | [whisper-small.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-small.en-gguf/resolve/main/whisper-small.en-Q4_K_M.gguf) | 172 MB | 3.09% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 3.05%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 6.14% | + ## Quick Start @@ -50,62 +69,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 107.1 ms (102.7×) | 102.6 ms (107.2×) | -| Metal | dots (35.3s) | 333.0 ms (106.1×) | 329.3 ms (107.3×) | -| CPU | jfk (11.0s) | 1.31 s (8.4×) | 1.13 s (9.8×) | -| CPU | dots (35.3s) | 2.89 s (12.2×) | 2.52 s (14.0×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 119.2 ms (92.31×) | 114.0 ms (96.47×) | +| Metal | dots (35.3s) | 384.3 ms (91.93×) | 387.5 ms (91.18×) | +| CPU | jfk (11.0s) | 597.0 ms (18.42×) | 953.9 ms (11.53×) | +| CPU | dots (35.3s) | 1.39 s (25.39×) | 1.60 s (22.10×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-small.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-small.en-publication +uv run scripts/bench/run.py --profile --models whisper-small.en ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 970 ms (11.3×) | 883 ms (12.5×) | -| Vulkan | dots (35.3s) | 2.48 s (14.3×) | 2.36 s (15.0×) | -| CPU | jfk (11.0s) | 3.68 s (3.0×) | 2.95 s (3.7×) | -| CPU | dots (35.3s) | 8.38 s (4.2×) | 7.16 s (4.9×) | +| Vulkan | jfk (11.0s) | 961 ms (11.44×) | 926 ms (11.89×) | +| Vulkan | dots (35.3s) | 2.62 s (13.51×) | 2.56 s (13.82×) | +| CPU | jfk (11.0s) | 1.98 s (5.57×) | 1.90 s (5.79×) | +| CPU | dots (35.3s) | 5.09 s (6.95×) | 4.96 s (7.12×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-small.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-small.en-publication +uv run scripts/bench/run.py --profile --models whisper-small.en ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance. Tolerance budget lives at -[`tests/tolerances/whisper-small.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-small.en.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-small.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-small.en.json). | Field | Value | | --- | --- | @@ -114,24 +123,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-small.en.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-small.en` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `5.901e-06` | `4.686e-08` | fp32 conv stem | -| `enc.conv2.out` | `1.574e-05` | `2.770e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `1.985e-05` | `6.654e-07` | first encoder block | -| `enc.block.11.out` | `1.855e-02` | `5.759e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `2.106e-03` | `3.011e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `1.621e-05` | `6.581e-07` | first decoder block, prompt pass | -| `dec.block.11.out` | `8.678e-05` | `3.321e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `2.718e-04` | `1.375e-05` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `5.913e-05` | `1.444e-05` | vocab projection (raw logits) | -| `dec.logits` | `6.914e-05` | `2.368e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `1.717e-05` | `2.999e-06` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-small.md b/docs/models/whisper-small.md index ac8c44e0..adaff728 100644 --- a/docs/models/whisper-small.md +++ b/docs/models/whisper-small.md @@ -1,8 +1,10 @@ # Whisper small -OpenAI's [`openai/whisper-small`](https://huggingface.co/openai/whisper-small) ported to transcribe.cpp. A 244M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-small`](https://huggingface.co/openai/whisper-small) at [`973afd2`](https://huggingface.co/openai/whisper-small/commit/973afd2). + +OpenAI Whisper small — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-small) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`973afd2`](https://huggingface.co/openai/whisper-small/commit/973afd2), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`973afd2`](https://huggingface.co/openai/whisper-small/commit/973afd2), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-small-F32.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-F32.gguf) | 924 MB | 3.33% | -| F16 | [whisper-small-F16.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-F16.gguf) | 470 MB | 3.34% | -| Q8_0 | [whisper-small-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q8_0.gguf) | 257 MB | 3.33% | -| Q6_K | [whisper-small-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q6_K.gguf) | 202 MB | 3.33% | -| Q5_K_M | [whisper-small-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q5_K_M.gguf) | 185 MB | 3.37% | -| Q4_K_M | [whisper-small-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q4_K_M.gguf) | 164 MB | 3.40% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-small-F32.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-F32.gguf) | 969 MB | 3.34% | +| F16 | [whisper-small-F16.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-F16.gguf) | 493 MB | 3.33% | +| Q8_0 | [whisper-small-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q8_0.gguf) | 270 MB | 3.33% | +| Q6_K | [whisper-small-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q6_K.gguf) | 212 MB | 3.33% | +| Q5_K_M | [whisper-small-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q5_K_M.gguf) | 194 MB | 3.37% | +| Q4_K_M | [whisper-small-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-small-gguf/resolve/main/whisper-small-Q4_K_M.gguf) | 172 MB | 3.40% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 3.432%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 62.20% | +| am | WER | 128.30% | +| ar | WER | 32.15% | +| as | WER | 104.15% | +| az | WER | 51.64% | +| be | WER | 76.85% | +| bg | WER | 40.39% | +| bn | WER | 103.04% | +| bs | WER | 42.35% | +| ca | WER | 14.19% | +| cs | WER | 40.57% | +| cy | WER | 64.47% | +| da | WER | 35.52% | +| de | WER | 9.86% | +| el | WER | 33.98% | +| en | WER | 6.51% | +| es | WER | 5.92% | +| et | WER | 54.79% | +| fa | WER | 58.44% | +| fi | WER | 26.48% | +| fil | WER | 28.52% | +| fr | WER | 13.30% | +| gl | WER | 32.72% | +| gu | WER | 104.02% | +| ha | WER | 94.21% | +| he | WER | 46.06% | +| hi | WER | 42.05% | +| hr | WER | 36.05% | +| hu | WER | 42.39% | +| hy | WER | 87.84% | +| id | WER | 18.02% | +| is | WER | 74.54% | +| it | WER | 7.97% | +| ja | CER | 12.81% | +| jv | WER | 94.42% | +| ka | WER | 130.64% | +| kk | WER | 73.54% | +| km | CER | 116.96% | +| kn | WER | 99.65% | +| ko | CER | 7.70% | +| lb | WER | 110.02% | +| ln | WER | 98.65% | +| lo | CER | 101.52% | +| lt | WER | 70.21% | +| lv | WER | 57.42% | +| mi | WER | 62.82% | +| mk | WER | 41.53% | +| ml | WER | 100.35% | +| mn | WER | 142.37% | +| mr | WER | 63.66% | +| ms | WER | 21.48% | +| mt | WER | 97.32% | +| my | CER | 132.91% | +| nb | WER | 25.53% | +| ne | WER | 70.48% | +| nl | WER | 18.48% | +| oc | WER | 90.43% | +| pa | WER | 101.31% | +| pl | WER | 16.82% | +| ps | WER | 93.61% | +| pt | WER | 7.65% | +| ro | WER | 33.88% | +| ru | WER | 11.90% | +| sd | WER | 112.24% | +| sk | WER | 36.05% | +| sl | WER | 52.70% | +| sn | WER | 132.90% | +| so | WER | 103.39% | +| sr | WER | 44.91% | +| sv | WER | 23.10% | +| sw | WER | 76.22% | +| ta | WER | 35.35% | +| te | WER | 102.23% | +| tg | WER | 86.78% | +| th | CER | 22.55% | +| tr | WER | 15.95% | +| uk | WER | 20.42% | +| ur | WER | 39.75% | +| uz | WER | 114.92% | +| vi | WER | 22.47% | +| yo | WER | 118.50% | +| zh | CER | 23.06% | + ## Quick Start @@ -50,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mel + encode + decode, mean over the recorded -iterations after warmup), with speedup over realtime in parentheses. Units: -`ms` below 1 s, `s` above (2 decimal places). Decode latency dominates as -model size grows; the encoder is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 113.1 ms (97.2×) | 113.5 ms (96.9×) | -| Metal | dots (35.3s) | 349.3 ms (101.2×) | 340.0 ms (103.9×) | -| CPU | jfk (11.0s) | 1.43 s (7.7×) | 1.30 s (8.4×) | -| CPU | dots (35.3s) | 3.01 s (11.8×) | 2.74 s (12.9×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | ----------------: | ----------------: | +| Metal | jfk (11.0s) | 123.1 ms (89.39×) | 122.8 ms (89.56×) | +| Metal | dots (35.3s) | 400.8 ms (88.15×) | 394.0 ms (89.68×) | +| CPU | jfk (11.0s) | 647.9 ms (16.98×) | 718.6 ms (15.31×) | +| CPU | dots (35.3s) | 1.46 s (24.28×) | 1.61 s (22.01×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-small \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-small-publication +uv run scripts/bench/run.py --profile --models whisper-small ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 1.03 s (10.6×) | 0.96 s (11.4×) | -| Vulkan | dots (35.3s) | 2.57 s (13.7×) | 2.47 s (14.3×) | -| CPU | jfk (11.0s) | 3.95 s (2.8×) | 3.27 s (3.4×) | -| CPU | dots (35.3s) | 8.91 s (4.0×) | 7.47 s (4.7×) | +| Vulkan | jfk (11.0s) | 1.02 s (10.82×) | 1.02 s (10.76×) | +| Vulkan | dots (35.3s) | 2.81 s (12.57×) | 2.74 s (12.91×) | +| CPU | jfk (11.0s) | 2.17 s (5.07×) | 2.11 s (5.22×) | +| CPU | dots (35.3s) | 5.31 s (6.65×) | 5.16 s (6.84×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-small \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-small-publication +uv run scripts/bench/run.py --profile --models whisper-small ``` ## Numerical Validation -transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's cases (`samples/jfk.wav` and `samples/german.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcripts match the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-small.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-small.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 23 checkpointed tensors fall within per-variant tolerance, and the transcripts match the HF reference verbatim. Tolerance budget lives at +[`tests/tolerances/whisper-small.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-small.json). | Field | Value | | --- | --- | @@ -114,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-small.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-small` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `5.454e-06` | `4.815e-08` | fp32 conv stem | -| `enc.conv2.out` | `2.146e-05` | `2.704e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `2.134e-05` | `6.858e-07` | first encoder block | -| `enc.block.11.out` | `1.669e-02` | `6.139e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `2.100e-03` | `3.882e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `4.768e-06` | `2.997e-07` | first decoder block, prompt pass | -| `dec.block.11.out` | `3.662e-04` | `2.471e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `1.755e-04` | `7.318e-06` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `3.719e-05` | `7.083e-06` | vocab projection (raw logits) | -| `dec.logits` | `8.965e-05` | `1.926e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `2.289e-05` | `8.815e-06` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-tiny.en.md b/docs/models/whisper-tiny.en.md index f1bb0b10..d6dee6e2 100644 --- a/docs/models/whisper-tiny.en.md +++ b/docs/models/whisper-tiny.en.md @@ -1,8 +1,10 @@ # Whisper tiny.en -OpenAI's [`openai/whisper-tiny.en`](https://huggingface.co/openai/whisper-tiny.en) ported to transcribe.cpp. A 39M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-tiny.en`](https://huggingface.co/openai/whisper-tiny.en) at [`87c7102`](https://huggingface.co/openai/whisper-tiny.en/commit/87c7102). + +OpenAI Whisper tiny.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,42 @@ Offline English speech-to-text. The model takes a 16 kHz mono WAV and returns a See the [upstream model card](https://huggingface.co/openai/whisper-tiny.en) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`87c7102`](https://huggingface.co/openai/whisper-tiny.en/commit/87c7102), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`87c7102`](https://huggingface.co/openai/whisper-tiny.en/commit/87c7102), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-tiny.en-F32.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-F32.gguf) | 146 MB | 5.77% | -| F16 | [whisper-tiny.en-F16.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-F16.gguf) | 76 MB | 5.77% | -| Q8_0 | [whisper-tiny.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q8_0.gguf) | 44 MB | 5.72% | -| Q6_K | [whisper-tiny.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q6_K.gguf) | 43 MB | 5.80% | -| Q5_K_M | [whisper-tiny.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q5_K_M.gguf) | 42 MB | 5.89% | -| Q4_K_M | [whisper-tiny.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q4_K_M.gguf) | 42 MB | 5.99% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-tiny.en-F32.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-F32.gguf) | 153 MB | 5.77% | +| F16 | [whisper-tiny.en-F16.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-F16.gguf) | 80 MB | 5.78% | +| Q8_0 | [whisper-tiny.en-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q8_0.gguf) | 46 MB | 5.72% | +| Q6_K | [whisper-tiny.en-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q6_K.gguf) | 45 MB | 5.83% | +| Q5_K_M | [whisper-tiny.en-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q5_K_M.gguf) | 44 MB | 5.91% | +| Q4_K_M | [whisper-tiny.en-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny.en-gguf/resolve/main/whisper-tiny.en-Q4_K_M.gguf) | 44 MB | 5.96% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 5.66%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| en | WER | 10.72% | + ## Quick Start @@ -50,62 +69,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). Decode latency dominates as model size grows; the encoder -is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 39.1 ms (281.2×) | 34.0 ms (323.8×) | -| Metal | dots (35.3s) | 127.0 ms (278.3×) | 125.8 ms (280.9×) | -| CPU | jfk (11.0s) | 165.0 ms (66.7×) | 161.4 ms (68.1×) | -| CPU | dots (35.3s) | 389.4 ms (90.7×) | 381.8 ms (92.5×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -----------------: | -----------------: | +| Metal | jfk (11.0s) | 38.1 ms (288.65×) | 37.8 ms (291.22×) | +| Metal | dots (35.3s) | 142.8 ms (247.49×) | 136.1 ms (259.57×) | +| CPU | jfk (11.0s) | 93.2 ms (118.02×) | 97.5 ms (112.83×) | +| CPU | dots (35.3s) | 246.8 ms (143.18×) | 249.8 ms (141.46×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-tiny.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-tiny.en-publication +uv run scripts/bench/run.py --profile --models whisper-tiny.en ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 197 ms (56.0×) | 193 ms (56.9×) | -| Vulkan | dots (35.3s) | 540 ms (65.4×) | 541 ms (65.3×) | -| CPU | jfk (11.0s) | 493 ms (22.3×) | 436 ms (25.2×) | -| CPU | dots (35.3s) | 1.19 s (29.8×) | 1.09 s (32.5×) | +| Vulkan | jfk (11.0s) | 253 ms (43.47×) | 251 ms (43.82×) | +| Vulkan | dots (35.3s) | 686 ms (51.49×) | 666 ms (53.09×) | +| CPU | jfk (11.0s) | 288 ms (38.24×) | 283 ms (38.88×) | +| CPU | dots (35.3s) | 799 ms (44.20×) | 790 ms (44.70×) | -Fedora 43, transcribe.cpp `e0fa0f6`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-tiny.en \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-tiny.en-publication +uv run scripts/bench/run.py --profile --models whisper-tiny.en ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's case (`samples/jfk.wav`). All 21 checkpointed tensors fall within per-variant tolerance. Tolerance budget lives at -[`tests/tolerances/whisper-tiny.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-tiny.en.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-tiny.en.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-tiny.en.json). | Field | Value | | --- | --- | @@ -114,24 +123,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-tiny.en.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-tiny.en` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `2.229e-05` | `3.381e-08` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `7.272e-06` | `7.293e-08` | fp32 conv stem | -| `enc.conv2.out` | `1.240e-05` | `2.603e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `2.623e-05` | `8.075e-07` | first encoder block | -| `enc.block.3.out` | `1.709e-02` | `3.026e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `4.005e-05` | `2.161e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `2.861e-05` | `3.417e-07` | first decoder block, prompt pass | -| `dec.block.3.out` | `4.196e-05` | `1.334e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `1.450e-04` | `2.361e-05` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `6.807e-05` | `3.062e-05` | vocab projection (raw logits) | -| `dec.logits` | `9.584e-05` | `3.682e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `7.248e-05` | `4.521e-05` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper-tiny.md b/docs/models/whisper-tiny.md index 9510ddfb..9b20eeb6 100644 --- a/docs/models/whisper-tiny.md +++ b/docs/models/whisper-tiny.md @@ -1,8 +1,10 @@ # Whisper tiny -OpenAI's [`openai/whisper-tiny`](https://huggingface.co/openai/whisper-tiny) ported to transcribe.cpp. A 39M-parameter -encoder-decoder transformer (audio encoder + autoregressive text decoder with -cross-attention). + +Upstream: [`openai/whisper-tiny`](https://huggingface.co/openai/whisper-tiny) at [`169d4a4`](https://huggingface.co/openai/whisper-tiny/commit/169d4a4). + +OpenAI Whisper tiny — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. + ## What it's for @@ -11,25 +13,123 @@ Offline multilingual speech-to-text and any-language → English speech translat See the [upstream model card](https://huggingface.co/openai/whisper-tiny) for training data, intended use, and the original evaluation methodology. -Licensed Apache-2.0. Ported from upstream commit -[`169d4a4`](https://huggingface.co/openai/whisper-tiny/commit/169d4a4), -pinned 2026-04-25. Validated against the transformers reference at -transcribe.cpp commit -[`5.6.1`](https://github.com/handy-computer/transcribe.cpp/tree/5.6.1) -on 2026-04-26. + +Licensed Apache-2.0. Ported from upstream commit [`169d4a4`](https://huggingface.co/openai/whisper-tiny/commit/169d4a4), pinned 2026-04-25. Validated against the transformers reference at transcribe.cpp commit [`0a26478`](https://github.com/handy-computer/transcribe.cpp/tree/0a26478) on 2026-09-13. + ## Download -| Quantization | Download | Size | WER (LibriSpeech test-clean) | + +| Quantization | Download | Size | WER (LibriSpeech test-clean) | | --- | --- | ---: | ---: | -| F32 | [whisper-tiny-F32.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F32.gguf) | 146 MB | 7.54% | -| F16 | [whisper-tiny-F16.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F16.gguf) | 76 MB | 7.49% | -| Q8_0 | [whisper-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q8_0.gguf) | 44 MB | 7.53% | -| Q6_K | [whisper-tiny-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q6_K.gguf) | 43 MB | 7.63% | -| Q5_K_M | [whisper-tiny-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q5_K_M.gguf) | 42 MB | 7.63% | -| Q4_K_M | [whisper-tiny-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q4_K_M.gguf) | 42 MB | 7.76% | - -WER measured on the full LibriSpeech test-clean split (2620 utterances) with transcribe.cpp's default greedy decode and segment timestamps enabled — the same runs summarized in the [Whisper family table](whisper.md#all-variants). Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions add ~0.1pp of run-to-run variance on the noise floor, and quantization is otherwise generally WER-neutral. See the [WER methodology](../tools/wer.md) for the harness. +| F32 | [whisper-tiny-F32.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F32.gguf) | 153 MB | 7.49% | +| F16 | [whisper-tiny-F16.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-F16.gguf) | 80 MB | 7.48% | +| Q8_0 | [whisper-tiny-Q8_0.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q8_0.gguf) | 46 MB | 7.52% | +| Q6_K | [whisper-tiny-Q6_K.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q6_K.gguf) | 45 MB | 7.54% | +| Q5_K_M | [whisper-tiny-Q5_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q5_K_M.gguf) | 44 MB | 7.82% | +| Q4_K_M | [whisper-tiny-Q4_K_M.gguf](https://huggingface.co/handy-computer/whisper-tiny-gguf/resolve/main/whisper-tiny-Q4_K_M.gguf) | 44 MB | 7.78% | + + + +WER on the full LibriSpeech test-clean split (2,620 utterances), batch size 1, timestamps none. Figures without a commit were published before provenance was recorded. + + + +OpenAI's self-reported number on the same split is 7.54%. Both are +short-form WER decoded without timestamps; OpenAI does not publish its exact +evaluation configuration, so small differences are expected. Single-run +figures: GPU reductions can shift corpus WER by about 0.1pp between runs, +mostly on short-clip hallucination outcomes at the noise floor. + + + +**FLEURS test** + +| Language | Metric | Q8_0 | +| --- | --- | ---: | +| af | WER | 95.30% | +| am | WER | 122.14% | +| ar | WER | 67.48% | +| as | WER | 100.56% | +| az | WER | 94.73% | +| be | WER | 96.03% | +| bg | WER | 84.87% | +| bn | WER | 100.37% | +| bs | WER | 87.22% | +| ca | WER | 46.33% | +| cs | WER | 87.17% | +| cy | WER | 116.68% | +| da | WER | 88.42% | +| de | WER | 31.56% | +| el | WER | 77.76% | +| en | WER | 13.84% | +| es | WER | 18.95% | +| et | WER | 101.33% | +| fa | WER | 100.18% | +| fi | WER | 68.58% | +| fil | WER | 70.63% | +| fr | WER | 44.19% | +| gl | WER | 59.44% | +| gu | WER | 100.26% | +| ha | WER | 105.17% | +| he | WER | 77.35% | +| hi | WER | 101.52% | +| hr | WER | 84.36% | +| hu | WER | 90.54% | +| hy | WER | 110.50% | +| id | WER | 60.49% | +| is | WER | 115.88% | +| it | WER | 31.24% | +| ja | CER | 39.38% | +| jv | WER | 106.07% | +| ka | WER | 110.34% | +| kk | WER | 136.56% | +| km | CER | 111.78% | +| kn | WER | 100.32% | +| ko | CER | 19.07% | +| lb | WER | 99.69% | +| ln | WER | 103.55% | +| lo | CER | 105.48% | +| lt | WER | 105.00% | +| lv | WER | 95.98% | +| mi | WER | 96.27% | +| mk | WER | 78.61% | +| ml | WER | 100.03% | +| mn | WER | 110.21% | +| mr | WER | 100.82% | +| ms | WER | 62.17% | +| mt | WER | 99.85% | +| my | CER | 108.53% | +| nb | WER | 67.36% | +| ne | WER | 101.30% | +| nl | WER | 54.86% | +| oc | WER | 96.44% | +| pa | WER | 100.54% | +| pl | WER | 54.22% | +| ps | WER | 101.04% | +| pt | WER | 24.07% | +| ro | WER | 82.43% | +| ru | WER | 35.64% | +| sd | WER | 105.42% | +| sk | WER | 82.20% | +| sl | WER | 91.29% | +| sn | WER | 118.33% | +| so | WER | 105.31% | +| sr | WER | 89.20% | +| sv | WER | 59.36% | +| sw | WER | 99.96% | +| ta | WER | 87.46% | +| te | WER | 101.18% | +| tg | WER | 102.39% | +| th | CER | 54.82% | +| tr | WER | 47.98% | +| uk | WER | 57.49% | +| ur | WER | 71.42% | +| uz | WER | 103.19% | +| vi | WER | 64.49% | +| yo | WER | 102.29% | +| zh | CER | 41.91% | + ## Quick Start @@ -50,62 +150,52 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav ## Performance -Cells are wall-clock latency (mean over 3 iterations after 1 warmup), with -speedup over realtime in parentheses. Units: `ms` below 1 s, `s` above (2 -decimal places). Decode latency dominates as model size grows; the encoder -is only run once per 30-second window. - ### Apple M4 Max -| Backend | Sample | Q8_0 | Q4_K_M | -| ------- | ------------ | ---------------: | ---------------: | -| Metal | jfk (11.0s) | 36.4 ms (302.5×) | 37.9 ms (290.1×) | -| Metal | dots (35.3s) | 117.1 ms (301.8×) | 117.3 ms (301.3×) | -| CPU | jfk (11.0s) | 174.7 ms (63.0×) | 169.8 ms (64.8×) | -| CPU | dots (35.3s) | 396.3 ms (89.2×) | 390.4 ms (90.5×) | + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. -macOS 26.4.1, transcribe.cpp `e0fa0f6`. +| Backend | Sample | Q8_0 | Q4_K_M | +| ------- | ------------ | -----------------: | -----------------: | +| Metal | jfk (11.0s) | 39.0 ms (282.04×) | 39.5 ms (278.34×) | +| Metal | dots (35.3s) | 139.4 ms (253.55×) | 141.4 ms (249.80×) | +| CPU | jfk (11.0s) | 99.9 ms (110.09×) | 99.7 ms (110.34×) | +| CPU | dots (35.3s) | 250.5 ms (141.07×) | 253.0 ms (139.65×) | + +Apple M4 Max: transcribe.cpp `77b0c93` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-tiny \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends metal,cpu \ - --iters 3 --warmup 1 \ - --name whisper-tiny-publication +uv run scripts/bench/run.py --profile --models whisper-tiny ``` ### AMD Ryzen 7 PRO 4750U + +Compute latency (mel + encode + decode), speedup over realtime in parentheses; profile `asr-publication-v2`: mean over 3 iterations after 1 warmup. + | Backend | Sample | Q8_0 | Q4_K_M | | ------- | ------------ | --------------: | --------------: | -| Vulkan | jfk (11.0s) | 200 ms (55.1×) | 209 ms (52.6×) | -| Vulkan | dots (35.3s) | 528 ms (66.9×) | 529 ms (66.8×) | -| CPU | jfk (11.0s) | 531 ms (20.7×) | 467 ms (23.6×) | -| CPU | dots (35.3s) | 1.23 s (28.8×) | 1.14 s (31.0×) | +| Vulkan | jfk (11.0s) | 254 ms (43.22×) | 253 ms (43.55×) | +| Vulkan | dots (35.3s) | 822 ms (42.96×) | 825 ms (42.82×) | +| CPU | jfk (11.0s) | 306 ms (35.96×) | 311 ms (35.39×) | +| CPU | dots (35.3s) | 811 ms (43.55×) | 820 ms (43.08×) | -Fedora 43, transcribe.cpp `2ab01b8`. Vulkan device: `AMD Radeon -Graphics (RADV RENOIR)`. +AMD Ryzen 7 PRO 4750U (Radeon RADV RENOIR): transcribe.cpp `218aeae3` on 2026-09-14. + Benchmark reproduction: ```bash -uv run scripts/bench/run.py \ - --models whisper-tiny \ - --quants q8_0,q4_k_m \ - --samples jfk,dots \ - --backends cpu,vulkan \ - --iters 3 --warmup 1 \ - --name whisper-tiny-publication +uv run scripts/bench/run.py --profile --models whisper-tiny ``` ## Numerical Validation transcribe.cpp is validated tensor-by-tensor against the transformers reference (`WhisperForConditionalGeneration`, fp32 CPU) on the manifest's cases (`samples/jfk.wav` and `samples/german.wav`). All 21 checkpointed tensors fall within per-variant tolerance, and the transcripts match the HF reference verbatim. Tolerance budget lives at -[`tests/tolerances/whisper-tiny.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-tiny.json). Last validated at commit [`1854f57`](https://github.com/handy-computer/transcribe.cpp/tree/1854f57). +[`tests/tolerances/whisper-tiny.json`](https://github.com/handy-computer/transcribe.cpp/blob/main/tests/tolerances/whisper-tiny.json). | Field | Value | | --- | --- | @@ -114,24 +204,6 @@ transcribe.cpp is validated tensor-by-tensor against the transformers reference | Tolerance file | `tests/tolerances/whisper-tiny.json` | | Command | `uv run scripts/validate.py all --family whisper --variant whisper-tiny` | -Selected tensors (worst observed across cases; see tolerance file for per-tensor budgets): - -| Tensor | Max abs diff | Mean abs diff | Notes | -| ---------------------- | ---: | ---: | --- | -| `enc.mel.in` | `3.946e-05` | `1.396e-07` | fp32 mixed-radix FFT vs torch fp64 frontend | -| `enc.conv1.out` | `1.365e-05` | `1.140e-07` | fp32 conv stem | -| `enc.conv2.out` | `1.550e-05` | `2.354e-07` | stride-2 conv stem (matches enc.embed.out) | -| `enc.block.0.out` | `1.407e-05` | `5.994e-07` | first encoder block | -| `enc.block.3.out` | `8.335e-03` | `4.857e-06` | final encoder block (peak signal grows with depth) | -| `enc.final` | `2.285e-03` | `3.926e-06` | post-LN encoder output | -| `dec.token_emb` | `0.000e+00` | `0.000e+00` | exact zero-drift (`ggml_get_rows` on the F32 GGUF) | -| `dec.block.0.out` | `6.676e-06` | `2.617e-07` | first decoder block, prompt pass | -| `dec.block.3.out` | `1.144e-04` | `1.228e-06` | final decoder block (accumulated) | -| `dec.out_before_head` | `3.204e-04` | `1.665e-05` | post final LN, pre-vocab projection | -| `dec.logits_raw` | `6.294e-05` | `1.858e-05` | vocab projection (raw logits) | -| `dec.logits` | `7.248e-05` | `2.055e-05` | log-softmax over vocab | -| `dec.logits_raw.gen20` | `4.959e-05` | `1.305e-05` | step-20 logits (KV-cached path) | - The C++ mel frontend (Slaney filterbank + Hann periodic window + whisper-style log-mel compression) drives `enc.mel.in` to fp32-vs-fp64 STFT precision drift; downstream tensors stay within budget. KV-cached diff --git a/docs/models/whisper.md b/docs/models/whisper.md index cb940ded..ac292a36 100644 --- a/docs/models/whisper.md +++ b/docs/models/whisper.md @@ -38,24 +38,28 @@ see the family doc at WER is on LibriSpeech test-clean for the **Q8_0** preset (the default recommended quant), measured by transcribe.cpp's WER pipeline with -segment timestamps enabled. See each per-variant doc for the full quant +timestamps off (`scripts/wer/run.py --timestamps none`, the WER harness +default). See each per-variant doc for the full quant matrix (F32/F16/Q8_0/Q6_K/Q5_K_M/Q4_K_M) and a discussion of how our numbers compare to OpenAI's self-reported figures. Numbers come from single Metal-backed runs; Metal's non-deterministic parallel reductions add ~0.1pp run-to-run variance on the noise floor. -| Variant | Languages | Q8_0 size | WER (Q8_0) | Doc | -| --- | --- | ---: | ---: | --- | -| `whisper-tiny` | 99 + auto-detect | 44 MB | 7.53% | [whisper-tiny.md](whisper-tiny.md) | -| `whisper-tiny.en` | English only | 44 MB | 5.72% | [whisper-tiny.en.md](whisper-tiny.en.md) | -| `whisper-base` | 99 + auto-detect | 81 MB | 5.12% | [whisper-base.md](whisper-base.md) | -| `whisper-base.en` | English only | 81 MB | 4.16% | [whisper-base.en.md](whisper-base.en.md) | -| `whisper-small` | 99 + auto-detect | 257 MB | 3.33% | [whisper-small.md](whisper-small.md) | -| `whisper-small.en` | English only | 257 MB | 3.09% | [whisper-small.en.md](whisper-small.en.md) | -| `whisper-medium` | 99 + auto-detect | 793 MB | 2.64% | [whisper-medium.md](whisper-medium.md) | -| `whisper-medium.en` | English only | 793 MB | 2.72% | [whisper-medium.en.md](whisper-medium.en.md) | -| `whisper-large` | 99 + auto-detect | 1.55 GB | 2.74% | [whisper-large.md](whisper-large.md) | -| `whisper-large-v2` | 99 + auto-detect | 1.55 GB | 2.65% | [whisper-large-v2.md](whisper-large-v2.md) | -| `whisper-large-v3` | 100 + auto-detect | 1.55 GB | 1.82% | [whisper-large-v3.md](whisper-large-v3.md) | -| `whisper-large-v3-turbo` | 100 + auto-detect | 845 MB | 2.01% | [whisper-large-v3-turbo.md](whisper-large-v3-turbo.md) | + +| Variant | Params | Languages | Q8_0 size | Benchmark | Q8_0 | Capabilities | Doc | +| --- | ---: | --- | ---: | --- | ---: | --- | --- | +| `breeze-asr-25` | 1.5B | zh, en + auto-detect | 1.67 GB | LibriSpeech test-clean (WER) | 2.27% | translate, segment timestamps | [handy-computer/Breeze-ASR-25-gguf](https://huggingface.co/handy-computer/Breeze-ASR-25-gguf) | +| `whisper-tiny` | 38M | 99 languages + auto-detect | 46 MB | LibriSpeech test-clean (WER) | 7.52% | translate, segment timestamps | [whisper-tiny.md](whisper-tiny.md) | +| `whisper-tiny.en` | 38M | en | 46 MB | LibriSpeech test-clean (WER) | 5.72% | segment timestamps | [whisper-tiny.en.md](whisper-tiny.en.md) | +| `whisper-base` | 73M | 99 languages + auto-detect | 85 MB | LibriSpeech test-clean (WER) | 5.12% | translate, segment timestamps | [whisper-base.md](whisper-base.md) | +| `whisper-base.en` | 73M | en | 85 MB | LibriSpeech test-clean (WER) | 4.16% | segment timestamps | [whisper-base.en.md](whisper-base.en.md) | +| `whisper-small` | 242M | 99 languages + auto-detect | 270 MB | LibriSpeech test-clean (WER) | 3.33% | translate, segment timestamps | [whisper-small.md](whisper-small.md) | +| `whisper-small.en` | 242M | en | 270 MB | LibriSpeech test-clean (WER) | 3.09% | segment timestamps | [whisper-small.en.md](whisper-small.en.md) | +| `whisper-medium` | 764M | 99 languages + auto-detect | 832 MB | LibriSpeech test-clean (WER) | 2.64% | translate, segment timestamps | [whisper-medium.md](whisper-medium.md) | +| `whisper-medium.en` | 764M | en | 831 MB | LibriSpeech test-clean (WER) | 2.72% | segment timestamps | [whisper-medium.en.md](whisper-medium.en.md) | +| `whisper-large` | 1.5B | 99 languages + auto-detect | 1.67 GB | LibriSpeech test-clean (WER) | 2.71% | translate, segment timestamps | [whisper-large.md](whisper-large.md) | +| `whisper-large-v2` | 1.5B | 99 languages + auto-detect | 1.67 GB | LibriSpeech test-clean (WER) | 2.97% | translate, segment timestamps | [whisper-large-v2.md](whisper-large-v2.md) | +| `whisper-large-v3` | 1.5B | 100 languages + auto-detect | 1.67 GB | LibriSpeech test-clean (WER) | 1.82% | translate, segment timestamps | [whisper-large-v3.md](whisper-large-v3.md) | +| `whisper-large-v3-turbo` | 809M | 100 languages + auto-detect | 886 MB | LibriSpeech test-clean (WER) | 2.01% | segment timestamps | [whisper-large-v3-turbo.md](whisper-large-v3-turbo.md) | + Pre-built GGUFs for every variant and quant are hosted under [`handy-computer` on Hugging Face](https://huggingface.co/handy-computer); diff --git a/docs/porting/0-porting.md b/docs/porting/0-porting.md index a0ed8d6c..ebcc538d 100644 --- a/docs/porting/0-porting.md +++ b/docs/porting/0-porting.md @@ -60,7 +60,7 @@ The high-level intent of each stage: the family-doc Capability Validation table, gate full ref-dtype WER vs the measured Oracle reference baseline. 5. **Quants** — generate the shipped quant matrix, CLI smoke each - produced GGUF, and take a tentative quant WER read for human review. + produced GGUF, and take a preliminary 512-utterance quant WER read. 6. **Bench** — performance matrix; every accepted iteration re-runs `validate.py all`. 7. **WER** — full release WER sweep; ref-dtype hard gate against the diff --git a/docs/porting/families/cohere.md b/docs/porting/families/cohere.md index d078566a..c3abae15 100644 --- a/docs/porting/families/cohere.md +++ b/docs/porting/families/cohere.md @@ -9,7 +9,11 @@ native Transformers. C++ CPU validation passes locally. - Upstream architecture string: `cohere_asr` - Current source directory shape: Hugging Face-style `cohere-transcribe-03-2026/` -- Variant: `cohere-transcribe-03-2026` +- Variants: `cohere-transcribe-03-2026`, + `cohere-transcribe-arabic-07-2026` (retrained Arabic-focused + checkpoint, identical architecture and tensor layout; languages + `[en, ar]`; config omits top-level `vocab_size` — the converter falls + back to `head.num_classes`; upstream repo is gated) ## References @@ -72,3 +76,10 @@ TRANSCRIBE_COHERE_GGUF=models/cohere-transcribe-03-2026/cohere-transcribe-03-202 - Manifest records `hf_revision` but not local artifact hashes. - Reference hardware should still be captured for benchmark reports. +- `cohere-transcribe-arabic-07-2026` has no per-variant golden tensor + manifest, converter report, or bench run; it is validated end-to-end by + WER parity against the native Transformers reference on FLEURS Arabic + (C++ BF16 11.02% vs reference 11.00%, 428 utts, batch 8, L40S; see + `reports/wer/cohere-transcribe-arabic-07-2026-*.fleurs-ar.b8.*`). The + WER baseline runner is + `scripts/wer/run_reference_cohere_transformers.py`. diff --git a/docs/tools/hf-metadata-schema.md b/docs/tools/hf-metadata-schema.md index 6ad7a22d..df0bf748 100644 --- a/docs/tools/hf-metadata-schema.md +++ b/docs/tools/hf-metadata-schema.md @@ -10,25 +10,28 @@ capability flags; any 0–100 score is left to the consumer to compute from thes ## Where it comes from -`scripts/hf_cards/generate.py` serializes the block from a per-model spec -(`scripts/hf_cards/.yaml`) — per-quant WER (`quants:`), realtime factors -(`perf:`), optional task-specific raw measurements (`metrics:`), and capability -flags (`capabilities:`) — into the card via `template.md.j2`. A spec with no -`perf:` emits no block, so the rollout is per-spec and never breaks an -un-migrated card. +`scripts/hf_cards/generate.py` serializes the block from the catalog record +(`catalog/.json`): one per-quant map for every accuracy result set the +record holds, realtime factors from the speed rows at the card's default quant, +and capability flags from the record's `capabilities` block. The editorial spec +(`scripts/hf_cards/.yaml`) contributes nothing to it. A record with no +speed rows at the default quant emits no block. ## Fields ```yaml transcribe_cpp: + schema_version: 2 # bumped when key names or shapes change wer_librispeech_test_clean: # raw %, per quant — lower is better f32: 1.68 q8_0: 1.69 q4_k_m: 1.72 - rtf_ryzen_4750u: { cpu: 8, vulkan: 15 } # raw ×realtime — higher is better - rtf_m4_max: { cpu: 29, metal: 175 } - cpwer_ami_ihm_test: # optional task metric, raw % - bundle_f32_kernel: 19.35 + cer_fleurs_zh: # one map per result set + q8_0: 8.10 + cpwer_ami_ihm_test_kernel: # a decoding mode is its own set + f32: 19.35 + rtf_ryzen_4750u: { cpu: 8.12, vulkan: 15.4 } # raw ×realtime — higher is better + rtf_m4_max: { cpu: 29.05, metal: 175.2 } streaming: false diarize: false translate: false @@ -38,25 +41,18 @@ transcribe_cpp: | Field | Meaning | | --- | --- | -| `wer_` | Word error rate (%) per quant, on the named dataset. Lower is better. | -| `rtf_` | Speedup-over-realtime (×RT) per backend, mean over the published bench samples. Higher is better. | -| Task metrics from `metrics:` | Optional raw measurements emitted verbatim under their spec key, such as `cpwer_ami_ihm_test`. | +| `schema_version` | 2. Version 1 cards (no field) carried one hand-named headline map; a CER or DER set could appear under a `wer_` key there. Consumers should read `.get()` and key on the metric prefix. | +| `__[_][_]` | Error rate (%) per quant on that result set. Lower is better. `wer`, `cer`, `der`, `cpwer` as the row's metric; FLEURS keys carry the language, other datasets the split; a scoring step (`opencc_t2s`) or decoding mode (`kernel`) makes a separate key. | +| `rtf_` | Speedup-over-realtime (×RT) per backend at the default quant, mean over the published bench samples. Higher is better. | | `streaming` | Model supports buffered/cache-aware streaming. | | `diarize` | Model can emit speaker-attributed transcript rows or speaker turns. | | `translate` | Model can emit a translation (not just transcription). | | `lang_detect` | Model auto-detects the input language (vs. requiring an explicit hint). | | `timestamps` | Finest timestamp granularity the model emits (`none`/`segment`/`word`/`token`, mirroring the library's `max_timestamp_kind`). | -The `` suffix is the spec's `wer.metadata_key` (default -`librispeech_test_clean`); the `` suffix is the `perf:` rig key with `-` -mapped to `_`. A spec can also publish secondary benchmarks inline: any dataset-named key -under `wer:` whose value is a `{quant: wer%}` map (e.g. `librispeech_test_clean:`) -is emitted as its own `wer_` block alongside the headline one — so a -model whose per-quant column is e.g. FLEURS can still expose LibriSpeech -machine-readably. Add a dataset by adding a key; no wrapper needed. Other -raw task measurements live under the spec's `metrics:` map and are emitted -verbatim. Standard HF keys (`license`, `language`, `pipeline_tag`, `base_model`, -`tags`, …) are emitted alongside and unchanged by this block. +The `` suffix is the catalog machine slug with `-` mapped to `_`. +Standard HF keys (`license`, `language`, `pipeline_tag`, `base_model`, `tags`, +…) are emitted alongside and unchanged by this block. ## Reading it diff --git a/docs/tools/wer.md b/docs/tools/wer.md index aa7fbba6..227f1da5 100644 --- a/docs/tools/wer.md +++ b/docs/tools/wer.md @@ -46,70 +46,41 @@ The recipe is **stamped into the hyp JSONL `batch_header`** (`recipe` field) by `run.py`, so every artifact is self-describing and a methodology drift shows up in the file rather than silently shifting the number. -> **ITN is pinned, not inherited.** The run-time ITN default is per-family and -> is a product decision that can move: `sensevoice` resolves it to *on* (there -> the ITN toggle is also the only source of casing and punctuation, so ITN-off -> hands an unconfigured caller lowercase unpunctuated text), while -> `funasr_nano` keeps upstream's `itn=False`. The benchmark follows neither: -> `run.py` always passes `--no-itn`, because the reference runs it is gated -> against produce spoken form. Explicit for every family, so a future default -> flip cannot silently restate what a published number means. -> -> This is not just a formatting difference that normalization would absorb. -> ITN changes the decode itself, so it moves WER on its own — **in the upstream -> model, not only in this port.** Measured on LibriSpeech test-clean: -> -> | Arm | ITN off | ITN on | Δ | n | -> | --- | ---: | ---: | ---: | ---: | -> | SenseVoice — FunASR 1.3.1 reference, FP32/CPU | 2.538% | 2.685% | **+0.147pp** | 512 | -> | SenseVoice — transcribe.cpp F32/CPU | 2.556% | 2.666% | **+0.110pp** | 512 | -> | SenseVoice — transcribe.cpp Q8_0/Metal | 2.556% | 2.731% | +0.175pp | 512 | -> | Fun-ASR-Nano-2512 BF16 | 1.754% | 1.840% | +0.086pp | 200 | -> | Fun-ASR-MLT-Nano-2512 BF16 | 1.668% | 1.711% | +0.043pp | 200 | -> -> The ITN cost is a property of the model. At matched dtype the port's ITN -> penalty (+0.110pp) is *smaller* than the reference's (+0.147pp), and the -> degradation is the same degradation: of the 32 utterances where the port -> regresses under ITN, 26 also regress in the reference, and the top cases are -> byte-identical on both sides (`arcadian` → `arrcadian`, `sententiously` → -> `sentiously`, `gilchrist` → `gilcht`, `pride` → `bride`). Comparing the Q8_0 -> arm against the FP32 reference overstates the gap: ITN-on is somewhat more -> quant-sensitive, which is the +0.065pp between the F32 and Q8_0 rows. -> -> On English it is not number rendering — only 2.5% of ITN-on hypotheses -> contain a digit, so `EnglishTextNormalizer` has almost nothing to absorb. -> -> **On Chinese the sign flips, and the reason is the reference's convention.** -> `BasicTextNormalizer` (used for every non-English language) strips -> punctuation but does no number mapping — and the FLEURS-zh reference is -> itself written with digits (`桥下垂直净空 15 米 … 于 2011 年 8 月完工`). So -> ITN-*off*, which emits spoken form (`二零一一年八月`), mismatches the -> reference on every date and quantity, while ITN-on matches it: -> -> | Model | FLEURS-zh CER off | on | Δ | n | -> | --- | ---: | ---: | ---: | ---: | -> | SenseVoiceSmall F32 | 10.100% | 8.070% | **−2.030pp** | 945 | -> | Fun-ASR-Nano-2512 BF16 | 7.920% | 6.640% | **−1.280pp** | 250 | -> | Fun-ASR-MLT-Nano-2512 BF16 | 7.980% | 6.960% | **−1.020pp** | 250 | -> -> The SenseVoice row is the full 945-utterance split and its arms' 95% CIs -> barely overlap ([9.19, 11.02] off vs [7.25, 8.96] on); the Fun-ASR rows are a -> 250-utterance subset, so compare deltas within a row, not absolutes across -> rows. -> -> Read that as "ITN-on matches the FLEURS-zh scoring convention," not "the -> model recognizes Chinese better with ITN on" — the gain is digit rendering -> lining up with the reference, not improved recognition. -> -> None of this argues for unpinning the harness. The published tables were -> measured at ITN-off and `run_reference_*.py` defaults ITN off; the pin exists -> to keep both sides on the same convention, whichever direction that -> convention happens to favor. -> -> If the library default is ever revisited, this pin stays put unless the -> reference side is re-run to match. `scripts/validate.py` pins `--no-itn` for -> the same two families and the same reason — there the prefix embedding / -> prompt change would break tensor comparison outright. +## Publication profile + +[`catalog/_benchmark_profiles.json`](../../catalog/_benchmark_profiles.json) +is the executable publication contract. For accuracy it requires LibriSpeech +test-clean at every downloaded quant for English-capable models, plus FLEURS +test Q8_0 for every supported language represented by FLEURS. Run missing +cells on Modal without restating datasets, quants, batching, language prompts, +or GPU as flags: + +```bash +modal run scripts/wer/remote/modal_sweep.py::publication_sweep \ + --models # add --plan-only to inspect the expansion +``` + +Score the JSONLs named by the sweep, then ingest and enforce the model's exact +published set: + +```bash +uv run scripts/catalog/ingest_accuracy.py --models +uv run scripts/catalog/check.py --publication-profile --models +``` + +Arbitrary `run.py` and `modal_sweep.py::sweep` invocations remain useful for +experiments, but only profile-stamped full-split reports can be ingested as +published accuracy. +Batch size is not part of a cell's identity: the profile recommends batch 8 +because it is faster and cheaper, a cell measured at batch 1 still satisfies +it, and the catalog row records whichever was run. + +> **ITN is pinned, not inherited.** The runtime ITN default is per-family +> (`sensevoice` resolves it on, `funasr_nano` keeps upstream's `itn=False`), +> but the benchmark follows neither: `run.py` always passes `--no-itn` because +> the reference runs it is scored against produce spoken form. If a family +> default ever flips, the pin stays unless the reference side is re-run to +> match. `scripts/validate.py` pins `--no-itn` for the same two families. **What does and doesn't move WER (measured on whisper-medium F16):** diff --git a/reports/porting/cohere/cohere-transcribe-arabic-07-2026/intake.json b/reports/porting/cohere/cohere-transcribe-arabic-07-2026/intake.json new file mode 100644 index 00000000..26e621bb --- /dev/null +++ b/reports/porting/cohere/cohere-transcribe-arabic-07-2026/intake.json @@ -0,0 +1,169 @@ +{ + "schema_version": "transcribe-intake-v1", + "family": "cohere", + "hf_repo": "CohereLabs/cohere-transcribe-arabic-07-2026", + "hf_revision": "0a8193caa4f3f92131471ab08824e488141cb392", + "sources": { + "config": { + "kind": "hf_file", + "path": "config.json", + "status": "found", + "detail": "Structurally identical to cohere-transcribe-03-2026 except: no top-level vocab_size (head.num_classes=16384 carries it), no auto_map, supported_languages=[en,ar], decoding.beam.max_generation_delta=-256" + }, + "preprocessor": { + "kind": "hf_file", + "path": "preprocessor_config.json", + "status": "found", + "detail": "Near-empty (processor_class only); frontend params live in config.json's preprocessor block, which is identical to 03-2026" + }, + "tokenizer_config": { + "kind": "hf_file", + "path": "tokenizer_config.json", + "status": "found" + }, + "tokenizer_json": { + "kind": "hf_file", + "path": "tokenizer.json", + "status": "found" + }, + "generation_config": { + "kind": "hf_file", + "path": "generation_config.json", + "status": "found", + "detail": "Same special token IDs as 03-2026 (bos=4, eos=3, pad=2, decoder_start=13764)" + }, + "safetensors_metadata": { + "kind": "hf_api", + "path": "safetensors header (range request)", + "status": "found", + "detail": "BF16=2104 (all floating tensors), I64=48 (integer buffers); identical layout and byte size to 03-2026, retrained weights" + }, + "model_card": { + "kind": "hf_api", + "path": "https://huggingface.co/CohereLabs/cohere-transcribe-arabic-07-2026", + "status": "found", + "detail": "Arabic-focused (dialects + code-switching) adaptation of the 03-2026 architecture; en secondary; gated:auto" + }, + "reference_modeling_code": { + "kind": "reference_code", + "path": "transcribe.cpp/scripts/convert-cohere.py, transcribe.cpp/src/arch/cohere/", + "status": "found", + "detail": "Same architecture as cohere-transcribe-03-2026; native Transformers (CohereAsrForConditionalGeneration, trust_remote_code=False) is the reference. tokenizer.model is blob-identical to 03-2026." + } + }, + "variants": [ + { + "name": "cohere-transcribe-arabic-07-2026", + "memory_gb": 5.0, + "files": ["model.safetensors", "config.json", "preprocessor_config.json", "tokenizer.json", "tokenizer.model"] + } + ], + "config": { + "architecture_candidates": ["encoder-decoder"], + "key_fields": { + "architectures": ["CohereAsrForConditionalGeneration"], + "model_type": "cohere_asr", + "vocab_size": 16384, + "is_encoder_decoder": true, + "max_seq_len": 1024, + "max_audio_clip_s": 35, + "encoder.type": "ConformerEncoder (FastConformer, NeMo-derived)", + "encoder.n_layers": 48, + "encoder.d_model": 1280, + "encoder.n_heads": 8, + "encoder.ff_expansion_factor": 4, + "encoder.conv_kernel_size": 9, + "encoder.subsampling": "dw_striding", + "encoder.subsampling_factor": 8, + "encoder.subsampling_conv_channels": 256, + "encoder.self_attention_model": "rel_pos", + "encoder.feat_in": 128, + "encoder.untie_biases": true, + "transf_decoder.hidden_size": 1024, + "transf_decoder.inner_size": 4096, + "transf_decoder.num_attention_heads": 8, + "transf_decoder.num_layers": 8, + "transf_decoder.hidden_act": "relu", + "transf_decoder.pre_ln": true, + "transf_decoder.max_sequence_length": 1024, + "transf_decoder.learn_positional_encodings": false, + "head.hidden_size": 1024, + "head.num_classes": 16384, + "head.activation": "relu", + "decoding.strategy": "beam", + "decoding.beam_size": 1, + "prompt_format": "cohere_asr" + }, + "varying_across_variants": [ + "supported_languages ([en,ar] vs 14 languages in 03-2026)", + "decoding.beam.max_generation_delta (-256 vs 50; not consumed by converter or C++)", + "top-level vocab_size absent (converter falls back to head.num_classes)" + ] + }, + "dtype": { + "expected": "bfloat16", + "source": "weights_header", + "evidence": "safetensors header BF16=2104 (all floating tensors), I64=48 (integer buffers), fetched via authenticated range request against revision 0a8193ca; identical distribution and total byte size to cohere-transcribe-03-2026.", + "details": { + "config_declared": null, + "header_distribution": { + "I64": 48, + "BF16": 2104 + } + }, + "expected_f32_tensors": [] + }, + "frontend": { + "sample_rate": 16000, + "n_mels": 128, + "hop_length": 160, + "fft_size": 512, + "window": "hann_periodic", + "normalization": "per_feature", + "preemphasis": 0.97, + "dither": 1e-05, + "center": true, + "padding_mode": "constant", + "mel_filterbank_norm": "slaney" + }, + "tokenizer": { + "type": "sentencepiece", + "vocab_size": 16384, + "special_tokens": { + "bos": 4, + "eos": 3, + "pad": 2, + "decoder_start": 13764 + }, + "has_language_tokens": true, + "vocab_sha256": "c6e708e8cd5c72afb475a2268fcbae2cf86651d0439aeff395734e3c3cc85de5" + }, + "capabilities": { + "languages": ["en", "ar"], + "language_detection": false, + "translation": false, + "timestamps": ["none"], + "streaming": false, + "speaker_diarization": false + }, + "upstream_benchmarks": [], + "reference_framework": "transformers", + "reference_rationale": "Same as cohere-transcribe-03-2026: native Hugging Face Transformers (CohereAsrForConditionalGeneration, trust_remote_code=False) is the canonical reference. This checkpoint is not packaged for NeMo; the encoder is merely FastConformer-derived. The repo ships trust_remote_code modeling files but the config carries no auto_map, and the family doc's discussion-#28 warning against the remote-code path stands.", + "architecture_pattern": "encoder-decoder", + "known_risks": [ + "Retrained weights on the 03-2026 architecture: all family-level structural validation carries over, but numerical goldens and WER baselines must be re-measured for this checkpoint.", + "tokenizer.model is blob-identical to 03-2026, so slot-token IDs (<|ar|>, <|pnc|>, ...) and the 10-token prompt shape are unchanged; tokenizer.json differs only in metadata.", + "config.json omits top-level vocab_size; converters/tools must fall back to head.num_classes.", + "FLEURS Arabic is the ar_eg (Egyptian) config; the model card claims dialect + code-switching coverage, so published Cohere numbers on other Arabic sets will not be directly comparable." + ], + "intake_gaps": [ + { + "field": "upstream_benchmarks", + "reason": "Model card Arabic WER/CER table not recorded at intake time; add from Cohere's technical report when available." + }, + { + "field": "variants[].memory_gb", + "reason": "Estimated from BF16 GGUF size; not measured against actual peak inference memory." + } + ] +} diff --git a/samples/README.md b/samples/README.md new file mode 100644 index 00000000..d3fd433f --- /dev/null +++ b/samples/README.md @@ -0,0 +1,67 @@ +# Bench and smoke-test samples + +Audio here is fixture, not test data: a published xRT figure is tied to the +exact file that produced it, so a clip named in `catalog/_benchmark_profiles.json` +must never be regenerated, re-encoded, or swapped for a different take. Add a +new file under a new name instead. + +All files are 16 kHz mono 16-bit PCM WAV, the only format the CLI accepts. + +## Per-language bench clips (FLEURS) + +Pulled from the FLEURS test split already on disk under `samples/wer/`, two per +language: a short clip at the same 11 s as `jfk.wav`, and the longest available +utterance, to stand in for `dots.wav`. A single-language fine-tune decodes +English out of distribution and can loop until its position cap, so these exist +to give those variants a benchmark that measures transcription rather than a +repetition loop. + +Source: [google/fleurs](https://huggingface.co/datasets/google/fleurs), test +split, licensed **CC-BY-4.0**. The FLEURS utterance id is the original filename +and is recorded here so each clip can be traced back. + +| file | duration | config | FLEURS utterance id | +| --- | ---: | --- | --- | +| `ar-short.wav` | 11.00 s | `ar_eg` | 13811922940508003061 | +| `ar-long.wav` | 25.74 s | `ar_eg` | 12943222207631713208 | +| `ja-short.wav` | 10.98 s | `ja_jp` | 8296538110626558656 | +| `ja-long.wav` | 28.20 s | `ja_jp` | 9518252661993015549 | +| `ko-short.wav` | 10.98 s | `ko_kr` | 11859537746411417197 | +| `ko-long.wav` | 25.80 s | `ko_kr` | 15152963524414515048 | +| `ru-short.wav` | 10.98 s | `ru_ru` | 2668014690611039917 | +| `ru-long.wav` | 33.84 s | `ru_ru` | 10388523902227354213 | +| `uk-short.wav` | 10.98 s | `uk_ua` | 7177130321767122387 | +| `uk-long.wav` | 28.92 s | `uk_ua` | 12340201221281017924 | +| `vi-short.wav` | 10.98 s | `vi_vn` | 9897090359729012443 | +| `vi-long.wav` | 25.10 s | `vi_vn` | 16802725889904809484 | +| `zh-short.wav` | 11.00 s | `cmn_hans_cn` | 17639585860488007329 | +| `zh-long.wav` | 31.12 s | `cmn_hans_cn` | 5655534691025010514 | + +Regenerate the source pool, not the clips themselves, with +`uv run scripts/wer/ingest.py fleurs `. + +## Everything else + +These predate this file and arrived inside unrelated commits, so their source +and licence were never recorded. Treat the provenance column as a known gap to +resolve, not as a statement that the files are unencumbered. + +| file | duration | first appeared in | source | +| --- | ---: | --- | --- | +| `jfk.wav` | 11.0 s | `785fe3e3` working parakeet impl | unrecorded | +| `dots.wav` | 35.3 s | `785fe3e3` working parakeet impl | unrecorded | +| `dots-full.wav` | 305.9 s | `f3d68a6b` add more reference audio | unrecorded | +| `german.wav` | 29.3 s | `d46b961d` init cohere support | unrecorded | +| `ja.wav` | 7.2 s | `3a8aa207` basic working sensevoice | unrecorded | +| `ko.wav` | 4.6 s | `3a8aa207` basic working sensevoice | unrecorded | +| `zh.wav` | 5.6 s | `3a8aa207` basic working sensevoice | unrecorded | +| `yue.wav` | 5.2 s | `3a8aa207` basic working sensevoice | unrecorded | +| `ru.wav` | 2.2 s | `c6a93377` working gigaam | unrecorded | +| `death.wav`, `love-loss.wav`, `whole-earth.wav`, `jobs-silence.wav`, `noise.wav` | 15 s - 233 s | `f3d68a6b` add more reference audio | unrecorded | +| `product-names.wav` | 56.1 s | `060e7afa` finish up parity with whisper | unrecorded | +| `cj-swimming-drop.wav` | 70.1 s | `98715301` add iOS example (#36) | unrecorded | +| `multitalker-2spk-mix.wav`, `sortformer-2spk-mix.wav`, `diar/` | various | diarization fixtures | unrecorded | + +The bare `ja.wav`, `ko.wav`, `zh.wav`, `yue.wav` and `ru.wav` clips are +referenced by family-doc smoke tests and by gigaam's profile override; they stay +where they are. The `-short` / `-long` pairs above are the bench fixtures. diff --git a/samples/ar-long.wav b/samples/ar-long.wav new file mode 100644 index 00000000..cffb4c13 Binary files /dev/null and b/samples/ar-long.wav differ diff --git a/samples/ar-short.wav b/samples/ar-short.wav new file mode 100644 index 00000000..e2c88518 Binary files /dev/null and b/samples/ar-short.wav differ diff --git a/samples/ja-long.wav b/samples/ja-long.wav new file mode 100644 index 00000000..5b2a89f5 Binary files /dev/null and b/samples/ja-long.wav differ diff --git a/samples/ja-short.wav b/samples/ja-short.wav new file mode 100644 index 00000000..12e0eaf3 Binary files /dev/null and b/samples/ja-short.wav differ diff --git a/samples/ko-long.wav b/samples/ko-long.wav new file mode 100644 index 00000000..be57cf15 Binary files /dev/null and b/samples/ko-long.wav differ diff --git a/samples/ko-short.wav b/samples/ko-short.wav new file mode 100644 index 00000000..ca416f5b Binary files /dev/null and b/samples/ko-short.wav differ diff --git a/samples/ru-long.wav b/samples/ru-long.wav new file mode 100644 index 00000000..d595d248 Binary files /dev/null and b/samples/ru-long.wav differ diff --git a/samples/ru-short.wav b/samples/ru-short.wav new file mode 100644 index 00000000..3f68ca60 Binary files /dev/null and b/samples/ru-short.wav differ diff --git a/samples/uk-long.wav b/samples/uk-long.wav new file mode 100644 index 00000000..5f34dcb7 Binary files /dev/null and b/samples/uk-long.wav differ diff --git a/samples/uk-short.wav b/samples/uk-short.wav new file mode 100644 index 00000000..61a5d73c Binary files /dev/null and b/samples/uk-short.wav differ diff --git a/samples/vi-long.wav b/samples/vi-long.wav new file mode 100644 index 00000000..d8feb2fd Binary files /dev/null and b/samples/vi-long.wav differ diff --git a/samples/vi-short.wav b/samples/vi-short.wav new file mode 100644 index 00000000..213bb50c Binary files /dev/null and b/samples/vi-short.wav differ diff --git a/samples/zh-long.wav b/samples/zh-long.wav new file mode 100644 index 00000000..758bf748 Binary files /dev/null and b/samples/zh-long.wav differ diff --git a/samples/zh-short.wav b/samples/zh-short.wav new file mode 100644 index 00000000..d21d2aea Binary files /dev/null and b/samples/zh-short.wav differ diff --git a/scripts/bench/run.py b/scripts/bench/run.py index 3e103bee..7aad67f2 100755 --- a/scripts/bench/run.py +++ b/scripts/bench/run.py @@ -17,7 +17,8 @@ Usage: uv run scripts/bench/run.py # all variants - uv run scripts/bench/run.py --models Qwen3-ASR-0.6B + uv run scripts/bench/run.py --profile --models Qwen3-ASR-0.6B # publication matrix + uv run scripts/bench/run.py --models Qwen3-ASR-0.6B # experiment uv run scripts/bench/run.py --models Qwen/Qwen3-ASR-0.6B # HF slug form uv run scripts/bench/run.py --models Qwen3-ASR-0.6B,Qwen3-ASR-1.7B uv run scripts/bench/run.py --models parakeet-tdt-0.6b-v3 --quants f16 @@ -103,6 +104,12 @@ from datetime import datetime, timezone from pathlib import Path +CATALOG_DIR = Path(__file__).resolve().parents[1] / "catalog" +if str(CATALOG_DIR) not in sys.path: + sys.path.insert(0, str(CATALOG_DIR)) +import common as catalog_common # noqa: E402 +import profiles as benchmark_profiles # noqa: E402 + DEFAULT_QUANTS = ["f16", "q8_0", "q4_k_m"] DEFAULT_SAMPLES = ["jfk", "dots"] @@ -334,6 +341,18 @@ def discover_matrix(repo: Path, model_tokens: list[str] | None, return cells +def catalog_key(records: dict, variant: str) -> str | None: + """Catalog variant for an on-disk dir name. Model dirs keep the upstream + repo casing (models/Qwen3-ASR-0.6B) while catalog keys are lowercase + (qwen3-asr-0.6b); the two must be reconciled for profile lookups.""" + if variant in records: + return variant + lower = variant.lower() + if lower in records: + return lower + return next((key for key in records if key.lower() == lower), None) + + def group_by_variant(cells: list[Cell]) -> dict[str, list[Cell]]: groups: dict[str, list[Cell]] = {} for cell in cells: @@ -657,16 +676,25 @@ def parse_args() -> argparse.Namespace: "'Qwen3-ASR-0.6B', HF form like 'Qwen/Qwen3-ASR-0.6B', " "or paths to .gguf files); default: all variants " "under models/") - p.add_argument("--quants", type=str, default=",".join(DEFAULT_QUANTS)) - p.add_argument("--samples", type=str, default=",".join(DEFAULT_SAMPLES)) - p.add_argument("--iters", type=int, default=2) - p.add_argument("--warmup", type=int, default=1) + p.add_argument("--quants", type=str, default=None, + help="comma-separated quants (default: f16,q8_0,q4_k_m). " + "Not used with --profile.") + p.add_argument("--samples", type=str, default=None, + help="comma-separated samples (default: jfk,dots). " + "Not used with --profile.") + p.add_argument("--iters", type=int, default=None) + p.add_argument("--warmup", type=int, default=None) p.add_argument("--backends", type=str, default=None, help="comma-separated list: metal,cpu,vulkan or 'all' " "(default: auto-detect)") p.add_argument("--name", type=str, default=None, help="stable label for named baselines " "(replaces timestamp in output filename)") + p.add_argument("--profile", nargs="?", const="", + help="run a catalog publication profile; optionally name " + "it (default: catalog/_benchmark_profiles.json default). " + "The profile supplies quants, samples, target backends, " + "iteration counts, and cooldown policy.") p.add_argument("--bench-bin", type=Path, default=None, help="legacy override for the bench binary " "(only valid when exactly one backend is selected)") @@ -674,7 +702,7 @@ def parse_args() -> argparse.Namespace: help="output root (default: reports/perf)") p.add_argument("--dry-run", action="store_true", help="print selected backends + matrix without running") - p.add_argument("--cooldown-tctl-c", type=float, default=0.0, + p.add_argument("--cooldown-tctl-c", type=float, default=None, help="if >0, wait between cells for k10temp Tctl to drop " "below this value (°C) to avoid thermal bias; " "publication benches use 55") @@ -686,6 +714,36 @@ def parse_args() -> argparse.Namespace: return p.parse_args() +# What a report must carry to be ingested (scripts/catalog/ingest_perf.py) +# and compared (scripts/bench/compare.py). A gap here is a bench-harness +# regression, so the driver refuses to write the file rather than leave a +# report that looks complete and is not. +REQUIRED_TOP = ("schema", "timestamp", "machine", "git_sha", "variant", "backend", + "iters", "warmup", "runs") +REQUIRED_MACHINE = ("slug", "os") +REQUIRED_RUN = ("model_path", "sample_path", "sample_duration_s", "per_iter", + "summary", "rtf_wall_mean", "transcript_sha256") +REQUIRED_ITER = ("mel_ms", "encode_ms", "decode_ms", "total_ms", "wall_ms") + + +def report_gaps(report: dict) -> list[str]: + """Names of required fields the report lacks; empty when it is complete.""" + gaps = [key for key in REQUIRED_TOP if key not in report] + gaps += [f"machine.{key}" for key in REQUIRED_MACHINE + if key not in (report.get("machine") or {})] + for index, run in enumerate(report.get("runs") or []): + gaps += [f"runs[{index}].{key}" for key in REQUIRED_RUN if key not in run] + if not run.get("per_iter"): + gaps.append(f"runs[{index}].per_iter is empty") + for it_index, it in enumerate(run.get("per_iter") or []): + gaps += [f"runs[{index}].per_iter[{it_index}].{key}" + for key in REQUIRED_ITER if key not in it] + summary = run.get("summary") or {} + gaps += [f"runs[{index}].summary.{key}.mean" for key in ("total_ms", "wall_ms") + if (summary.get(key) or {}).get("mean") is None] + return gaps + + def _run_one_backend(backend: BackendSpec, by_variant: dict[str, list[Cell]], args: argparse.Namespace, repo: Path, machine: dict, @@ -693,9 +751,33 @@ def _run_one_backend(backend: BackendSpec, git_sha: str) -> int: """Run the full variant matrix against a single backend. Returns exit code.""" exit_code = 0 - name_slug = slugify(args.name) if args.name else None for variant, group in by_variant.items(): + # Profile runs contain only cells assigned to this machine/backend, + # including model-specific reviewed exceptions. + if args.profile is not None: + key = catalog_key(args._catalog_records, variant) + if key is None: + continue + record = args._catalog_records[key] + expected = benchmark_profiles.apply_exceptions( + record, "speed", + benchmark_profiles.expected_speed(record, args._profile_data)) + expected_keys = { + (cell["machine"], cell["backend"], cell["quant"].lower(), cell["sample"]) + for cell in expected + } + machine_slug = benchmark_profiles.canonical_machine(machine["slug"]) + group = [cell for cell in group + if (machine_slug, backend.name, cell.quant.lower(), cell.sample) + in expected_keys] + if not group: + continue + + # A publication run names itself after the variant unless told + # otherwise, so the file on disk says what it is. + run_name = args.name or (f"{variant}-publication" if args.profile is not None else None) + name_slug = slugify(run_name) if run_name else None runs: list[dict] = [] for cell in group: print(f"[{backend.name}][{variant}] {cell.quant} \u00d7 {cell.sample} ...", @@ -723,7 +805,10 @@ def _run_one_backend(backend: BackendSpec, aggregate = { "schema": "transcribe-bench-driver-v1", "timestamp": timestamp, - "name": args.name or "", + "name": run_name or "", + # Eligibility for the catalog is a property the run declares, not + # something an importer infers from the filename later. + "publication_profile": args._profile_id, "machine": machine, "git_sha": git_sha, "variant": variant, @@ -732,6 +817,12 @@ def _run_one_backend(backend: BackendSpec, "warmup": args.warmup, "runs": runs, } + missing = report_gaps(aggregate) + if missing: + print(f"[{backend.name}][{variant}] refusing to write an incomplete " + f"report: {', '.join(missing)}", file=sys.stderr) + exit_code = 1 + continue out_path.write_text(json.dumps(aggregate, indent=2) + "\n") try: rel = out_path.relative_to(repo) @@ -753,15 +844,71 @@ def main() -> int: timestamp = now_utc_iso() slug_ts = timestamp_for_filename(timestamp) - quants = [q.strip() for q in args.quants.split(",") if q.strip()] - sample_stems = [s.strip() for s in args.samples.split(",") if s.strip()] model_tokens: list[str] | None = None if args.models: model_tokens = [t.strip() for t in args.models.split(",") if t.strip()] + args._profile_id = None + args._profile_data = None + args._catalog_records = {} + if args.profile is not None: + conflicting = [name for name, value in ( + ("--quants", args.quants), ("--samples", args.samples), + ("--backends", args.backends), ("--iters", args.iters), + ("--warmup", args.warmup), ("--cooldown-tctl-c", args.cooldown_tctl_c), + ("--name", args.name)) if value is not None] + if conflicting: + print(f"error: --profile supplies {', '.join(conflicting)}; do not override it", + file=sys.stderr) + return 2 + try: + args._profile_id, args._profile_data = benchmark_profiles.load_profile( + args.profile or None) + except (OSError, ValueError, json.JSONDecodeError) as exc: + print(f"error: {exc}", file=sys.stderr) + return 2 + args._catalog_records = catalog_common.load_records() + target = benchmark_profiles.target_for_machine( + args._profile_data, machine["slug"]) + if target is None: + print(f"error: machine {machine['slug']!r} is not a target in " + f"profile {args._profile_id}", file=sys.stderr) + return 2 + speed = args._profile_data["speed"] + quants = sorted({item["quant"] for record in args._catalog_records.values() + for item in record.get("downloads", [])}) + # A single-language variant benches on its own language rather than + # jfk/dots. The candidate matrix has to contain every such clip before + # the per-variant expected-cell filter in _run_one_backend can select + # it, or those variants match nothing and are silently skipped. + sample_stems = benchmark_profiles.all_speed_samples( + args._profile_data, args._catalog_records) + args.backends = ",".join(target["backends"]) + args.iters = int(speed["iterations"]) + args.warmup = int(speed["warmup"]) + args.cooldown_tctl_c = float(target.get("cooldown_tctl_c", 0.0)) + else: + quants = [q.strip() for q in (args.quants or ",".join(DEFAULT_QUANTS)).split(",") + if q.strip()] + sample_stems = [s.strip() for s in (args.samples or ",".join(DEFAULT_SAMPLES)).split(",") + if s.strip()] + args.iters = 2 if args.iters is None else args.iters + args.warmup = 1 if args.warmup is None else args.warmup + args.cooldown_tctl_c = 0.0 if args.cooldown_tctl_c is None else args.cooldown_tctl_c + backends = resolve_backends(repo, args.backends, args.bench_bin) cells = discover_matrix(repo, model_tokens, quants, sample_stems) + if args.profile is not None: + # A local models directory may contain unpublished experiments. A + # profile runs only files named by catalog downloads. + allowed = { + (variant.lower(), item["quant"].lower()) + for variant, record in args._catalog_records.items() + for item in record.get("downloads", []) + } + cells = [cell for cell in cells + if (cell.variant.lower(), cell.quant.lower()) in allowed] by_variant = group_by_variant(cells) if args.dry_run: diff --git a/scripts/build_canary_qwen_oracle.py b/scripts/build_canary_qwen_oracle.py deleted file mode 100644 index b09debda..00000000 --- a/scripts/build_canary_qwen_oracle.py +++ /dev/null @@ -1,196 +0,0 @@ -#!/usr/bin/env python3 -""" -Build the per-variant dump_coverage.json files and the family-level -tests/tolerances/canary_qwen.json from the on-disk tensor sidecars -produced by scripts/dump_reference_canary_qwen_nemo.py. - -One-shot helper used at the end of Stage 2. Not invoked from the runtime. -""" -from __future__ import annotations - -import json -import sys -from pathlib import Path - -ROOT = Path(__file__).resolve().parent.parent -BUILD = ROOT / "build" / "validate" / "canary_qwen" -TOL = ROOT / "tests" / "tolerances" / "canary_qwen.json" -FAMILY = "canary_qwen" - -VARIANTS = [ - "canary-qwen-2.5b", -] - - -def is_tensor_sidecar(meta: dict) -> bool: - return all(k in meta for k in ("name", "shape", "dtype", "layout")) - - -def walk_variant(variant: str) -> list[dict]: - """Return one entry per tensor sidecar under build/validate/canary_qwen/.""" - variant_dir = BUILD / variant - if not variant_dir.exists(): - print(f" warn: {variant_dir} missing") - return [] - entries: list[dict] = [] - for json_path in sorted(variant_dir.rglob("*.json")): - try: - meta = json.loads(json_path.read_text()) - except json.JSONDecodeError as e: - print(f" warn: bad JSON at {json_path}: {e}") - continue - if not is_tensor_sidecar(meta): - continue - rel = json_path.relative_to(variant_dir) - parts = rel.parts - # Two acceptable layouts: - # //ref/.json (stage 2 oracle layout) - # /ref/.json (single-pass dumper layout) - if len(parts) >= 4 and parts[-2] == "ref": - case = parts[0] - stage_dir = parts[1] - elif len(parts) == 3 and parts[1] == "ref": - case = parts[0] - stage_dir = parts[0] # case == stage_dir; ref dumper writes everything in one shot - else: - # cpp dumps and unknown layouts are skipped silently — the - # coverage file is for ref tensors only. - continue - entries.append({ - "case": case, - "stage_dir": stage_dir, - "stage": meta.get("stage", ""), - "name": meta["name"], - "shape": list(meta["shape"]), - "dtype": meta["dtype"], - "rel_path": str(rel), - }) - return entries - - -def write_coverage(variant: str, entries: list[dict]) -> Path: - out = BUILD / variant / "dump_coverage.json" - payload = { - "family": FAMILY, - "variant": variant, - "tensors": entries, - } - out.write_text(json.dumps(payload, indent=2) + "\n") - return out - - -def aggregate_tolerances(per_variant: dict[str, list[dict]]) -> dict[str, dict]: - """For each tensor name, take the max p99_abs and rms across variants - where that tensor exists, and size the magnitude-aware budget against - that worst-case magnitude. - - Returns: {name: {max_abs, mean_abs, _provisional, _seen_in}} - """ - stats: dict[str, list[tuple[str, float, float]]] = {} - for variant, entries in per_variant.items(): - for e in entries: - json_path = BUILD / variant / e["rel_path"] - meta = json.loads(json_path.read_text()) - p99 = float(meta.get("p99_abs", 0.0)) - rms = float(meta.get("rms", 0.0)) - stats.setdefault(e["name"], []).append((variant, p99, rms)) - - out: dict[str, dict] = {} - for name, rows in sorted(stats.items()): - worst_p99 = max(r[1] for r in rows) - worst_rms = max(r[2] for r in rows) - max_abs = max(1e-4 * worst_p99, 1e-6) - mean_abs = max(1e-5 * worst_rms, 1e-6) - out[name] = { - "max_abs": max_abs, - "mean_abs": mean_abs, - "_provisional": True, - "_seen_in": sorted({r[0] for r in rows}), - } - return out - - -def main() -> int: - if not BUILD.exists(): - print(f"error: {BUILD} does not exist; run the dumper first") - return 2 - - per_variant: dict[str, list[dict]] = {} - print("=== dump_coverage.json per variant ===") - for v in VARIANTS: - entries = walk_variant(v) - if not entries: - print(f" {v}: 0 tensors (skipping coverage write)") - continue - cov_path = write_coverage(v, entries) - per_variant[v] = entries - print(f" {v}: {len(entries)} tensors -> {cov_path.relative_to(ROOT)}") - - print() - print("=== tolerances aggregation ===") - tols = aggregate_tolerances(per_variant) - payload: dict = { - "_comment": [ - "Canary-Qwen SALM (audio-LLM) per-tensor tolerances for", - "compare_tensors.py.", - "", - "CORRECTNESS REGIME", - "- Reference: NVIDIA NeMo (>=2.5.0)", - " nemo.collections.speechlm2.models.SALM running BF16", - " inference on CPU. Dumper: scripts/dump_reference_canary_qwen_nemo.py.", - "- C++: ggml compute (CPU/Metal/Vulkan), weights stored as BF16", - " in the accuracy GGUF.", - "- KV cache dtype: AUTO -> F16 for the autoregressive Qwen3 LM.", - "- Mel frontend: NeMo AudioToMelSpectrogramPreprocessor with", - " dither overridden to 0.0 at dump time (config declares 1e-5).", - "", - "DUMP POINTS", - " enc.mel.in preprocessor mel-spectrogram output", - " enc.pre_encode.out after FastConformer dw-striding subsampling (factor 8)", - " enc.pos_emb relative positional encoding tensor", - " enc.block.{i}.out encoder conformer block samples (0, mid=16, last=31)", - " enc.final final encoder output (B, T_enc, 1024)", - " perception.proj.out after AudioPerceptionModule projection (B, T_enc, 2048)", - " dec.token_emb LM input embeddings pre-injection", - " dec.audio_injected LM input embeddings AFTER audio scatter", - " dec.block.{i}.out Qwen3 LM block outputs (0, mid=14, last=27)", - " dec.out_before_head after LM final RMSNorm", - " dec.logits_raw.gen0 first-step LM head logits over the Qwen3 vocab", - "", - "ENTRY SOURCING (provisional)", - "- Per-tensor max_abs = max(1e-4 x p99_abs, 1e-6)", - "- Per-tensor mean_abs = max(1e-5 x rms, 1e-6)", - "- All entries carry _provisional: true. Stage 4 finalizes", - " against observed C++ drift and removes the flag tensor-by-tensor.", - "", - "EXPECTED STAGE 4 RETUNING", - "- enc.* magnitudes mirror canary-1b-flash (FastConformer is shared);", - " dominant drift source remains STFT precision (C++ fp64 vs NeMo fp32),", - " concentrated at the trailing-edge mel frame.", - "- dec.audio_injected and downstream LM tensors will see additional", - " drift from the BF16 LM weights and the perception projection. Expect", - " to widen dec.block.{mid,last} and dec.logits_raw.gen0 against", - " observed C++ values once Stage 4 is brought up.", - "", - "DO NOT SHIP a model while _provisional entries remain.", - ], - **tols, - } - TOL.parent.mkdir(parents=True, exist_ok=True) - TOL.write_text(json.dumps(payload, indent=2) + "\n") - - tol_max_vals = sorted(v["max_abs"] for v in tols.values()) - n = len(tol_max_vals) - print(f" wrote {TOL.relative_to(ROOT)} with {n} tensor entries") - if n: - print( - f" max_abs distribution: " - f"min={tol_max_vals[0]:.3e} " - f"median={tol_max_vals[n // 2]:.3e} " - f"max={tol_max_vals[-1]:.3e}" - ) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/build_gigaam_oracle.py b/scripts/build_gigaam_oracle.py deleted file mode 100644 index 65cd889f..00000000 --- a/scripts/build_gigaam_oracle.py +++ /dev/null @@ -1,173 +0,0 @@ -#!/usr/bin/env python3 -""" -Build the per-variant dump_coverage.json files and the family-level -tests/tolerances/gigaam.json from the on-disk tensor sidecars produced -by scripts/dump_reference_gigaam_author.py. - -One-shot helper used at the end of Stage 2. Not invoked from the -runtime. -""" -from __future__ import annotations - -import json -import sys -from pathlib import Path - -ROOT = Path(__file__).resolve().parent.parent -BUILD = ROOT / "build" / "validate" / "gigaam" -TOL = ROOT / "tests" / "tolerances" / "gigaam.json" -FAMILY = "gigaam" - -VARIANTS = [ - "gigaam-v3-e2e-rnnt", - "gigaam-v3-e2e-ctc", - "gigaam-v3-rnnt", - "gigaam-v3-ctc", -] - - -def is_tensor_sidecar(meta: dict) -> bool: - return all(k in meta for k in ("name", "shape", "dtype", "layout")) - - -def walk_variant(variant: str) -> list[dict]: - """Return one entry per tensor sidecar under build/validate/gigaam/.""" - variant_dir = BUILD / variant - if not variant_dir.exists(): - print(f" warn: {variant_dir} missing") - return [] - entries: list[dict] = [] - for json_path in sorted(variant_dir.rglob("*.json")): - try: - meta = json.loads(json_path.read_text()) - except json.JSONDecodeError as e: - print(f" warn: bad JSON at {json_path}: {e}") - continue - if not is_tensor_sidecar(meta): - continue - rel = json_path.relative_to(variant_dir) - parts = rel.parts - # Layout: //ref/.json - if len(parts) < 4 or parts[-2] != "ref": - print(f" warn: unexpected layout at {json_path}") - continue - case = parts[0] - stage_dir = parts[1] - entries.append({ - "case": case, - "stage_dir": stage_dir, - "stage": meta.get("stage", ""), - "name": meta["name"], - "shape": list(meta["shape"]), - "dtype": meta["dtype"], - "rel_path": str(rel), - }) - return entries - - -def write_coverage(variant: str, entries: list[dict]) -> Path: - out = BUILD / variant / "dump_coverage.json" - payload = { - "family": FAMILY, - "variant": variant, - "tensors": entries, - } - out.write_text(json.dumps(payload, indent=2) + "\n") - return out - - -def aggregate_tolerances(per_variant: dict[str, list[dict]]) -> dict[str, dict]: - """For each tensor name, take the max p99_abs and rms across variants - where that tensor exists, and size the magnitude-aware budget against - that worst-case magnitude. - - Returns: {name: {max_abs, mean_abs, _provisional, _seen_in}} - """ - # Collect (name -> [(variant, rel_path, p99_abs, rms)]) - stats: dict[str, list[tuple[str, float, float]]] = {} - for variant, entries in per_variant.items(): - for e in entries: - json_path = BUILD / variant / e["rel_path"] - meta = json.loads(json_path.read_text()) - p99 = float(meta.get("p99_abs", 0.0)) - rms = float(meta.get("rms", 0.0)) - stats.setdefault(e["name"], []).append((variant, p99, rms)) - - out: dict[str, dict] = {} - for name, rows in sorted(stats.items()): - worst_p99 = max(r[1] for r in rows) - worst_rms = max(r[2] for r in rows) - max_abs = max(1e-4 * worst_p99, 1e-6) - mean_abs = max(1e-5 * worst_rms, 1e-6) - out[name] = { - "max_abs": max_abs, - "mean_abs": mean_abs, - "_provisional": True, - "_seen_in": sorted({r[0] for r in rows}), - } - return out - - -def main() -> int: - if not BUILD.exists(): - print(f"error: {BUILD} does not exist; run the dumper first") - return 2 - - per_variant: dict[str, list[dict]] = {} - print("=== dump_coverage.json per variant ===") - for v in VARIANTS: - entries = walk_variant(v) - if not entries: - print(f" {v}: 0 tensors (skipping coverage write)") - continue - cov_path = write_coverage(v, entries) - per_variant[v] = entries - print(f" {v}: {len(entries)} tensors -> {cov_path.relative_to(ROOT)}") - - print() - print("=== tolerances aggregation ===") - tols = aggregate_tolerances(per_variant) - payload: dict = { - "_comment": [ - "GigaAM per-tensor tolerances for compare_tensors.py.", - "", - "CORRECTNESS REGIME", - "- Reference: salute-developers/GigaAM @ 6e4b027c (author repo)", - " loaded via gigaam.load_model(..., fp16_encoder=False, device='cpu').", - " Dumper: scripts/dump_reference_gigaam_author.py.", - "- C++: ggml CPU fp32 compute, weights stored as F32 in GGUF.", - "- KV cache dtype: not applicable (RNN-T / CTC heads, no attention KV).", - "- Mel frontend: C++ MelFrontend with center=false, htk mel, no slaney norm,", - " log(clamp(x, 1e-9, 1e9)) scaling.", - "", - "ENTRY SOURCING (provisional)", - "- Per-tensor max_abs = max(1e-4 × p99_abs, 1e-6)", - "- Per-tensor mean_abs = max(1e-5 × rms, 1e-6)", - "- When the same tensor name appears in multiple variants, the worst-case", - " p99_abs and rms across variants drive the budget (so a single budget", - " works for all five variants).", - "- All entries carry _provisional: true. Stage 4 finalizes against", - " observed C++ drift and removes the flag tensor-by-tensor.", - "", - "DO NOT SHIP a model while _provisional entries remain.", - ], - **tols, - } - TOL.parent.mkdir(parents=True, exist_ok=True) - TOL.write_text(json.dumps(payload, indent=2) + "\n") - - tol_max_vals = sorted(v["max_abs"] for v in tols.values()) - n = len(tol_max_vals) - print(f" wrote {TOL.relative_to(ROOT)} with {n} tensor entries") - if n: - print( - f" max_abs distribution: " - f"min={tol_max_vals[0]:.3e} " - f"median={tol_max_vals[n // 2]:.3e} " - f"max={tol_max_vals[-1]:.3e}" - ) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/build_granite5_ctc_oracle.py b/scripts/build_granite5_ctc_oracle.py deleted file mode 100755 index 63fac522..00000000 --- a/scripts/build_granite5_ctc_oracle.py +++ /dev/null @@ -1,202 +0,0 @@ -#!/usr/bin/env python3 -""" -Build the per-variant dump_coverage.json file and the family-level -tests/tolerances/granite5_ctc.json from the on-disk tensor sidecars produced -by scripts/dump_reference_granite5_ctc_transformers.py. - -One-shot helper used at the end of Stage 2. Not invoked from the runtime. -""" -from __future__ import annotations - -import json -import sys -from pathlib import Path - -ROOT = Path(__file__).resolve().parent.parent -BUILD = ROOT / "build" / "validate" / "granite5_ctc" -TOL = ROOT / "tests" / "tolerances" / "granite5_ctc.json" -FAMILY = "granite5_ctc" - -VARIANTS = ["granite-speech-5.0-470m-turboctc"] - - -def is_tensor_sidecar(meta: dict) -> bool: - return all(k in meta for k in ("name", "shape", "dtype", "layout")) - - -def walk_variant(variant: str) -> list[dict]: - variant_dir = BUILD / variant - if not variant_dir.exists(): - print(f" warn: {variant_dir} missing") - return [] - entries: list[dict] = [] - for json_path in sorted(variant_dir.rglob("*.json")): - try: - meta = json.loads(json_path.read_text()) - except json.JSONDecodeError as e: - print(f" warn: bad JSON at {json_path}: {e}") - continue - if not is_tensor_sidecar(meta): - continue - rel = json_path.relative_to(variant_dir) - parts = rel.parts - # Layout: //ref/.json - if len(parts) < 4 or parts[-2] != "ref": - print(f" warn: unexpected layout at {json_path}") - continue - entries.append({ - "case": parts[0], - "stage_dir": parts[1], - "stage": meta.get("stage", ""), - "name": meta["name"], - "shape": list(meta["shape"]), - "dtype": meta["dtype"], - "rel_path": str(rel), - }) - return entries - - -def write_coverage(variant: str, entries: list[dict]) -> Path: - out = BUILD / variant / "dump_coverage.json" - out.write_text(json.dumps( - {"family": FAMILY, "variant": variant, "tensors": entries}, indent=2) + "\n") - return out - - -def aggregate_tolerances(per_variant: dict[str, list[dict]]) -> dict[str, dict]: - stats: dict[str, list[tuple[str, str, float, float]]] = {} - for variant, entries in per_variant.items(): - for e in entries: - meta = json.loads((BUILD / variant / e["rel_path"]).read_text()) - stats.setdefault(e["name"], []).append( - (variant, e["case"], - float(meta.get("p99_abs", 0.0)), float(meta.get("rms", 0.0)))) - - out: dict[str, dict] = {} - for name, rows in sorted(stats.items()): - worst_p99 = max(r[2] for r in rows) - worst_rms = max(r[3] for r in rows) - out[name] = { - "max_abs": max(1e-4 * worst_p99, 1e-6), - "mean_abs": max(1e-5 * worst_rms, 1e-6), - "_provisional": True, - "_seen_in": sorted({r[0] for r in rows}), - "_cases": sorted({r[1] for r in rows}), - } - return out - - -COMMENT = [ - "Granite Speech 5.0 TurboCTC per-tensor tolerances for compare_tensors.py.", - "", - "CORRECTNESS REGIME", - "- Reference: ibm-granite/granite-speech-5.0-470m-turboctc @", - " 18ca3c1de6cd092b5a30c39fb0f04550b38ed1a0, loaded via mainline", - " transformers 5.17.0 (AutoModelForCTC + AutoProcessor,", - " attn_implementation=eager). No trust_remote_code.", - " Dumper: scripts/dump_reference_granite5_ctc_transformers.py.", - "- ORACLE DTYPE IS F32, not the BF16 the weights ship as. The BF16->F32", - " upcast is lossless, so these are the shipped weights unchanged, and", - " F32 activations over BF16 weights is exactly the transcribe.cpp", - " compute regime. Measured on jfk, a BF16 forward gives an IDENTICAL", - " transcript and identical token ids but tensors that differ from F32", - " by up to 2-3% of p99_abs (worst: enc.block.1.post_ff2, max |diff| =", - " 0.44) -- roughly 250x the 1e-4 x p99_abs budget below. Dumping a BF16", - " oracle would have forced blind tolerance widening at Stage 4 and", - " hidden real implementation bugs. Do not re-dump at bf16.", - "- C++ (expected at Stage 4): ggml F32 activations on BF16 GGUF weights.", - "- KV cache dtype: not applicable (encoder-only; the 'decoder' is one", - " tied Linear plus an argmax).", - "- Mel frontend: GraniteSpeech5FeatureExtractor -- torchaudio", - " MelSpectrogram (power=2.0, Hann periodic, center=True, reflect,", - " htk scale, norm=None), log10, per-utterance floor at max-8.0, x/4+1,", - " concat deltas(win=3), stack 2 frames -> 320-dim input.", - "", - "CASES", - "- jfk : mel_frames=1100 (EVEN) -> 550 stacked frames, 137 enc frames.", - "- dots : mel_frames=3533 (ODD) -> 1767 stacked frames, 441 enc frames.", - " The two parities take different feature-extractor paths: the odd case", - " right-pads the waveform and keeps the trailing center-pad frame, the", - " even case drops it. Per-tensor budgets take the MAX p99_abs / rms", - " across both cases, so they cover the worst-magnitude instance.", - "- Between them the two cases also cover BOTH residual-pooling parities", - " at BOTH subsampling blocks, which one case cannot do:", - " jfk 550 -> 275 (even in, no frame dropped)", - " -> 137 (odd in, one frame dropped)", - " dots 1767 -> 883 (odd in, one frame dropped)", - " -> 441 (odd in, one frame dropped)", - " The pooled residual is unfold(1,2,2).mean(-1), which silently drops a", - " trailing odd frame, and the conv output is then TRIMMED to that", - " length. A C++ implementation that rounds the other way passes jfk", - " block 0 and fails everywhere else.", - "", - "ENTRY SOURCING (provisional)", - "- Per-tensor max_abs = max(1e-4 x p99_abs, 1e-6)", - "- Per-tensor mean_abs = max(1e-5 x rms, 1e-6)", - "- All entries carry _provisional: true. Stage 4 finalizes against", - " observed C++ drift and removes the flag tensor-by-tensor.", - "", - "TENSORS THAT MATTER MOST AT STAGE 4", - "- enc.block.{0,1}.post_conv is the subsampling observable: it sits", - " AFTER the mean-pooled residual plus the TRIMMED stride-2 conv output,", - " at the halved frame rate (jfk: 550->275 at block 0, 275->137 at", - " block 1). Getting the pool/trim order or the odd-frame drop wrong", - " shows up here first, as a shape or a whole-tensor mismatch.", - "- enc.ctc.mid_logits / enc.ctc.mid_injection are the self-conditioned", - " CTC path after block index 7. mid_logits is a 16384-wide softmax", - " input; a BF16 softmax there is a plausible precision trap.", - "- enc.block.7.out is captured BEFORE the mid-injection is added, so", - " block 8's input equals enc.block.7.out + enc.ctc.mid_injection.", - "- enc.ctc_logits comes from ctc_head, which is weight-TIED to", - " encoder.out (the checkpoint has no ctc_head.* tensors).", - "", - "HOOK SEMANTICS VERIFIED AT STAGE 2 (bit-exact on jfk, max|diff| = 0.0):", - " encoder.out(enc.block.7.out) == enc.ctc.mid_logits", - " encoder.out_mid(softmax(enc.ctc.mid_logits)) == enc.ctc.mid_injection", - " ctc_head(enc.out) == enc.ctc_logits", - " ctc_head.weight/bias are the SAME torch storage as encoder.out", - " (identical data_ptr), confirming tie_word_embeddings=true. The", - " converter must emit that [16384, 1024] matrix once and point both", - " consumers at it. These identities are the Stage 4 contract: they let", - " the C++ be checked sub-step by sub-step without re-deriving which", - " projection feeds which consumer.", - "", - "DO NOT SHIP a model while _provisional entries remain.", -] - - -def main() -> int: - if not BUILD.exists(): - print(f"error: {BUILD} does not exist; run the dumper first") - return 2 - - per_variant: dict[str, list[dict]] = {} - print("=== dump_coverage.json per variant ===") - for v in VARIANTS: - entries = walk_variant(v) - if not entries: - print(f" {v}: 0 tensors (skipping coverage write)") - continue - cov_path = write_coverage(v, entries) - per_variant[v] = entries - n_cases = len({e["case"] for e in entries}) - print(f" {v}: {len(entries)} tensors across {n_cases} case(s) " - f"-> {cov_path.relative_to(ROOT)}") - - print() - print("=== tolerances aggregation ===") - tols = aggregate_tolerances(per_variant) - TOL.parent.mkdir(parents=True, exist_ok=True) - TOL.write_text(json.dumps({"_comment": COMMENT, **tols}, indent=2) + "\n") - - vals = sorted(v["max_abs"] for v in tols.values()) - n = len(vals) - print(f" wrote {TOL.relative_to(ROOT)} with {n} tensor entries") - if n: - print(f" max_abs distribution: min={vals[0]:.3e} " - f"median={vals[n // 2]:.3e} max={vals[-1]:.3e}") - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/build_medasr_oracle.py b/scripts/build_medasr_oracle.py deleted file mode 100644 index 594ee6a3..00000000 --- a/scripts/build_medasr_oracle.py +++ /dev/null @@ -1,167 +0,0 @@ -#!/usr/bin/env python3 -""" -Build the per-variant dump_coverage.json file and the family-level -tests/tolerances/medasr.json from the on-disk tensor sidecars produced -by scripts/dump_reference_medasr_transformers.py. - -One-shot helper used at the end of Stage 2. Not invoked from the runtime. -""" -from __future__ import annotations - -import json -import sys -from pathlib import Path - -ROOT = Path(__file__).resolve().parent.parent -BUILD = ROOT / "build" / "validate" / "medasr" -TOL = ROOT / "tests" / "tolerances" / "medasr.json" -FAMILY = "medasr" - -VARIANTS = ["medasr"] - - -def is_tensor_sidecar(meta: dict) -> bool: - return all(k in meta for k in ("name", "shape", "dtype", "layout")) - - -def walk_variant(variant: str) -> list[dict]: - variant_dir = BUILD / variant - if not variant_dir.exists(): - print(f" warn: {variant_dir} missing") - return [] - entries: list[dict] = [] - for json_path in sorted(variant_dir.rglob("*.json")): - try: - meta = json.loads(json_path.read_text()) - except json.JSONDecodeError as e: - print(f" warn: bad JSON at {json_path}: {e}") - continue - if not is_tensor_sidecar(meta): - continue - rel = json_path.relative_to(variant_dir) - parts = rel.parts - # Layout: //ref/.json - if len(parts) < 4 or parts[-2] != "ref": - print(f" warn: unexpected layout at {json_path}") - continue - case = parts[0] - stage_dir = parts[1] - entries.append({ - "case": case, - "stage_dir": stage_dir, - "stage": meta.get("stage", ""), - "name": meta["name"], - "shape": list(meta["shape"]), - "dtype": meta["dtype"], - "rel_path": str(rel), - }) - return entries - - -def write_coverage(variant: str, entries: list[dict]) -> Path: - out = BUILD / variant / "dump_coverage.json" - payload = { - "family": FAMILY, - "variant": variant, - "tensors": entries, - } - out.write_text(json.dumps(payload, indent=2) + "\n") - return out - - -def aggregate_tolerances(per_variant: dict[str, list[dict]]) -> dict[str, dict]: - stats: dict[str, list[tuple[str, float, float]]] = {} - for variant, entries in per_variant.items(): - for e in entries: - json_path = BUILD / variant / e["rel_path"] - meta = json.loads(json_path.read_text()) - p99 = float(meta.get("p99_abs", 0.0)) - rms = float(meta.get("rms", 0.0)) - stats.setdefault(e["name"], []).append((variant, p99, rms)) - - out: dict[str, dict] = {} - for name, rows in sorted(stats.items()): - worst_p99 = max(r[1] for r in rows) - worst_rms = max(r[2] for r in rows) - max_abs = max(1e-4 * worst_p99, 1e-6) - mean_abs = max(1e-5 * worst_rms, 1e-6) - out[name] = { - "max_abs": max_abs, - "mean_abs": mean_abs, - "_provisional": True, - "_seen_in": sorted({r[0] for r in rows}), - } - return out - - -def main() -> int: - if not BUILD.exists(): - print(f"error: {BUILD} does not exist; run the dumper first") - return 2 - - per_variant: dict[str, list[dict]] = {} - print("=== dump_coverage.json per variant ===") - for v in VARIANTS: - entries = walk_variant(v) - if not entries: - print(f" {v}: 0 tensors (skipping coverage write)") - continue - cov_path = write_coverage(v, entries) - per_variant[v] = entries - print(f" {v}: {len(entries)} tensors -> {cov_path.relative_to(ROOT)}") - - print() - print("=== tolerances aggregation ===") - tols = aggregate_tolerances(per_variant) - payload: dict = { - "_comment": [ - "MedASR per-tensor tolerances for compare_tensors.py.", - "", - "CORRECTNESS REGIME", - "- Reference: google/medasr @ ae1e4845b4b07479735d93e1e591e566435b7104", - " loaded via transformers v5.0.0.dev0 (commit 65dc261512cbdb1ee72b88ae5b222f2605aad8e5,", - " AutoModelForCTC + AutoProcessor, attn_implementation=eager, dtype=float32).", - " Dumper: scripts/dump_reference_medasr_transformers.py.", - "- C++: ggml CPU fp32 compute, weights stored as F32 in GGUF.", - "- KV cache dtype: not applicable (CTC head, no autoregressive decoder).", - "- Mel frontend: LasrFeatureExtractor — manual unfold + rfft, Hann symmetric,", - " no preemph, no dither, no CMVN; kaldi-mel weight matrix with HTK-style", - " DC-bin exclusion (125-7500 Hz, 128 bands); log(clamp(power, min=1e-5)).", - "", - "ENTRY SOURCING (provisional)", - "- Per-tensor max_abs = max(1e-4 × p99_abs, 1e-6)", - "- Per-tensor mean_abs = max(1e-5 × rms, 1e-6)", - "- All entries carry _provisional: true. Stage 4 finalizes against", - " observed C++ drift and removes the flag tensor-by-tensor.", - "", - "NOTE ON MID-BLOCK SUB-STEP MAGNITUDES", - "- enc.block.0.post_{ff1,attn,conv,ff2} carry huge activations (up to ~2e6)", - " because the macaron residual scalars (feed_forward_residual_weights=[1.5, 0.5],", - " conv_residual_weights=[2.0, 1.0]) amplify the running residual within a block;", - " the per-block out_norm collapses them back to ~O(1). The provisional max_abs", - " budget reflects the worst-magnitude raw value; do not be alarmed that", - " individual post-step budgets are ~1e2 in absolute units — they are scaled", - " to ~1e-4 × p99_abs as designed.", - "", - "DO NOT SHIP a model while _provisional entries remain.", - ], - **tols, - } - TOL.parent.mkdir(parents=True, exist_ok=True) - TOL.write_text(json.dumps(payload, indent=2) + "\n") - - tol_max_vals = sorted(v["max_abs"] for v in tols.values()) - n = len(tol_max_vals) - print(f" wrote {TOL.relative_to(ROOT)} with {n} tensor entries") - if n: - print( - f" max_abs distribution: " - f"min={tol_max_vals[0]:.3e} " - f"median={tol_max_vals[n // 2]:.3e} " - f"max={tol_max_vals[-1]:.3e}" - ) - return 0 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/catalog/check.py b/scripts/catalog/check.py new file mode 100755 index 00000000..a0facf24 --- /dev/null +++ b/scripts/catalog/check.py @@ -0,0 +1,301 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = ["jsonschema"] +# /// +"""Validate the durable catalog JSON records. + +Checks the JSON schema, cross-row integrity the schema cannot express (a +benchmark row referencing a quant the variant does not publish), that every +record is paired with the card spec and doc the schema says it owns. + + uv run scripts/catalog/check.py + uv run scripts/catalog/check.py --publication-profile + uv run scripts/catalog/check.py --publication-profile --models whisper-tiny + uv run scripts/catalog/check.py --dir catalog +""" +from __future__ import annotations + +import argparse +import collections +import json +import pathlib +import sys + +from jsonschema import Draft202012Validator + +REPO = pathlib.Path(__file__).resolve().parents[2] +sys.path.insert(0, str(REPO / "scripts" / "catalog")) +import common # noqa: E402 +import profiles # noqa: E402 + + +def schema_pass(records: dict, schema: dict) -> int: + v, bad = Draft202012Validator(schema), 0 + for name, rec in records.items(): + errs = sorted(v.iter_errors(rec), key=lambda e: list(e.path)) + if errs: + bad += 1 + print(f" FAIL {name}: {len(errs)} error(s)") + for e in errs[:4]: + print(f" {'/'.join(map(str, e.path)) or ''}: {e.message[:110]}") + print(f"schema {len(records) - bad}/{len(records)} valid") + return bad + + +def integrity_pass(records: dict) -> int: + bad = 0 + machines: dict[str, set[str]] = collections.defaultdict(set) + for name, rec in records.items(): + published = {d["quant"] for d in rec.get("downloads", [])} + for sect in ("accuracy_benchmarks", "speed_benchmarks"): + missing = {r["quant"] for r in rec.get(sect, []) if r["quant"] not in published} + if missing: + bad += 1 + print(f" FAIL {name}: {sect} references unpublished quant(s) {sorted(missing)}") + accuracy_counts = collections.Counter( + profiles.cell_key(row, "accuracy") + for row in rec.get("accuracy_benchmarks", [])) + duplicate_accuracy = sum(count - 1 for count in accuracy_counts.values() + if count > 1) + if duplicate_accuracy: + bad += 1 + print(f" FAIL {name}: {duplicate_accuracy} duplicate published " + "accuracy cell(s); batch size is recipe metadata, not a " + "separate result") + # A shipped model always says how fast it runs somewhere. The + # publication profile decides which cells are required; this is the + # weaker floor underneath it, so a record can never render a page or + # a card with no performance at all. + if not rec.get("speed_benchmarks"): + bad += 1 + print(f" FAIL {name}: no speed_benchmarks; every shipped model " + f"carries at least one measured cell") + for r in rec.get("speed_benchmarks", []): + if r.get("machine"): + machines[r["machine"]].add(name) + print(f"integrity {len(records) - bad}/{len(records)} clean; " + f"{len(machines)} machine slug(s): {', '.join(sorted(machines))}") + return bad + + +def pairing_pass(records: dict, selected: bool = False) -> int: + """Catalog records and card specs pair exactly; docs may be shared. + + Every record names its docs page (its own, or the family page whose + roll-up lists it), and that file must exist. The editorial card specs under scripts/hf_cards/ + pair one to one with records: generate.py reads both, so an orphan spec + has no catalog to render from and a record with no spec has no card. + """ + card_names = {path.stem for path in (REPO / "scripts" / "hf_cards").glob("*.yaml")} + record_names = set(records) + missing_cards = sorted(record_names - card_names) + # Orphan specs are a whole-catalog question; a --models run only asks + # whether the selected records have their card. + missing_records = [] if selected else sorted(card_names - record_names) + if selected: + card_names &= record_names + undocumented = sorted(name for name in record_names if not records[name].get("docs_page")) + bad_pages = sorted( + f"{name}: docs/models/{records[name]['docs_page']} does not exist" + for name in record_names + if records[name].get("docs_page") + and not (REPO / "docs" / "models" / records[name]["docs_page"]).exists()) + for line in bad_pages: + print(f" FAIL {line}") + for name in missing_cards: + print(f" FAIL {name}: no scripts/hf_cards/{name}.yaml") + for name in missing_records: + print(f" FAIL scripts/hf_cards/{name}.yaml: no catalog/{name}.json") + paired = len(record_names & card_names) + print(f"pairing {paired}/{len(record_names | card_names)} catalog/card pairs; " + f"{len(records) - len(undocumented)}/{len(records)} name a docs page") + if undocumented: + print(f" no docs_page: {', '.join(undocumented)}") + return len(missing_cards) + len(missing_records) + len(bad_pages) + + +def publication_pass(records: dict, profile_id: str | None, enforce: bool) -> int: + """Check publication matrices, including explicit legacy accuracy rows.""" + try: + resolved_id, profile = profiles.load_profile(profile_id) + except (OSError, ValueError, json.JSONDecodeError) as exc: + print(f"publication FAIL: {exc}") + return 1 + + problems = 0 + totals = collections.Counter() + for name, record in records.items(): + # An ASR publication profile does not apply to standalone diarizers. + if not record.get("capabilities", {}).get("transcribe", {}).get("supported"): + continue + accuracy_raw = profiles.expected_accuracy(record, profile) + speed_raw = profiles.expected_speed(record, profile) + accuracy = profiles.apply_exceptions(record, "accuracy", accuracy_raw) + speed = profiles.apply_exceptions(record, "speed", speed_raw) + exceptions = record.get("benchmark_exceptions") or [] + + # An exception must exclude a real profile requirement. This catches + # stale or misspelled waivers instead of retaining them forever. + stale_exceptions = 0 + for exception in exceptions: + kind = exception.get("kind") + candidates = accuracy_raw if kind == "accuracy" else speed_raw + if kind not in ("accuracy", "speed") or not any( + profiles.exception_matches(exception, kind, cell) + for cell in candidates): + stale_exceptions += 1 + per_model = collections.Counter(stale_exception=stale_exceptions) + + # Accuracy is closed by dataset/language/quant/metric/timestamps. Any + # batch size satisfies a cell (the row records which); explicitly + # marked legacy results retain the published recipe (or null when it + # did not survive). + expected_by_key = {profiles.profile_key(cell): cell for cell in accuracy} + expected_keys = set(expected_by_key) + expected_by_core = { + profiles.accuracy_core_key(cell): profiles.profile_key(cell) + for cell in accuracy + } + accuracy_rows = record.get("accuracy_benchmarks", []) + accuracy_counts = collections.Counter( + profiles.cell_key(row, "accuracy") for row in accuracy_rows) + accuracy_covered, accuracy_extra, accuracy_invalid = set(), set(), set() + for row in accuracy_rows: + key = profiles.profile_key(row) + legacy = row.get("measurement_provenance") == "legacy-published" + if key in expected_keys: + target_key = key + elif legacy: + # Grandfather a published pre-profile result under its honest + # batch/timestamp recipe; do not relabel it as the new recipe. + target_key = expected_by_core.get(profiles.accuracy_core_key(row)) + else: + target_key = None + if target_key is None: + if legacy: + # Preserve pre-profile rows that were already published, + # even when their quant was not selected by today's + # publication matrix. They are archive data, not drift. + totals["accuracy_archived"] += 1 + else: + accuracy_extra.add(key) + continue + accuracy_covered.add(target_key) + target = expected_by_key[target_key] + if (not profiles.has_measurement_provenance(row) + or (not legacy and ( + row.get("backend") != target.get("backend") + or row.get("language_hint") != target.get("runtime_language")))): + accuracy_invalid.add(target_key) + per_model["accuracy_missing"] = len(expected_keys - accuracy_covered) + per_model["accuracy_invalid"] = len(accuracy_invalid) + per_model["accuracy_extra"] = len(accuracy_extra) + per_model["accuracy_duplicate"] = sum( + count - 1 for count in accuracy_counts.values() if count > 1) + totals["accuracy_required"] += len(expected_keys) + for suffix in ("missing", "invalid", "extra", "duplicate"): + totals[f"accuracy_{suffix}"] += per_model[f"accuracy_{suffix}"] + + # Both published samples are required for each quant selected by the + # profile on every machine/backend target. + speed_expected = {profiles.cell_key(cell, "speed") for cell in speed} + speed_rows = record.get("speed_benchmarks", []) + speed_counts = collections.Counter( + profiles.cell_key(row, "speed") for row in speed_rows) + speed_actual = set(speed_counts) + invalid_speed = { + profiles.cell_key(row, "speed") for row in speed_rows + if profiles.cell_key(row, "speed") in speed_expected + and (row.get("xrt_compute") is None + or not profiles.has_measurement_provenance(row)) + } + per_model["speed_missing"] = len(speed_expected - speed_actual) + per_model["speed_invalid"] = len(invalid_speed) + per_model["speed_extra"] = len(speed_actual - speed_expected) + per_model["speed_duplicate"] = sum( + count - 1 for count in speed_counts.values() if count > 1) + totals["speed_required"] += len(speed_expected) + for suffix in ("missing", "invalid", "extra", "duplicate"): + totals[f"speed_{suffix}"] += per_model[f"speed_{suffix}"] + + count = sum(per_model.values()) + if count: + problems += count + details = ", ".join(f"{key}={value}" for key, value in per_model.items() if value) + print(f" {'FAIL' if enforce else 'TODO'} {name}: {details}") + + print(f"publication {resolved_id}: accuracy {totals['accuracy_required']} required, " + f"{totals['accuracy_missing']} missing, {totals['accuracy_invalid']} invalid, " + f"{totals['accuracy_extra']} extra, {totals['accuracy_archived']} archived legacy; speed " + f"{totals['speed_required']} required, {totals['speed_missing']} missing, " + f"{totals['speed_invalid']} invalid, {totals['speed_extra']} extra") + if problems and not enforce: + print(" audit only; pass --publication-profile to enforce this gate") + return problems if enforce else 0 + + +def provenance_pass(records: dict) -> int: + """Every number names its run or explicitly declares its legacy origin.""" + bad, legacy_speed, legacy_acc = 0, 0, 0 + total_speed = total_acc = 0 + for name, record in records.items(): + for section, kind in (("accuracy_benchmarks", "accuracy"), + ("speed_benchmarks", "speed")): + for row in record.get(section, []): + if kind == "accuracy": + total_acc += 1 + else: + total_speed += 1 + if row.get("measurement_provenance") == "legacy-published": + if kind == "accuracy": + legacy_acc += 1 + else: + legacy_speed += 1 + continue + if not row.get("engine_sha"): + bad += 1 + print(f" FAIL {name}: {kind} row has neither engine_sha nor " + f"measurement_provenance=legacy-published") + print(f"provenance {total_speed - legacy_speed}/{total_speed} speed and " + f"{total_acc - legacy_acc}/{total_acc} accuracy row(s) name a build; " + f"{legacy_speed + legacy_acc} explicitly marked legacy-published") + if legacy_speed: + print(f" {legacy_speed} legacy speed row(s) retain published xRT only; " + f"re-benchmark for stage timings, long-form, and memory") + return bad + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--dir", default=str(REPO / "catalog")) + ap.add_argument("--publication-profile", nargs="?", const="", + help="enforce exact accuracy and speed matrices; optionally " + "name a profile (default: " + "catalog/_benchmark_profiles.json default)") + ap.add_argument("--models", default="", + help="comma-separated variants (default: all)") + args = ap.parse_args() + d = pathlib.Path(args.dir) + schema = json.loads((REPO / "catalog/_schema.json").read_text()) + records = common.load_records(d) + selected = {item.strip() for item in args.models.split(",") if item.strip()} + unknown = selected - records.keys() + if unknown: + print(f"unknown catalog variant(s): {', '.join(sorted(unknown))}", file=sys.stderr) + return 2 + if selected: + records = {name: record for name, record in records.items() if name in selected} + if not records: + print(f"no records in {d}", file=sys.stderr) + return 2 + enforce_publication = args.publication_profile is not None + selected_profile = args.publication_profile or None + bad = (schema_pass(records, schema) + integrity_pass(records) + + pairing_pass(records, bool(selected)) + provenance_pass(records) + + publication_pass(records, selected_profile, enforce_publication)) + return 1 if bad else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/catalog/common.py b/scripts/catalog/common.py new file mode 100644 index 00000000..f1bba3ad --- /dev/null +++ b/scripts/catalog/common.py @@ -0,0 +1,387 @@ +"""Shared helpers for reading catalog records and rendering them. + +Pure stdlib, so every consumer -- check.py, db.py, render.py and +scripts/hf_cards/generate.py -- can import it without a dependency block. + +The catalog stores identity and exact numbers. Everything about how a number +LOOKS (units, decimal places, column padding, a dataset's display name) is a +rendering concern and lives here or in the marker that calls the renderer. +""" +from __future__ import annotations + +import json +import pathlib + +REPO = pathlib.Path(__file__).resolve().parents[2] +CATALOG_DIR = REPO / "catalog" +DOCS_DIR = REPO / "docs" / "models" +CARDS_DIR = REPO / "scripts" / "hf_cards" + +HEADLINE_KEYS = ("dataset", "split", "language", "metric", "batch_size", "timestamps") +# The recipe half of the pointer may be null, meaning "any"; the identity +# half never is. db.py's headline view encodes the same predicate in SQL. +HEADLINE_WILDCARD_KEYS = ("batch_size", "timestamps") + + +# -------------------------------------------------------------------------- +# loading + + +def load_records(directory: pathlib.Path | None = None) -> dict[str, dict]: + """Every catalog record, keyed by variant. `_`-prefixed files are tooling.""" + directory = directory or CATALOG_DIR + return {path.stem: json.loads(path.read_text()) + for path in sorted(directory.glob("*.json")) + if not path.name.startswith("_")} + + +def load_record(variant: str, directory: pathlib.Path | None = None) -> dict: + path = (directory or CATALOG_DIR) / f"{variant}.json" + if not path.exists(): + raise FileNotFoundError(f"no catalog record for {variant!r} at {path}") + return json.loads(path.read_text()) + + +# -------------------------------------------------------------------------- +# sizes + + +def fmt_size(size_bytes: int) -> str: + """Render a byte count the way a download table prints it: decimal MB + below a gigabyte, decimal GB to two places above.""" + if size_bytes < 10**9: + return f"{size_bytes / 10**6:.0f} MB" + return f"{size_bytes / 10**9:.2f} GB" + + +# -------------------------------------------------------------------------- +# dataset identity +# +# A benchmark row names its dataset with three fields (dataset, split, +# language), but every consumer wants a single string: a `:` +# spec for the WER harness, a slug for a report filename, a key for the +# database. They differ in punctuation, not in meaning, so the rule that +# picks the value lives here once. scripts/wer/remote/dataset_specs.py owns +# the other direction (spec string -> manifest and volume paths) and +# `dataset_spec` below emits exactly what its parse_dataset_spec accepts. + +# FLEURS publishes one split across many languages, so its language is what +# identifies a result; every other dataset varies by split instead. The WER +# harness agrees: `fleurs:zh` names a language and carries the split as a +# separate --split flag, while `librispeech:test-clean` names a split. +LANGUAGE_KEYED_DATASETS = ("fleurs", "eka-medical-asr") +# The split a language-keyed dataset is published at, which its spec leaves +# implicit. +PUBLISHED_SPLIT = {"fleurs": "test", "eka-medical-asr": "test"} +# Datasets that publish a single language, so a slug need not name one. +SINGLE_LANGUAGE_DATASETS = ("librispeech",) + + +def dataset_tail(row: dict) -> str: + """The value half of a dataset spec: which FLEURS language, which + LibriSpeech split.""" + if row["dataset"] in LANGUAGE_KEYED_DATASETS: + return str(row["language"]) + return str(row["split"]) + + +def dataset_slug(row: dict) -> str: + """`fleurs-es`, `librispeech-test-clean`. The dataset half of a WER report + filename, and the stem the catalog looks for when ingesting a score.""" + return f"{row['dataset']}-{dataset_tail(row)}" + + +def dataset_spec(row: dict) -> str: + """`fleurs:es`, `librispeech:test-clean`. What run.py and the Modal sweep + take as `--dataset`.""" + return f"{row['dataset']}:{dataset_tail(row)}" + + +# -------------------------------------------------------------------------- +# accuracy + +DATASET_LABELS = { + ("librispeech", "test-clean"): "LibriSpeech test-clean", + ("ami", "ihm-test"): "AMI IHM test", +} + + +def dataset_label(dataset: str, split: str, language: str) -> str: + if dataset == "fleurs": + return f"FLEURS {language}" + return DATASET_LABELS.get((dataset, split), f"{dataset} {split}") + + +def headline(record: dict) -> dict | None: + """The benchmark row-set a variant publishes in its download table. + + A variant can carry several runs of the same dataset that differ only in + batch size or timestamp mode, so the pointer names the full identity + tuple rather than just the dataset. + """ + return record.get("headline_benchmark") + + +def headline_rows(record: dict) -> dict[str, dict]: + """{quant: accuracy row} for the headline benchmark. Empty if unset.""" + target = headline(record) + if not target: + return {} + rows = {} + for row in record.get("accuracy_benchmarks", []): + # A null batch_size or timestamps is an intentional wildcard for a + # legacy table assembled before recipe metadata was standardized. + if all((key in HEADLINE_WILDCARD_KEYS and target[key] is None) + or row.get(key) == target[key] + for key in HEADLINE_KEYS): + rows[row["quant"]] = row + return rows + + +def headline_label(record: dict) -> str: + target = headline(record) + if not target: + return "" + return dataset_label(target["dataset"], target["split"], target["language"]) + + +def headline_recipe(record: dict) -> str: + """The mechanical half of a WER note, from the headline rows themselves: + dataset, size, batch, timestamps, backend, and which build measured it.""" + rows = list(headline_rows(record).values()) + target = headline(record) + if not rows or not target: + return "" + measured = [row for row in rows if row.get("engine_sha")] or rows + sample = measured[0] + n_utts = max(row["n_utts"] for row in rows) + unit = "meetings" if target["metric"] in ("der", "cpwer") else "utterances" + parts = [f"{target['metric'].upper()} on the full {headline_label(record)} split " + f"({n_utts:,} {unit})"] + batch_sizes = sorted({row["batch_size"] for row in rows + if row.get("batch_size") is not None}) + if len(batch_sizes) == 1: + parts.append(f"batch size {batch_sizes[0]}") + elif batch_sizes: + parts.append("batch sizes " + " and ".join(str(size) for size in batch_sizes)) + if sample.get("timestamps"): + parts.append(f"timestamps {sample['timestamps']}") + if sample.get("language_hint"): + parts.append(f"language hint `{sample['language_hint']}`") + if sample.get("backend"): + parts.append(f"decoded on {sample['backend']}") + text = ", ".join(parts) + "." + shas = sorted({(row["engine_sha"], row.get("measured_on") or "") + for row in rows if row.get("engine_sha")}) + if shas: + text += " Measured at " + "; ".join( + f"transcribe.cpp `{sha}`" + (f" on {date}" if date else "") for sha, date in shas) + "." + if any(not row.get("engine_sha") for row in rows): + text += " Figures without a commit were published before provenance was recorded." + return text + + +def fmt_err(row: dict | None, dp: int = 2) -> str: + """An error rate as a card prints it. `-` when the cell was not measured.""" + if row is None: + return "-" + return f"{row['err_pct']:.{dp}f}%" + + +# -------------------------------------------------------------------------- +# speed + + +def fmt_ms(total_ms: float, dp_ms: int = 0, dp_s: int = 2) -> str: + if total_ms < 1000: + return f"{total_ms:.{dp_ms}f} ms" + return f"{total_ms / 1000:.{dp_s}f} s" + + +XRT_DP = 2 + + +def fmt_xrt(row: dict) -> str: + """Speedup over realtime, two decimals. A legacy row carries only the + precision its doc published (an integer or one decimal), so it is printed + as stored rather than padded to a fidelity it never had.""" + xrt = row["xrt_compute"] + if row.get("engine_sha"): + return f"{xrt:.{XRT_DP}f}×" + text = repr(float(xrt)) + decimals = len(text.split(".")[1].rstrip("0")) + return f"{xrt:.{min(decimals, XRT_DP)}f}×" + + +def perf_rows(record: dict, machine: str) -> dict[tuple[str, str, str], dict]: + """{(backend, sample, quant): row} for one machine.""" + return {(row["backend"], row["sample"], row["quant"]): row + for row in record.get("speed_benchmarks", []) + if row["machine"] == machine} + + +# -------------------------------------------------------------------------- +# summaries + + +def fmt_params(params: int) -> str: + if params >= 10**9: + return f"{params / 10**9:.1f}B".replace(".0B", "B") + return f"{round(params / 10**6):.0f}M" + + +def languages_summary(record: dict) -> str: + """`en`, `en, de, fr`, or a count, plus a note when the model auto-detects.""" + langs = [str(lang) for lang in record.get("languages", [])] + text = ", ".join(langs) if len(langs) <= 4 else f"{len(langs)} languages" + if record.get("capabilities", {}).get("lang_detect", {}).get("supported"): + text += " + auto-detect" + return text or "-" + + +def capabilities_summary(record: dict) -> str: + """The extras beyond plain transcription, as a short comma list.""" + caps = record.get("capabilities", {}) + out = [] + for name, label in (("translate", "translate"), ("streaming", "streaming"), + ("diarize", "diarize")): + if caps.get(name, {}).get("supported"): + out.append(label) + grans = caps.get("timestamps", {}).get("granularities") or [] + if grans: + out.append(f"{grans[0]} timestamps") + return ", ".join(out) or "-" + + +# -------------------------------------------------------------------------- +# downloads + + +def download_url(record: dict, filename: str) -> str: + repo = record.get("published_repo") + if not repo: + return "" + return f"https://huggingface.co/{repo}/resolve/main/{filename}" + + +# -------------------------------------------------------------------------- +# markdown tables + + +MAX_PAD = 14 + + +def render_table(header: list[str], aligns: list[str], rows: list[list[str]], + rule_fill: bool = False, max_pad: int = MAX_PAD, + pad_header: bool = True) -> list[str]: + """A GitHub markdown table, columns padded so the source reads as a grid. + + `aligns` is "l" or "r" per column. `rule_fill` draws the separator out to + the column width (`| ------- |`) instead of the short form (`| --- |`); + both are used in docs/models and neither renders differently. + + Columns wider than `max_pad` are left ragged: a download table's link + column runs past 120 characters, and padding it buys nothing while making + every other cell unreadable in the source. + """ + source = [header] + rows if pad_header else rows + widths = [max(len(row[i]) for row in source) for i in range(len(header))] + widths = [0 if width > max_pad else width for width in widths] + + def line(cells: list[str], pad: bool = True) -> str: + return "| " + " | ".join( + (cell.rjust(width) if align == "r" else cell.ljust(width)) if pad else cell + for cell, width, align in zip(cells, widths, aligns)).rstrip() + " |" + + head = line(header, pad_header) + if rule_fill: + rules = ["-" * max(width - 1, 2) + ":" if align == "r" else "-" * max(width, 3) + for width, align in zip(widths, aligns)] + return [head, "| " + " | ".join(rules) + " |"] + [line(row) for row in rows] + rules = ["---:" if align == "r" else "---" for align in aligns] + return [head, "| " + " | ".join(rules) + " |"] + [line(row) for row in rows] + + +# -------------------------------------------------------------------------- +# writing records + +WRAP_WIDTH = 79 + + +def _compact(value) -> str: + return json.dumps(value, separators=(",", ":"), ensure_ascii=False) + + +def _fill(items: list[str], pad: str, inner: str) -> str: + """A scalar array too long for one line, filled to WRAP_WIDTH.""" + lines, current = [], "" + for index, item in enumerate(items): + piece = item + ("," if index < len(items) - 1 else "") + candidate = (current + " " + piece) if current else inner + piece + if current and len(candidate) > WRAP_WIDTH: + lines.append(current) + current = inner + piece + else: + current = candidate + if current: + lines.append(current) + return "[\n" + "\n".join(lines) + "\n" + pad + "]" + + +def _holds_records(value) -> bool: + """True when `value` contains a list of objects somewhere inside. + + That is the shape worth expanding: a streaming block's `presets` is a list + of rows a reader scans, while an accuracy row's `errors` is three counts + that belong on the row's own line. Both sit at the same depth, so depth + alone cannot tell them apart. + """ + if isinstance(value, list): + return any(isinstance(item, dict) for item in value) or \ + any(_holds_records(item) for item in value) + if isinstance(value, dict): + return any(_holds_records(item) for item in value.values()) + return False + + +def _fmt(value, depth: int = 0, indent: int = 2) -> str: + pad, inner = " " * (indent * depth), " " * (indent * (depth + 1)) + if isinstance(value, list): + if not any(isinstance(item, (dict, list)) for item in value): + one = _compact(value) + if len(pad) + len(one) <= WRAP_WIDTH or not value: + return one + return _fill([json.dumps(v, ensure_ascii=False) for v in value], pad, inner) + if depth >= 2 and not _holds_records(value): + return _compact(value) + if not value: + return "[]" + body = ",\n".join(inner + _fmt(item, depth + 1, indent) for item in value) + return "[\n" + body + "\n" + pad + "]" + if isinstance(value, dict): + if depth >= 2 and not _holds_records(value): + return _compact(value) + if not value: + return "{}" + body = ",\n".join(f"{inner}{json.dumps(key, ensure_ascii=False)}: " + f"{_fmt(val, depth + 1, indent)}" + for key, val in value.items()) + return "{\n" + body + "\n" + pad + "}" + return _compact(value) + + +def dumps_record(record: dict) -> str: + """Serialize a record the way every checked-in record is written. + + Top level and depth-1 containers expand one entry per line; anything + deeper, and any array of scalars, stays compact -- so a benchmark row is + one greppable line and a 99-language list wraps instead of running 99 + lines. Plain `json.dumps(indent=2)` writes the same data as a file five to + eight times longer, which turns a one-value correction into an + unreviewable diff. Every writer here goes through this. + """ + return _fmt(record) + "\n" + + +def write_record(path: pathlib.Path, record: dict) -> None: + path.write_text(dumps_record(record)) diff --git a/scripts/catalog/db.py b/scripts/catalog/db.py new file mode 100755 index 00000000..fbd39628 --- /dev/null +++ b/scripts/catalog/db.py @@ -0,0 +1,299 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# /// +"""Build the portable catalog database from catalog JSON records. + +The SQLite file is a disposable query artifact; catalog/*.json is the source +of truth. Tables mirror the record sections one to one (downloads, accuracy, +speed) so a query reads like the JSON it came from. + + uv run scripts/catalog/db.py # build/catalog.db + uv run scripts/catalog/db.py --out path/to/catalog.db +""" +from __future__ import annotations + +import argparse +import json +import os +import pathlib +import sqlite3 +import sys +from datetime import datetime, timezone + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 +import profiles # noqa: E402 + +DEFAULT_DB = common.REPO / "build" / "catalog.db" + +SCHEMA = """ +PRAGMA user_version = 1; +CREATE TABLE meta(key TEXT PRIMARY KEY, value TEXT NOT NULL); + +CREATE TABLE models( + variant TEXT PRIMARY KEY, + family TEXT NOT NULL, + display_name TEXT NOT NULL, + params INTEGER NOT NULL, + license_spdx TEXT NOT NULL, + license_display TEXT NOT NULL, + upstream_repo TEXT NOT NULL, + upstream_commit TEXT NOT NULL, + published_repo TEXT, + language_tag_form TEXT, + encoder_window_s REAL, + long_form_strategy TEXT NOT NULL, + max_audio_s REAL, + max_output_tokens INTEGER, + -- Which accuracy row-set this model publishes as its headline number. + -- Recipe fields may be null to span quant rows measured with different + -- batch sizes while each quant still has one published result. + headline_dataset TEXT, + headline_metric TEXT, + headline_batch_size INTEGER, + headline_timestamps TEXT +); + +CREATE TABLE languages( + lang TEXT PRIMARY KEY +); +CREATE TABLE model_languages( + variant TEXT NOT NULL REFERENCES models(variant), + lang TEXT NOT NULL REFERENCES languages(lang), + PRIMARY KEY(variant, lang) +); +CREATE TABLE language_aliases( + variant TEXT NOT NULL REFERENCES models(variant), + alias TEXT NOT NULL REFERENCES languages(lang), + canonical TEXT NOT NULL REFERENCES languages(lang), + PRIMARY KEY(variant, alias) +); + +CREATE TABLE capabilities( + variant TEXT NOT NULL REFERENCES models(variant), + capability TEXT NOT NULL, + supported INTEGER NOT NULL, + verified INTEGER, + note TEXT, + details_json TEXT NOT NULL, + PRIMARY KEY(variant, capability) +); + +CREATE TABLE downloads( + variant TEXT NOT NULL REFERENCES models(variant), + quant TEXT NOT NULL, + filename TEXT NOT NULL, + size_bytes INTEGER NOT NULL, + PRIMARY KEY(variant, quant) +); + +CREATE TABLE datasets( + dataset_id TEXT PRIMARY KEY, + dataset TEXT NOT NULL, + split TEXT NOT NULL, + language TEXT NOT NULL REFERENCES languages(lang) +); +CREATE TABLE accuracy( + dataset_id TEXT NOT NULL REFERENCES datasets(dataset_id), + variant TEXT NOT NULL REFERENCES models(variant), + quant TEXT NOT NULL, + metric TEXT NOT NULL, + language_hint TEXT, + backend TEXT, + err_pct REAL NOT NULL CHECK(err_pct >= 0), + ci_lo REAL, + ci_hi REAL, + n_utts INTEGER NOT NULL CHECK(n_utts > 0), + batch_size INTEGER, + timestamps TEXT, + engine_sha TEXT, + measurement_provenance TEXT, + measured_on TEXT, + substitutions INTEGER, + deletions INTEGER, + insertions INTEGER, + empty_hyp INTEGER, + utts_over_50pct INTEGER, + publication_profile TEXT, + scoring TEXT, + mode TEXT +); +CREATE UNIQUE INDEX accuracy_identity ON accuracy( + dataset_id, variant, quant, metric, + IFNULL(timestamps, ''), IFNULL(scoring, ''), IFNULL(mode, '') +); + +CREATE TABLE machines( + machine TEXT PRIMARY KEY +); +CREATE TABLE speed( + variant TEXT NOT NULL REFERENCES models(variant), + machine TEXT NOT NULL REFERENCES machines(machine), + backend TEXT NOT NULL, + quant TEXT NOT NULL, + sample TEXT NOT NULL, + sample_duration_s REAL NOT NULL, + total_ms REAL, + xrt_compute REAL NOT NULL, + wall_ms REAL, + xrt_wall REAL, + load_ms REAL, + mel_ms REAL, + encode_ms REAL, + decode_ms REAL, + engine_sha TEXT, + measurement_provenance TEXT, + measured_on TEXT, + thermal_gated INTEGER, + publication_profile TEXT, + PRIMARY KEY(variant, machine, backend, quant, sample) +); + +-- The per-quant column a model card and its doc print. +CREATE VIEW headline AS +SELECT a.variant, d.dataset, d.split, d.language, + a.quant, a.metric, a.err_pct, a.ci_lo, a.ci_hi, a.n_utts +FROM accuracy a +JOIN models m ON m.variant = a.variant +JOIN datasets d ON d.dataset_id = a.dataset_id +WHERE a.dataset_id = m.headline_dataset + AND a.metric = m.headline_metric + AND (m.headline_batch_size IS NULL OR a.batch_size = m.headline_batch_size) + AND (m.headline_timestamps IS NULL OR a.timestamps = m.headline_timestamps); +""" + + +def dataset_id(row: dict) -> str: + """Primary key of the `datasets` table. + + The report slug drops whichever dimension its dataset holds constant, so + it is not always a full identity: `fleurs-es` names its language and + `librispeech-test-clean` has only one, but a dataset that varies by both + keeps the language segment so two languages cannot collide on one key. + """ + dataset, split, language = row["dataset"], row["split"], row["language"] + if (dataset in common.LANGUAGE_KEYED_DATASETS + and split == common.PUBLISHED_SPLIT.get(dataset)): + return common.dataset_slug(row) # fleurs-es + if dataset in common.SINGLE_LANGUAGE_DATASETS: + return common.dataset_slug(row) # librispeech-test-clean + return f"{dataset}-{split}-{language}" # ami-ihm-test-en + + +def build(records: dict[str, dict], out: pathlib.Path) -> dict[str, int]: + if not records: + raise RuntimeError("no catalog records") + + langs = {str(lang) for record in records.values() for lang in record.get("languages", [])} + langs.update(row["language"] for record in records.values() + for row in record.get("accuracy_benchmarks", [])) + for record in records.values(): + for alias, canonical in (record.get("language_aliases") or {}).items(): + langs.update((alias, canonical)) + machines = {row["machine"] for record in records.values() + for row in record.get("speed_benchmarks", [])} + + out.parent.mkdir(parents=True, exist_ok=True) + tmp = out.with_suffix(out.suffix + ".tmp") + tmp.unlink(missing_ok=True) + con = sqlite3.connect(tmp) + try: + con.execute("PRAGMA foreign_keys = ON") + con.executescript(SCHEMA) + con.executemany("INSERT INTO languages VALUES (?)", [(lang,) for lang in sorted(langs)]) + con.executemany("INSERT INTO machines VALUES (?)", [(m,) for m in sorted(machines)]) + + datasets: dict[str, tuple[str, str, str]] = {} + for record in records.values(): + for row in record.get("accuracy_benchmarks", []): + datasets[dataset_id(row)] = (row["dataset"], row["split"], row["language"]) + con.executemany("INSERT INTO datasets VALUES (?,?,?,?)", [ + (key, *value) for key, value in sorted(datasets.items())]) + + for variant, record in records.items(): + license_info = record["license"] + headline = record.get("headline_benchmark") or {} + con.execute("INSERT INTO models VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", ( + variant, record["family"], record["display_name"], record["params"], + license_info["spdx"], license_info["display"], record["upstream_repo"], + record["upstream_commit"], record.get("published_repo"), + record.get("language_tag_form"), record.get("encoder_window_s"), + record["long_form_strategy"], record.get("max_audio_s"), + record.get("max_output_tokens"), + dataset_id(headline) if headline else None, headline.get("metric"), + headline.get("batch_size"), headline.get("timestamps"))) + con.executemany("INSERT INTO model_languages VALUES (?,?)", [ + (variant, str(lang)) for lang in record.get("languages", [])]) + con.executemany("INSERT INTO language_aliases VALUES (?,?,?)", [ + (variant, alias, canonical) + for alias, canonical in (record.get("language_aliases") or {}).items()]) + con.executemany("INSERT INTO capabilities VALUES (?,?,?,?,?,?)", [ + (variant, name, int(bool(cap.get("supported"))), + None if cap.get("verified") is None else int(cap["verified"]), + cap.get("note"), json.dumps(cap, separators=(",", ":"), sort_keys=True)) + for name, cap in record.get("capabilities", {}).items()]) + con.executemany("INSERT INTO downloads VALUES (?,?,?,?)", [ + (variant, item["quant"], item["filename"], item["size_bytes"]) + for item in record.get("downloads", [])]) + con.executemany( + "INSERT INTO accuracy VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", [ + (dataset_id(row), variant, row["quant"], row["metric"], + row.get("language_hint"), row.get("backend"), row["err_pct"], + (row.get("ci95") or [None, None])[0], + (row.get("ci95") or [None, None])[1], row["n_utts"], + row.get("batch_size"), row.get("timestamps"), row.get("engine_sha"), + row.get("measurement_provenance"), row.get("measured_on"), + (row.get("errors") or {}).get("sub"), + (row.get("errors") or {}).get("del"), + (row.get("errors") or {}).get("ins"), row.get("empty_hyp"), + row.get("utts_over_50pct"), row.get("publication_profile"), + row.get("scoring"), row.get("mode")) + for row in record.get("accuracy_benchmarks", [])]) + con.executemany( + "INSERT INTO speed VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", [ + (variant, row["machine"], row["backend"], row["quant"], row["sample"], + row["sample_duration_s"], row.get("total_ms"), row["xrt_compute"], + row.get("wall_ms"), row.get("xrt_wall"), row.get("load_ms"), row.get("mel_ms"), row.get("encode_ms"), + row.get("decode_ms"), row.get("engine_sha"), + row.get("measurement_provenance"), row.get("measured_on"), + None if row.get("thermal_gated") is None else int(row["thermal_gated"]), + row.get("publication_profile")) + for row in record.get("speed_benchmarks", [])]) + + profile_id, _ = profiles.load_profile() + con.executemany("INSERT INTO meta VALUES (?,?)", [ + ("generated", datetime.now(timezone.utc).isoformat(timespec="seconds")), + ("source", "catalog/*.json"), + ("benchmark_profile", profile_id), + ("rebuild", "uv run scripts/catalog/db.py (drops and recreates; never hand-edit)"), + ]) + con.commit() + counts = {table: con.execute(f"SELECT count(*) FROM {table}").fetchone()[0] + for table in ("models", "languages", "model_languages", + "language_aliases", "capabilities", "downloads", "datasets", + "accuracy", "machines", "speed")} + finally: + con.close() + os.replace(tmp, out) + return counts + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--dir", default=str(common.CATALOG_DIR)) + parser.add_argument("--out", default=str(DEFAULT_DB)) + args = parser.parse_args() + try: + counts = build(common.load_records(pathlib.Path(args.dir)), pathlib.Path(args.out)) + except (OSError, ValueError, KeyError, sqlite3.Error, RuntimeError) as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + for table, count in counts.items(): + print(f" {table:20s} {count:>6}") + print(f"\n{args.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/catalog/format.py b/scripts/catalog/format.py new file mode 100755 index 00000000..90304de6 --- /dev/null +++ b/scripts/catalog/format.py @@ -0,0 +1,49 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# /// +"""Rewrite catalog records in the canonical layout, or check that they are. + +The layout rules live in common.dumps_record, which every catalog writer +already uses; this is the same serializer applied to files edited by hand. + + uv run scripts/catalog/format.py # rewrite every record + uv run scripts/catalog/format.py --check # exit 1 if any record differs +""" +from __future__ import annotations + +import argparse +import json +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--check", action="store_true") + parser.add_argument("paths", nargs="*", type=pathlib.Path, + help="records to format (default: every catalog record)") + args = parser.parse_args() + paths = args.paths or sorted( + path for path in common.CATALOG_DIR.glob("*.json") if not path.name.startswith("_")) + stale = [] + for path in paths: + current = path.read_text() + rendered = common.dumps_record(json.loads(current)) + if current == rendered: + continue + stale.append(path) + if not args.check: + path.write_text(rendered) + verb = "need formatting" if args.check else "rewritten" + print(f"{len(paths)} record(s) checked; {len(stale)} {verb}") + for path in stale: + print(f" {path.relative_to(common.REPO)}") + return 1 if (args.check and stale) else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/catalog/ingest_accuracy.py b/scripts/catalog/ingest_accuracy.py new file mode 100755 index 00000000..85a85691 --- /dev/null +++ b/scripts/catalog/ingest_accuracy.py @@ -0,0 +1,166 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# /// +"""Ingest profile-stamped WER scores into accuracy_benchmarks. + +Only exact cells selected by the publication profile are eligible. The Modal +publication sweep writes the hypotheses; score them locally first, then run: + + for f in reports/wer/*.jsonl; do uv run scripts/wer/score.py "$f"; done + uv run scripts/catalog/ingest_accuracy.py --dry-run + uv run scripts/catalog/ingest_accuracy.py +""" +from __future__ import annotations + +import argparse +import json +import pathlib +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 +import profiles # noqa: E402 + +REPORTS = common.REPO / "reports" / "wer" + + +def score_path(record: dict, cell: dict, reports: pathlib.Path) -> pathlib.Path: + filename = next(item["filename"] for item in record["downloads"] + if item["quant"] == cell["quant"]) + model = pathlib.Path(filename).stem + dataset = common.dataset_slug(cell) + batch = "" if cell["batch_size"] <= 1 else f".b{cell['batch_size']}" + timestamps = "" if cell["timestamps"] == "none" else f".ts-{cell['timestamps']}" + return reports / f"{model}.{dataset}{batch}{timestamps}.score.json" + + +def row_from_score(cell: dict, score: dict, profile_id: str) -> dict: + per_utterance = score.get("per_utterance") or [] + metric = cell["metric"] + return { + "dataset": cell["dataset"], + "split": cell["split"], + "language": cell["language"], + "language_hint": cell["runtime_language"], + "backend": cell["backend"], + "quant": cell["quant"], + "metric": metric, + "err_pct": score["error_rate_pct"], + "ci95": [round(score["error_rate_ci_lo"] * 100, 2), + round(score["error_rate_ci_hi"] * 100, 2)], + "n_utts": score["n"], + "batch_size": score.get("batch_size") if score.get("batch_size") is not None else cell["batch_size"], + "timestamps": cell["timestamps"], + "engine_sha": score["engine_sha"], + "publication_profile": profile_id, + "measured_on": None, + "errors": { + "sub": score["substitutions"], + "del": score["deletions"], + "ins": score["insertions"], + }, + "empty_hyp": sum(1 for row in per_utterance + if not str(row.get("hyp") or "").strip()), + "utts_over_50pct": sum(1 for row in per_utterance + if float(row.get(metric, 0.0)) > 0.5), + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--reports", default=str(REPORTS)) + parser.add_argument("--profile", default=None) + parser.add_argument("--models", default="", + help="comma-separated variants (default: all)") + parser.add_argument("--dry-run", action="store_true") + args = parser.parse_args() + + profile_id, profile = profiles.load_profile(args.profile) + reports = pathlib.Path(args.reports) + selected = {item.strip() for item in args.models.split(",") if item.strip()} + records = common.load_records() + unknown = selected - records.keys() + if unknown: + print(f"error: no catalog record for {', '.join(sorted(unknown))}", file=sys.stderr) + return 2 + + added = replaced = rejected = missing = unstamped = 0 + for variant, record in records.items(): + if selected and variant not in selected: + continue + path = common.CATALOG_DIR / f"{variant}.json" + rows = record.get("accuracy_benchmarks", []) + changed = False + expected = profiles.apply_exceptions( + record, "accuracy", profiles.expected_accuracy(record, profile)) + for cell in expected: + source_path = score_path(record, cell, reports) + if not source_path.exists(): + missing += 1 + continue + score = json.loads(source_path.read_text()) + recipe = score.get("recipe") or {} + covered = any(profiles.profile_key(row) == profiles.profile_key(cell) + or (row.get("measurement_provenance") == "legacy-published" + and profiles.accuracy_core_key(row) == profiles.accuracy_core_key(cell)) + for row in rows) + if not recipe.get("publication_profile") and covered: + # A score from before profile stamping, for a cell the catalog + # already publishes: superseded history, not a problem. + unstamped += 1 + continue + reasons = [] + if recipe.get("publication_profile") != profile_id: + reasons.append(f"profile={recipe.get('publication_profile')!r}") + if score.get("metric") != cell["metric"]: + reasons.append(f"metric={score.get('metric')!r}") + if score.get("timestamps") != cell["timestamps"]: + reasons.append(f"timestamps={score.get('timestamps')!r}") + if recipe.get("backend") != cell["backend"]: + reasons.append(f"backend={recipe.get('backend')!r}") + if not score.get("engine_sha"): + reasons.append("engine_sha is empty") + if reasons: + rejected += 1 + print(f" reject {source_path.name}: {', '.join(reasons)}") + continue + + # Any batch size satisfies the cell; the newest measurement + # replaces whatever the cell held and records its own batch size. + key = profiles.profile_key(cell) + indices = [index for index, row in enumerate(rows) + if profiles.profile_key(row) == key] + if not indices: + # A fresh exact run supersedes the matching historical table + # row even if that row used an older/unknown recipe. + core = profiles.accuracy_core_key(cell) + indices = [index for index, row in enumerate(rows) + if row.get("measurement_provenance") == "legacy-published" + and profiles.accuracy_core_key(row) == core] + new_row = row_from_score(cell, score, profile_id) + if indices: + first = indices[0] + if rows[first] == new_row and len(indices) == 1: + continue + rows[first] = new_row + for index in reversed(indices[1:]): + del rows[index] + replaced += 1 + else: + rows.append(new_row) + added += 1 + changed = True + if changed and not args.dry_run: + common.write_record(path, record) + + print(f"profile {profile_id}: {added} added, {replaced} replaced, " + f"{rejected} rejected, {unstamped} unstamped score(s) for already published " + f"cells skipped, {missing} score file(s) absent") + if args.dry_run: + print("dry run: nothing written") + return 1 if rejected else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/catalog/ingest_perf.py b/scripts/catalog/ingest_perf.py new file mode 100755 index 00000000..1c4c6237 --- /dev/null +++ b/scripts/catalog/ingest_perf.py @@ -0,0 +1,327 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# /// +"""Fold bench driver reports into the catalog's speed_benchmarks rows. + +`scripts/bench/run.py` writes one report per (variant, backend) under +reports/perf//, and reports/ is gitignored -- so the latency +breakdown only exists on the machine that measured it. This is the hop that +moves it into the catalog, where it is durable and publishable. + +A cell is identified by (machine, backend, quant, sample). Many reports cover +the same cell, because porting-6-bench runs a hypothesis loop over it, and +those iterations are NOT interchangeable with the published figure -- CPU +cells in particular swing tens of percent with thermal state. Only a +profile-stamped run (`scripts/bench/run.py --profile`) is eligible, and among +those the newest wins. + + uv run scripts/catalog/ingest_perf.py --dry-run + uv run scripts/catalog/ingest_perf.py + uv run scripts/catalog/ingest_perf.py --reports reports/perf +""" +from __future__ import annotations + +import argparse +import collections +import functools +import json +import pathlib +import re +import sys + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 +import profiles # noqa: E402 + +REPORTS = common.REPO / "reports" / "perf" + +# models//-.gguf -- the quant is the last dash-separated +# field, and K-quants carry underscores (Q4_K_M) so the split is on "-". +QUANT_RE = re.compile(r"-([A-Za-z0-9_]+)\.gguf$") + + +def quant_of(model_path: str) -> str | None: + match = QUANT_RE.search(model_path) + return match.group(1) if match else None + + +@functools.lru_cache(maxsize=1) +def variant_by_directory() -> dict[str, str]: + """Catalog variant by the lowercased models/ directory that holds it. + + A directory mirrors the upstream repo name (`Qwen3-ASR-0.6B`, + `SenseVoiceSmall`) while the record is kebab-case, so a report's path + names its variant only up to case, and sometimes not even that. An + upstream repo that several records share (the four gigaam variants ship + from one repo) is ambiguous, so it resolves to nothing rather than to an + arbitrary one of them; those variants have directories of their own. + """ + records = common.load_records() + lookup = {variant.lower(): variant for variant in records} + by_upstream = collections.defaultdict(list) + for variant, record in records.items(): + by_upstream[record["upstream_repo"].rsplit("/", 1)[-1].lower()].append(variant) + for slug, variants in by_upstream.items(): + if len(variants) == 1: + lookup.setdefault(slug, variants[0]) + return lookup + + +def variant_of(report: dict, model_path: str) -> str | None: + """Reports carry `variant` or the older `family`; the path is definitive.""" + name = report.get("variant") + parts = pathlib.PurePosixPath(model_path.replace("\\", "/")).parts + if "models" in parts: + index = len(parts) - 1 - parts[::-1].index("models") + if index + 1 < len(parts): + name = parts[index + 1] + if not name: + return None + return variant_by_directory().get(name.lower(), name) + + +def publishable(report: dict) -> bool: + """Only a profile-stamped run can become a catalog row: the stamp names + the recipe (iterations, warmup, samples, thermal policy) the number was + measured under. Hypothesis-loop and hand-flagged runs are experiments.""" + return bool(report.get("publication_profile")) + + +def cells(report: dict) -> list[dict]: + """One catalog-shaped row per run in a bench driver report.""" + out = [] + for run in report.get("runs", []): + model_path, quant = run.get("model_path", ""), quant_of(run.get("model_path", "")) + variant = variant_of(report, model_path) + summary, duration = run.get("summary") or {}, run.get("sample_duration_s") + total = (summary.get("total_ms") or {}).get("mean") + wall = (summary.get("wall_ms") or {}).get("mean") + if not (variant and quant and duration and total): + continue + + def mean(field: str): + value = (summary.get(field) or {}).get("mean") + return None if value is None else round(value, 1) + + out.append({ + "_profile": report.get("publication_profile"), + "variant": variant, + "machine": profiles.canonical_machine(report["machine"]["slug"]), + # The run's own `backend` is the runtime device name (MTL0), not + # the canonical backend; the driver records that at the top level. + "backend": (report.get("backend") or run.get("backend", "")).lower(), + "quant": quant, + "sample": pathlib.PurePosixPath(run.get("sample_path", "")).stem, + "sample_duration_s": duration, + "total_ms": round(total, 1), + # xrt is recomputed from the unrounded mean rather than carried + # over: a stored value that no longer matches its own latency is + # the drift this ingest exists to remove. + "xrt_compute": round(duration / (total / 1000), 2), + "wall_ms": None if wall is None else round(wall, 1), + "xrt_wall": None if wall is None else round(duration / (wall / 1000), 2), + "load_ms": None if run.get("load_ms") is None else round(run["load_ms"], 1), + "mel_ms": mean("mel_ms"), + "encode_ms": mean("encode_ms"), + "decode_ms": mean("decode_ms"), + "engine_sha": report.get("git_sha"), + "publication_profile": report.get("publication_profile"), + "measured_on": (report.get("timestamp") or "")[:10] or None, + "_when": report.get("timestamp") or "", + "_file": report["_file"], + }) + return out + + +def collect(reports_dir: pathlib.Path) -> tuple[dict, list[str]]: + """Newest profile-stamped measurement per cell, plus notes on what was + skipped.""" + best: dict[tuple, dict] = {} + superseded, unreadable, experiments = collections.Counter(), [], 0 + for path in sorted(reports_dir.glob("*/*.json")): + try: + report = json.loads(path.read_text()) + except (json.JSONDecodeError, UnicodeDecodeError): + unreadable.append(str(path.relative_to(common.REPO))) + continue + if not isinstance(report, dict) or report.get("schema") != "transcribe-bench-driver-v1": + kind = report.get("schema") if isinstance(report, dict) else type(report).__name__ + unreadable.append(f"{path.relative_to(common.REPO)} (schema {kind!r})") + continue + if not publishable(report): + experiments += 1 + continue + # Reports normally live under the repo, but --reports can name a + # staging directory elsewhere (or a relative one); the label is only + # for the operator, so fall back to the path as given. + try: + report["_file"] = str(path.resolve().relative_to(common.REPO)) + except ValueError: + report["_file"] = str(path) + for row in cells(report): + key = (row["variant"], row["machine"], row["backend"], row["quant"], row["sample"]) + previous = best.get(key) + if previous is None or row["_when"] > previous["_when"]: + if previous is not None: + superseded[key] += 1 + best[key] = row + else: + superseded[key] += 1 + notes = [f"{path}: unreadable or not a bench report" for path in unreadable] + if experiments: + notes.append(f"{experiments} report(s) without a profile stamp ignored") + if superseded: + notes.append(f"{sum(superseded.values())} older report(s) superseded on " + f"{len(superseded)} cell(s)") + return best, notes + + +FIELDS = ("sample_duration_s", "total_ms", "xrt_compute", "wall_ms", "xrt_wall", + "load_ms", "mel_ms", "encode_ms", "decode_ms", "engine_sha", + "publication_profile", "measured_on") + + +def catalog_row(source: dict) -> dict: + """Strip importer bookkeeping from one measured, catalog-shaped cell.""" + return {field: source[field] for field in ( + "machine", "backend", "quant", "sample", *FIELDS + )} | {"thermal_gated": None} + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--reports", default=str(REPORTS)) + parser.add_argument("--dry-run", action="store_true") + parser.add_argument("--drift", type=float, default=2.0, + help="percent xrt change worth reporting (default 2)") + parser.add_argument("--max-drift", type=float, default=5.0, + help="refuse a row whose xRT would move more than this " + "percent, since the stored value is what the doc " + "published (default 5). --force takes it anyway.") + parser.add_argument("--force", action="store_true", + help="apply measurements even when they contradict the " + "published xRT") + args = parser.parse_args() + + reports_dir = pathlib.Path(args.reports) + if not reports_dir.exists(): + print(f"no reports at {reports_dir}", file=sys.stderr) + return 2 + measured, notes = collect(reports_dir) + print(f"{len(measured)} measured cell(s) across " + f"{len({key[1] for key in measured})} machine slug(s)") + for note in notes: + print(f" note: {note}") + + profile_id, profile = profiles.load_profile() + filled = updated = added = matched = 0 + drift, refused, unmatched = [], [], [] + for variant, record in common.load_records().items(): + path = common.CATALOG_DIR / f"{variant}.json" + rows = record.get("speed_benchmarks", []) + changed = False + for row in rows: + key = (variant, row["machine"], row["backend"], row["quant"], row["sample"]) + source = measured.pop(key, None) + if source is None: + continue + matched += 1 + was_null = row.get("total_ms") is None + before = row.get("xrt_compute") + # A stored xRT may have been published as compute or as wall (the + # granite tables quote wall, where the two differ by up to 1.46x). + # Gate on whichever the doc evidently used, so a definition + # mismatch is not mistaken for a stale build. + candidates = [c for c in (source["xrt_compute"], source["xrt_wall"]) + if c is not None] + after = min(candidates, key=lambda c: abs(c - before)) if before else source["xrt_compute"] + moved = abs(after - before) / before * 100 if before else 0.0 + if before and moved > args.max_drift and not args.force: + # The stored xRT is the number the doc published. A report that + # disagrees this much is a different build, not a better + # reading of the same one -- refuse it and name the gap. + refused.append((variant, row["machine"], row["backend"], row["quant"], + row["sample"], before, after, row.get("engine_sha"), + source["engine_sha"])) + continue + for field in FIELDS: + if row.get(field) != source[field]: + row[field] = source[field] + changed = True + if source.get("engine_sha") and row.pop("measurement_provenance", None): + changed = True + if was_null: + filled += 1 + elif changed: + updated += 1 + if before and moved > args.drift: + drift.append((variant, row["machine"], row["backend"], row["quant"], + row["sample"], before, after)) + # Profile runs can create rows; the old importer could only refresh + # placeholders, which made a newly required quant impossible to ingest + # without first hand-authoring empty catalog cells. + expected = profiles.apply_exceptions( + record, "speed", profiles.expected_speed(record, profile)) + expected_keys = { + (variant, cell["machine"], cell["backend"], cell["quant"], cell["sample"]) + for cell in expected + } + existing_keys = { + (variant, row["machine"], row["backend"], row["quant"], row["sample"]) + for row in rows + } + for key in sorted(expected_keys - existing_keys): + source = measured.get(key) + if source is None or source.get("_profile") != profile_id: + continue + rows.append(catalog_row(source)) + measured.pop(key) + matched += 1 + added += 1 + changed = True + if changed and not args.dry_run: + common.write_record(path, record) + + for key in measured: + unmatched.append(key) + print(f"\nmatched {matched} catalog row(s): {added} added, " + f"{filled} had no latency, {updated} already did and were refreshed, " + f"{len(refused)} refused") + if refused: + print(f"\n{len(refused)} row(s) refused: the newest report on file is a " + f"different build from the one the doc published, so taking it " + f"would silently restate a published number.") + print(" re-run the publication sweep, or pass --force:") + by_variant = collections.Counter() + shas = collections.defaultdict(set) + for variant, _, _, _, _, _, _, _, report_sha in refused: + by_variant[variant] += 1 + shas[variant].add(report_sha or "?") + for variant, count in by_variant.most_common(16): + print(f" {count:3d} cell(s) {variant:42s} on file: " + f"{', '.join(sorted(shas[variant]))}") + if len(by_variant) > 16: + print(f" ... and {len(by_variant) - 16} more variant(s)") + if drift: + print(f"\n{len(drift)} cell(s) whose xRT moved more than {args.drift}%:") + for variant, machine, backend, quant, sample, before, after in sorted( + drift, key=lambda d: -abs(d[6] - d[5]) / d[5]): + pct = round(100 * (after - before) / before) + print(f" {variant:38s} {machine:12s} {backend:7s} {quant:7s} " + f"{sample:5s} {before:>8} -> {after:<8} {pct:+d}%") + if unmatched: + print(f"\n{len(unmatched)} measured cell(s) with no catalog row " + f"(a bench of something the catalog does not publish):") + counts = collections.Counter(key[0] for key in unmatched) + for variant, count in counts.most_common(12): + print(f" {variant:42s} {count}") + if len(counts) > 12: + print(f" ... and {len(counts) - 12} more variant(s)") + if args.dry_run: + print("\ndry run: nothing written") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/catalog/new_record.py b/scripts/catalog/new_record.py new file mode 100755 index 00000000..a8755359 --- /dev/null +++ b/scripts/catalog/new_record.py @@ -0,0 +1,125 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = ["gguf", "numpy"] +# /// +"""Seed catalog/.json for a newly ported model. + +Everything mechanical comes from artifacts that already exist by the end of +Stage 5: the intake (family, upstream repo and revision, languages), the +local GGUFs under models// (downloads, byte sizes, parameter count, +capability KVs, licence, display name). The few editorial facts the artifacts +cannot supply are flags. Benchmark rows are left empty for Stages 6 and 7. + + uv run scripts/catalog/new_record.py --long-form soft-window --docs-page .md +""" +from __future__ import annotations + +import argparse +import pathlib +import sys + +import numpy as np +from gguf import GGUFReader + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 +import sync_capabilities # noqa: E402 + +QUANT_ORDER = ("F32", "BF16", "F16", "Q8_0", "Q6_K", "Q5_K_M", "Q4_K_M") +LICENSE_DISPLAY = {"apache-2.0": "Apache-2.0", "mit": "MIT", "cc-by-4.0": "CC-BY-4.0", + "cc-by-nc-4.0": "CC-BY-NC-4.0", "cc-by-nc-sa-4.0": "CC-BY-NC-SA-4.0"} + + +def quant_of(filename: str) -> str: + return filename.rsplit("-", 1)[-1].removesuffix(".gguf") + + +def kv(reader: GGUFReader, key: str, default=None): + field = reader.fields.get(key) + return field.contents() if field is not None else default + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("variant") + ap.add_argument("--long-form", required=True, choices=("chunked-unbounded", "hard-cap", "soft-window"), + help="which bucket in docs/input-limits.md the family falls into") + ap.add_argument("--docs-page", required=True, help="page under docs/models/ that documents it") + ap.add_argument("--published-repo", default=None, help="default handy-computer/-gguf") + ap.add_argument("--display-name", default=None, help="default: the GGUF's general.name") + ap.add_argument("--license", default=None, help="SPDX id; default: the GGUF's general.license") + ap.add_argument("--license-display", default=None) + ap.add_argument("--language-tag-form", default="bare-bcp47", choices=("bare-bcp47", "locale", "mixed")) + ap.add_argument("--force", action="store_true", help="overwrite an existing record") + args = ap.parse_args() + + out = common.CATALOG_DIR / f"{args.variant}.json" + if out.exists() and not args.force: + print(f"{out.relative_to(common.REPO)} exists; pass --force to overwrite", file=sys.stderr) + return 2 + intakes = list((common.REPO / "reports" / "porting").glob(f"*/{args.variant}/intake.json")) + if len(intakes) != 1: + print(f"expected one intake for {args.variant}, found {len(intakes)}", file=sys.stderr) + return 2 + import json + intake = json.loads(intakes[0].read_text()) + model_dir = common.REPO / "models" / args.variant + files = sorted(model_dir.glob("*.gguf"), key=lambda p: (QUANT_ORDER.index(quant_of(p.name)) + if quant_of(p.name) in QUANT_ORDER else 99)) + if not files: + print(f"no GGUFs under {model_dir}", file=sys.stderr) + return 2 + if not (common.DOCS_DIR / args.docs_page).exists(): + print(f"docs/models/{args.docs_page} does not exist", file=sys.stderr) + return 2 + + reference = files[0] + reader = GGUFReader(str(reference)) + params = sum(int(np.prod(t.shape)) for t in reader.tensors) + spdx = args.license or str(kv(reader, "general.license", "")).lower() + if not spdx: + print("no licence in the GGUF; pass --license", file=sys.stderr) + return 2 + display = args.license_display or LICENSE_DISPLAY.get(spdx) + if not display: + print(f"no display form known for licence {spdx!r}; pass --license-display", file=sys.stderr) + return 2 + caps = intake.get("capabilities") or {} + languages = [str(lang) for lang in caps.get("languages", [])] + acceptance = next((b for b in intake.get("upstream_benchmarks", []) + if str(b.get("dataset", "")).lower().startswith("librispeech")), None) + + record = { + "schema": "transcribe-catalog-v1", + "variant": args.variant, + "family": intake["family"], + "display_name": args.display_name or str(kv(reader, "general.name", args.variant)), + "params": params, + "license": {"spdx": spdx, "display": display}, + "upstream_repo": intake["hf_repo"], + "upstream_commit": str(intake["hf_revision"])[:7], + "published_repo": args.published_repo or f"handy-computer/{args.variant}-gguf", + "docs_page": args.docs_page, + "languages": languages, + "language_tag_form": args.language_tag_form, + "long_form_strategy": args.long_form, + "capabilities": {}, + "downloads": [{"quant": quant_of(p.name), "filename": p.name, "size_bytes": p.stat().st_size} + for p in files], + "accuracy_benchmarks": [], + "headline_benchmark": ({"dataset": "librispeech", "split": "test-clean", "language": "en", + "metric": "wer", "batch_size": None, "timestamps": "none"} + if acceptance else None), + "speed_benchmarks": [], + } + # Capabilities come from the file, never from hand: same reader the sweep uses. + record["capabilities"] = sync_capabilities.build(record, sync_capabilities.read_kvs(reader)) + common.write_record(out, record) + print(f"wrote {out.relative_to(common.REPO)}: {len(files)} downloads, {params:,} params, " + f"{len(languages)} language(s), capabilities from {reference.name}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/catalog/profiles.py b/scripts/catalog/profiles.py new file mode 100644 index 00000000..6d926da0 --- /dev/null +++ b/scripts/catalog/profiles.py @@ -0,0 +1,248 @@ +"""Publication-profile loading and benchmark matrix expansion. + +The profile is policy: it defines the exact rows the catalog publishes. The +catalog arrays contain measurements only; experiments remain in reports/. +Keep this module stdlib-only so local checks, bench runs, and Modal dispatch +all consume the same expansion logic. +""" +from __future__ import annotations + +import json +import pathlib +import sys +from typing import Iterable + +REPO = pathlib.Path(__file__).resolve().parents[2] +PROFILE_PATH = REPO / "catalog" / "_benchmark_profiles.json" +WER_DIR = REPO / "scripts" / "wer" +if str(WER_DIR) not in sys.path: + sys.path.insert(0, str(WER_DIR)) + +from languages import ( # noqa: E402 + CER_LANGUAGES, + FLEURS_CANONICAL_BY_CONFIG, + FLEURS_LANGS, +) + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 + +MACHINE_ALIASES = { + "apple-m4": "m4", + "apple-m4-max": "m4-max", + "amd-ryzen-7-pro-4750u-with-radeon-graphics": "ryzen-4750u", + "amd-ryzen-7-4750u-pro": "ryzen-4750u", +} + + +def load_profiles(path: pathlib.Path = PROFILE_PATH) -> dict: + data = json.loads(path.read_text()) + if not isinstance(data.get("profiles"), dict) or not data["profiles"]: + raise ValueError(f"{path}: profiles must be a non-empty object") + default = data.get("default") + if default not in data["profiles"]: + raise ValueError(f"{path}: default profile {default!r} is not defined") + return data + + +def load_profile(profile_id: str | None = None) -> tuple[str, dict]: + data = load_profiles() + profile_id = profile_id or data["default"] + try: + return profile_id, data["profiles"][profile_id] + except KeyError as exc: + raise ValueError( + f"unknown benchmark profile {profile_id!r}; choose one of " + f"{sorted(data['profiles'])}" + ) from exc + + +def canonical_machine(slug: str) -> str: + return MACHINE_ALIASES.get(slug, slug) + + +def _aliases(record: dict) -> dict[str, str]: + return {str(k).lower(): str(v).lower() + for k, v in (record.get("language_aliases") or {}).items()} + + +def canonical_fleurs_language(tag: str, record: dict) -> str | None: + """Return the canonical FLEURS language represented by a model tag. + + The catalog stores model-advertised spellings, while accuracy rows store + dataset spellings. Locale tags normally collapse to their primary subtag; + zh-TW is deliberately excluded because FLEURS has only Simplified + Mandarin. Explicit language_aliases handle tl/fil, no/nb, and jw/jv. + """ + value = str(tag).lower() + value = _aliases(record).get(value, value) + if value == "zh-tw": + return None + candidate = value if value in FLEURS_LANGS else value.split("-", 1)[0] + candidate = _aliases(record).get(candidate, candidate) + config = FLEURS_LANGS.get(candidate) + return FLEURS_CANONICAL_BY_CONFIG.get(config) if config else None + + +def fleurs_languages(record: dict) -> list[str]: + out: list[str] = [] + for advertised in record.get("languages", []): + canonical = canonical_fleurs_language(str(advertised), record) + if canonical and canonical not in out: + out.append(canonical) + return out + + +def runtime_language(record: dict, canonical: str) -> str: + """Choose the model spelling used to run one canonical dataset language.""" + for advertised in record.get("languages", []): + if canonical_fleurs_language(str(advertised), record) == canonical: + return str(advertised) + return canonical + + +def download_quants(record: dict) -> list[str]: + return [str(item["quant"]) for item in record.get("downloads", [])] + + +def _quants(spec: str | list[str], record: dict) -> list[str]: + if spec == "all-downloads": + return download_quants(record) + if not isinstance(spec, list): + raise ValueError(f"invalid quant selector {spec!r}") + available = set(download_quants(record)) + return [str(quant) for quant in spec if str(quant) in available] + + +def expected_accuracy(record: dict, profile: dict) -> list[dict]: + """Expand a profile into publication accuracy cells for one model.""" + cells: list[dict] = [] + if not (record.get("capabilities", {}).get("transcribe", {}).get("supported")): + return cells + for suite in profile.get("accuracy", []): + selector = suite["languages"] + if selector == "english-if-supported": + languages = ["en"] if "en" in fleurs_languages(record) else [] + elif selector == "supported-intersect-fleurs": + languages = fleurs_languages(record) + else: + raise ValueError(f"unknown language selector {selector!r}") + for language in languages: + for quant in _quants(suite["quants"], record): + cells.append({ + "dataset": suite["dataset"], + "split": suite["split"], + "language": language, + "runtime_language": runtime_language(record, language), + "quant": quant, + "metric": "cer" if language in CER_LANGUAGES else "wer", + "batch_size": suite["batch_size"], + "sort_by_length": suite.get("sort_by_length", False), + "timestamps": suite["timestamps"], + "gpu": suite.get("gpu"), + "backend": suite.get("backend"), + }) + return cells + + +def speed_samples(record: dict, profile: dict) -> list[str]: + """Which clips a variant is benched on. + + The default pair is English. A variant that supports exactly one other + language is benched on that language instead, at the same two lengths: + English audio decodes out of distribution on a single-language fine-tune + and the figure would not be comparable. The rule replaces what used to be + one hand-written override per such variant. + """ + spec = profile["speed"] + samples = spec.get("samples", []) + rule = spec.get("single_language_samples") + languages = record.get("languages") or [] + if rule and len(languages) == 1 and languages[0] != "en": + return [part.format(lang=languages[0]) for part in rule["pattern"]] + return samples + + +def all_speed_samples(profile: dict, records: dict[str, dict]) -> list[str]: + """Every clip the profile can ask for, across all known variants. The bench + driver needs this union up front to build its candidate matrix.""" + samples = set(profile["speed"].get("samples", [])) + for record in records.values(): + samples.update(speed_samples(record, profile)) + return sorted(samples) + + +def expected_speed(record: dict, profile: dict) -> list[dict]: + """Expand the exact publication speed matrix for one model.""" + spec = profile["speed"] + samples = speed_samples(record, profile) + cells: list[dict] = [] + for target in spec.get("targets", []): + for backend in target.get("backends", []): + for quant in _quants(spec["quants"], record): + for sample in samples: + cells.append({ + "machine": target["machine"], + "backend": backend, + "quant": quant, + "sample": sample, + }) + return cells + + +def has_measurement_provenance(row: dict) -> bool: + return bool(row.get("engine_sha")) or row.get("measurement_provenance") == "legacy-published" + + +ACCURACY_CORE_KEY = ("dataset", "split", "language", "quant", "metric") +# What a profile cell requires. Batch size is deliberately absent: a cell is +# satisfied at any batch size and the row records the one that was run. The +# profile's batch_size is the recommendation for new runs, not an identity. +PROFILE_KEY = (*ACCURACY_CORE_KEY, "timestamps", "scoring", "mode") +# Published accuracy has one row per profile cell. Batch size is measured +# recipe metadata, not a second publishable identity for the same result. +ACCURACY_KEY = PROFILE_KEY +SPEED_KEY = ("machine", "backend", "quant", "sample") + + +def accuracy_core_key(cell: dict) -> tuple: + return tuple(cell.get(field) for field in ACCURACY_CORE_KEY) + + +def profile_key(cell: dict) -> tuple: + return tuple(cell.get(field) for field in PROFILE_KEY) + + +def cell_key(cell: dict, kind: str) -> tuple: + fields = ACCURACY_KEY if kind == "accuracy" else SPEED_KEY + return tuple(cell.get(field) for field in fields) + + +def exception_matches(exception: dict, kind: str, cell: dict) -> bool: + if exception.get("kind") != kind: + return False + match = exception.get("match") or {} + return all(value == "*" or cell.get(field) == value + for field, value in match.items()) + + +def apply_exceptions(record: dict, kind: str, cells: Iterable[dict]) -> list[dict]: + exceptions = record.get("benchmark_exceptions") or [] + return [cell for cell in cells + if not any(exception_matches(exc, kind, cell) for exc in exceptions)] + + +def target_for_machine(profile: dict, machine_slug: str) -> dict | None: + machine = canonical_machine(machine_slug) + return next((target for target in profile["speed"].get("targets", []) + if target["machine"] == machine), None) + + +def dataset_spec(cell: dict) -> str: + """The `--dataset` string for one profile cell. See common.dataset_spec.""" + return common.dataset_spec(cell) + + +def machine_display(profile: dict, machine_slug: str) -> str: + target = target_for_machine(profile, machine_slug) + return (target or {}).get("display") or machine_slug diff --git a/scripts/catalog/render.py b/scripts/catalog/render.py new file mode 100755 index 00000000..44d3947f --- /dev/null +++ b/scripts/catalog/render.py @@ -0,0 +1,493 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = ["pyyaml"] +# /// +"""Render catalog-derived regions into docs/models/*.md. + +The docs are hand-written pages with a few regions that restate what the +catalog or the HF card spec already owns: the download and perf tables, the +intro summary, and the WER methodology note. Rather than generate whole +files, this rewrites only the regions a doc explicitly delegates: + + + | Quantization | Download | Size | WER (LibriSpeech test-clean) | + ... + + +Blocks: `downloads`, `perf machine=`, `accuracy` (one table per +dataset split beyond the headline), `recipe` (the mechanical WER sentence +from the headline rows), `pin` (licence, upstream and validation pins), +`intro` (upstream link plus the card spec's `summary`), `prose +field=wer.notes` (any `|` text field of the spec, dotted path), `family variants=a,b,c` (a roll-up row per variant, for +family pages), and `family-index` (the root README's supported-models table, +one row per documentation page). Everything outside a marker pair is +untouched. The root README is rendered along with docs/models. The variant is the file stem +unless the marker overrides it with `variant=`, so family docs can pull a +table for a model they are not named after. + + uv run scripts/catalog/render.py # rewrite marked regions + uv run scripts/catalog/render.py --check # fail if any is stale +""" +from __future__ import annotations + +import argparse +import difflib +import pathlib +import re +import shlex +import sys + +import yaml + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 +import profiles # noqa: E402 + +OPEN = re.compile(r"^(\s*)\s*$") +CLOSE = re.compile(r"^\s*\s*$") +SHA = re.compile(r"^[0-9a-f]{7,40}$") + + +class RenderError(Exception): + """A marker names something the catalog cannot currently supply.""" + + +def parse_attrs(text: str) -> dict[str, str]: + attrs = {} + for token in shlex.split(text): + key, _, value = token.partition("=") + attrs[key] = value + return attrs + + +def as_bool(value: str | None, default: bool) -> bool: + if value is None: + return default + return value.lower() in ("1", "true", "yes") + + +# -------------------------------------------------------------------------- +# blocks + + +def block_downloads(record: dict, attrs: dict[str, str]) -> list[str]: + """The Download table: one row per published GGUF, plus the headline metric.""" + want_metric = as_bool(attrs.get("metric"), True) + + rows_by_quant = common.headline_rows(record) if want_metric else {} + target = common.headline(record) + if want_metric and not target: + raise RenderError("metric column requested but headline_benchmark is null") + + header = ["Quantization", "Download", "Size"] + aligns = ["l", "l", "r"] + if want_metric: + label = attrs.get("label") or common.headline_label(record) + metric = attrs.get("metric_name") or target["metric"].upper() + header.append(f"{metric} ({label})") + aligns.append("r") + + body = [] + for item in record.get("downloads", []): + url = common.download_url(record, item["filename"]) + if not url: + raise RenderError("published_repo is null, so downloads have no URL") + cells = [item["quant"], f"[{item['filename']}]({url})", + common.fmt_size(item["size_bytes"])] + if want_metric: + cells.append(common.fmt_err(rows_by_quant.get(item["quant"]))) + body.append(cells) + if not body: + raise RenderError("no downloads") + return common.render_table(header, aligns, body) + + +def block_perf(record: dict, attrs: dict[str, str]) -> list[str]: + """A per-machine latency grid: rows are (backend, sample), columns quants.""" + machine = attrs.get("machine") + if not machine: + raise RenderError("perf block needs machine=") + rows = common.perf_rows(record, machine) + if not rows: + raise RenderError(f"no speed_benchmarks rows for machine {machine!r}") + + def ordered(index: int, override: str | None, rank) -> list[str]: + if override: + return override.split(",") + return sorted({key[index] for key in rows}, key=rank) + + # GPU backends first, then CPU; samples shortest first; quants in the + # record's download order (reference dtype down to the smallest quant). + backend_rank = {"metal": 0, "cuda": 1, "vulkan": 2, "cpu": 9} + duration = {key[1]: row["sample_duration_s"] for key, row in rows.items()} + quant_rank = {item["quant"]: i for i, item in enumerate(record.get("downloads", []))} + backends = ordered(0, attrs.get("backends"), lambda b: (backend_rank.get(b, 5), b)) + samples = ordered(1, attrs.get("samples"), lambda s: (duration.get(s, 0), s)) + quants = ordered(2, attrs.get("quants"), lambda q: (quant_rank.get(q, 99), q)) + dp_ms = int(attrs.get("dp_ms", 0)) + + body, blocked = [], [] + for backend in backends: + for sample in samples: + present = [rows.get((backend, sample, q)) for q in quants] + if not any(present): + continue + duration = next(r["sample_duration_s"] for r in present if r) + cells = [backend.capitalize() if backend != "cpu" else "CPU", + f"{sample} ({duration:.1f}s)"] + for quant, row in zip(quants, present): + if row is None: + cells.append("-") + continue + if row.get("total_ms") is None: + blocked.append(f"{backend}/{sample}/{quant}") + cells.append("-") + continue + cells.append(f"{common.fmt_ms(row['total_ms'], dp_ms)} " + f"({common.fmt_xrt(row)})" + ("" if row.get("engine_sha") else "†")) + body.append(cells) + if blocked: + raise RenderError( + f"{len(blocked)} cell(s) on {machine} have no total_ms, so latency " + f"cannot be rendered: {', '.join(blocked[:4])}" + + (" ..." if len(blocked) > 4 else "")) + if not body: + raise RenderError(f"no rows matched on {machine}") + table = common.render_table(["Backend", "Sample"] + quants, + ["l", "l"] + ["r"] * len(quants), body, + rule_fill=True, max_pad=20) + return perf_methodology(rows) + [""] + table + [""] + perf_provenance(machine, rows) + + +def perf_methodology(rows: dict) -> list[str]: + """What a cell is. Iterations and warmup are claimed only for rows that + name the profile they were measured under.""" + line = "Compute latency (mel + encode + decode), speedup over realtime in parentheses" + ids = sorted({row["publication_profile"] for row in rows.values() + if row.get("publication_profile")}) + claims = [] + for profile_id in ids: + speed = profiles.load_profile(profile_id)[1]["speed"] + claims.append(f"profile `{profile_id}`: mean over {speed['iterations']} iterations " + f"after {speed['warmup']} warmup") + if claims: + line += "; " + "; ".join(claims) + return [line + "."] + + +def perf_provenance(machine: str, rows: dict) -> list[str]: + """Where the numbers came from: machine, engine commit, date.""" + _, profile = profiles.load_profile() + display = profiles.machine_display(profile, machine) + builds: dict[tuple, int] = {} + for row in rows.values(): + if row.get("engine_sha"): + key = (row["engine_sha"], row.get("measured_on") or "") + builds[key] = builds.get(key, 0) + 1 + legacy = sum(1 for row in rows.values() if not row.get("engine_sha")) + parts = [] + for (sha, date), _ in sorted(builds.items(), key=lambda kv: -kv[1]): + parts.append(f"transcribe.cpp `{sha}`" + (f" on {date}" if date else "")) + line = f"{display}: " + "; ".join(parts) + "." if parts else f"{display}." + if legacy: + line += " † published before provenance was recorded; not yet re-measured." + return [line] + + +_SPECS: dict[str, dict] = {} + + +def spec_for(record: dict) -> dict: + """The editorial HF card spec, the prose source of truth for a variant.""" + variant = record["variant"] + if variant not in _SPECS: + path = common.CARDS_DIR / f"{variant}.yaml" + if not path.exists(): + raise RenderError(f"no card spec at {path.relative_to(common.REPO)}") + _SPECS[variant] = yaml.safe_load(path.read_text()) or {} + return _SPECS[variant] + + +def prose_lines(text: object, what: str) -> list[str]: + if not isinstance(text, str) or not text.strip(): + raise RenderError(f"{what} is empty") + return text.strip().split("\n") + + +def block_intro(record: dict, attrs: dict[str, str]) -> list[str]: + """Upstream pointer from the catalog, then the card spec's summary.""" + repo = record["upstream_repo"] + line = (f"Upstream: [`{repo}`](https://huggingface.co/{repo}) at " + f"[`{record['upstream_commit']}`]" + f"(https://huggingface.co/{repo}/commit/{record['upstream_commit']}).") + return [line, ""] + prose_lines(spec_for(record).get("summary"), "summary") + + +def block_recipe(record: dict, attrs: dict[str, str]) -> list[str]: + """The mechanical WER sentence, from the headline rows.""" + text = common.headline_recipe(record) + if not text: + raise RenderError("no headline rows to describe") + return [text] + + +def block_pin(record: dict, attrs: dict[str, str]) -> list[str]: + """Licence, upstream pin and validation pin, from the record and the + card spec's release fields.""" + spec = spec_for(record) + repo, commit = record["upstream_repo"], record["upstream_commit"] + validation = spec.get("validation") or {} + if not (spec.get("pin_date") and validation.get("commit") and validation.get("date")): + raise RenderError("spec needs pin_date and validation.{commit,date}") + # This renders as a link into the transcribe.cpp tree, so it has to be a + # commit. Three whisper specs carried the reference framework's version + # here ("5.6.1") and published a dead link claiming to be a validation + # pin; the field's meaning changed and nothing re-read the old values. + if not SHA.match(str(validation["commit"])): + raise RenderError( + f"validation.commit {validation['commit']!r} is not a transcribe.cpp " + f"commit SHA. Re-validate and record the commit it ran at " + f"(reference-framework versions belong in validation.reference)") + # A licence with no SPDX id carries its own URL; link the display name to + # it rather than leaving the reader to find the terms. Same field the HF + # card emits as license_link. + licence = record["license"] + display = (f"[{licence['display']}]({licence['link']})" + if licence.get("link") else licence["display"]) + return [f"Licensed {display}. Ported from upstream commit " + f"[`{commit}`](https://huggingface.co/{repo}/commit/{commit}), pinned " + f"{spec['pin_date']}. Validated against the {validation.get('reference', 'reference')} " + f"reference at transcribe.cpp commit [`{validation['commit']}`]" + f"(https://github.com/handy-computer/transcribe.cpp/tree/{validation['commit']}) " + f"on {validation['date']}."] + + +def block_prose(record: dict, attrs: dict[str, str]) -> list[str]: + """A text field of the card spec, named by dotted path (`wer.notes`).""" + field = attrs.get("field") + if not field: + raise RenderError("prose block needs field=") + value: object = spec_for(record) + for part in field.split("."): + value = value.get(part) if isinstance(value, dict) else None + return prose_lines(value, f"spec field {field!r}") + + +def block_accuracy(record: dict, attrs: dict[str, str]) -> list[str]: + """One table per dataset split: language rows, quant columns. + + The headline cell (dataset, split, language) is already the download + table's last column, so it is left out unless `all=true`; the rest of its + split still renders. `datasets=fleurs:test,librispeech:test-clean` narrows + to named splits. + """ + headline = common.headline(record) or {} + wanted = None + if attrs.get("datasets"): + wanted = {tuple(item.split(":", 1)) for item in attrs["datasets"].split(",")} + groups: dict[tuple, list[dict]] = {} + for row in record.get("accuracy_benchmarks", []): + split_key = (row["dataset"], row["split"]) + if wanted is not None and split_key not in wanted: + continue + if wanted is None and not as_bool(attrs.get("all"), False) \ + and not row.get("scoring") and not row.get("mode") \ + and (*split_key, row["language"]) == (headline.get("dataset"), headline.get("split"), + headline.get("language")): + continue + # A scoring step or decoding mode makes a separate result set. + groups.setdefault((*split_key, row.get("scoring") or "", row.get("mode") or ""), + []).append(row) + if not groups: + raise RenderError("no accuracy rows beyond the headline benchmark") + + quant_rank = {item["quant"]: i for i, item in enumerate(record.get("downloads", []))} + out: list[str] = [] + for (dataset, split, scoring, mode), rows in sorted(groups.items()): + quants = sorted({row["quant"] for row in rows}, key=lambda q: (quant_rank.get(q, 99), q)) + cells: dict[tuple[str, str], dict] = {} + for row in rows: + # Several recipes of one cell can coexist (batch size, timestamps); + # the profile recipe wins, else the first listed. + key = (row["language"], row["quant"]) + if key not in cells or row.get("engine_sha") and not cells[key].get("engine_sha"): + cells[key] = row + body = [] + for language in sorted({row["language"] for row in rows}): + metric = next(cells[(language, q)]["metric"] for q in quants if (language, q) in cells) + body.append([language, metric.upper()] + + [common.fmt_err(cells.get((language, q))) for q in quants]) + if out: + out.append("") + label = f"FLEURS {split}" if dataset == "fleurs" else common.dataset_label(dataset, split, "") + if scoring: + label += f", scoring `{scoring}`" + if mode: + label += f", `{mode}` mode" + out.extend([f"**{label}**", ""]) + out.extend(common.render_table(["Language", "Metric"] + quants, + ["l", "l"] + ["r"] * len(quants), body)) + return out + + +def block_family(records: dict[str, dict], attrs: dict[str, str]) -> list[str]: + """A family roll-up: one row per variant, headline number at one quant.""" + names = [name for name in attrs.get("variants", "").split(",") if name] + if not names: + raise RenderError("family block needs variants=") + quant = attrs.get("quant", "Q8_0") + body = [] + for name in names: + record = records.get(name) + if record is None: + raise RenderError(f"no catalog record for {name!r}") + download = next((d for d in record.get("downloads", []) if d["quant"] == quant), None) + headline = common.headline(record) or {} + # A variant with a page of its own links there; one documented only + # on this family page links to its published repo. + own = f"{name}.md" + link = (f"[{own}]({own})" if (common.DOCS_DIR / own).exists() and own != attrs.get("_page") + else f"[{record['published_repo']}](https://huggingface.co/{record['published_repo']})") + body.append([ + f"`{name}`", common.fmt_params(record["params"]), common.languages_summary(record), + common.fmt_size(download["size_bytes"]) if download else "-", + (f"{common.headline_label(record)} ({headline['metric'].upper()})" + if headline else "-"), + common.fmt_err(common.headline_rows(record).get(quant)), + common.capabilities_summary(record), link]) + return common.render_table( + ["Variant", "Params", "Languages", f"{quant} size", "Benchmark", quant, + "Capabilities", "Doc"], + ["l", "r", "l", "r", "l", "r", "l", "l"], body, max_pad=34) + + +def doc_for(record: dict) -> pathlib.Path | None: + page = record.get("docs_page") + return common.DOCS_DIR / page if page else None + + +def block_family_index(records: dict[str, dict], attrs: dict[str, str]) -> list[str]: + """The root README's supported-models table: one row per documentation + page, listing the variants it covers. `transcribe=false` selects the + models that only diarize.""" + want = as_bool(attrs.get("transcribe"), True) + groups: dict[str, dict] = {} + for variant, record in records.items(): + if bool(record.get("capabilities", {}).get("transcribe", {}).get("supported")) != want: + continue + doc = doc_for(record) + if doc is not None: + key = doc.stem + title = doc.read_text().splitlines()[0].lstrip("# ").strip() + link = f"[docs/models/{doc.name}](docs/models/{doc.name})" + else: + key = variant + title = record["display_name"] + link = f"[{record['published_repo']}](https://huggingface.co/{record['published_repo']})" + group = groups.setdefault(key, {"title": title, "link": link, "variants": [], "caps": set()}) + group["variants"].append(variant) + group["caps"].update(c for c in common.capabilities_summary(record).split(", ") if c != "-") + if not groups: + raise RenderError("no models matched") + body = [] + for group in sorted(groups.values(), key=lambda g: g["title"].lower()): + body.append([group["title"], ", ".join(f"`{v}`" for v in sorted(group["variants"])), + ", ".join(sorted(group["caps"])) or "-", group["link"]]) + return common.render_table(["Family", "Variants", "Available capabilities", "Docs"], + ["l", "l", "l", "l"], body) + + +BLOCKS = {"downloads": block_downloads, "perf": block_perf, + "intro": block_intro, "prose": block_prose, "accuracy": block_accuracy, + "recipe": block_recipe, "pin": block_pin} + + +# -------------------------------------------------------------------------- +# file rewriting + + +def rewrite(path: pathlib.Path, records: dict[str, dict]) -> tuple[str, list[str]]: + lines = path.read_text().splitlines() + out, errors, index, fenced = [], [], 0, False + while index < len(lines): + if lines[index].lstrip().startswith("```"): + fenced = not fenced # a marker quoted in a code block is documentation + match = None if fenced else OPEN.match(lines[index]) + if not match: + out.append(lines[index]) + index += 1 + continue + indent, name, raw = match.groups() + close = next((j for j in range(index + 1, len(lines)) if CLOSE.match(lines[j])), None) + if close is None: + errors.append(f"{path.name}:{index + 1}: catalog:{name} has no ") + out.append(lines[index]) + index += 1 + continue + out.append(lines[index]) + attrs = parse_attrs(raw) + variant = attrs.get("variant", path.stem) + try: + if name == "family": + rendered = block_family(records, {**attrs, "_page": path.name}) + elif name == "family-index": + rendered = block_family_index(records, attrs) + elif name not in BLOCKS: + raise RenderError(f"unknown block type {name!r}") + elif variant not in records: + raise RenderError(f"no catalog record for {variant!r}") + else: + rendered = BLOCKS[name](records[variant], attrs) + except RenderError as exc: + errors.append(f"{path.name}:{index + 1}: catalog:{name} {variant}: {exc}") + out.extend(lines[index + 1:close]) # leave the region alone + else: + out.extend(indent + line for line in rendered) + out.append(lines[close]) + index = close + 1 + return "\n".join(out) + "\n", errors + + +# -------------------------------------------------------------------------- + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--check", action="store_true", + help="report stale regions and exit non-zero; write nothing") + parser.add_argument("--docs", default=str(common.DOCS_DIR)) + parser.add_argument("paths", nargs="*", help="limit to these files") + args = parser.parse_args() + + records = common.load_records() + docs = ([pathlib.Path(p) for p in args.paths] + or sorted(pathlib.Path(args.docs).glob("*.md")) + [common.REPO / "README.md"]) + + stale, errors, rendered = [], [], 0 + for path in docs: + current = path.read_text() + text, file_errors = rewrite(path, records) + errors.extend(file_errors) + if not any(OPEN.match(line) for line in current.splitlines()): + continue + rendered += 1 + if text == current: + continue + stale.append(path) + if args.check: + diff = difflib.unified_diff(current.splitlines(), text.splitlines(), + f"a/{path}", f"b/{path}", lineterm="", n=1) + print("\n".join(diff)) + else: + path.write_text(text) + + for error in errors: + print(f" error: {error}", file=sys.stderr) + verb = "stale" if args.check else "rewritten" + print(f"{rendered} doc(s) with markers; {len(stale)} {verb}; {len(errors)} error(s)") + return 1 if (args.check and stale) or errors else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/catalog/sync_capabilities.py b/scripts/catalog/sync_capabilities.py new file mode 100755 index 00000000..8417794a --- /dev/null +++ b/scripts/catalog/sync_capabilities.py @@ -0,0 +1,316 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = ["gguf", "huggingface-hub", "requests"] +# /// +"""Read a record's `capabilities` block back out of its GGUF's KVs. + +The GGUF is the truth and the catalog follows it. If the file is wrong, the +fix is a converter change plus a re-export, never an edit to the record: +absence is not falsity, since read_capability_bool() leaves a field alone +when its KV is missing and the family default then applies. The shared +writer factory (scripts/lib/gguf_common.py) fills any unset capability KV +with false, so every fresh export states all four. + +Hand-writing this block is how moss-transcribe-diarize shipped as +diarize:false and how whisper-large-v3 came to claim translate:false while its +own GGUF says otherwise. This reads the truth back out instead. + +Sources, and the reason each is what it is: + + translate / lang_detect / streaming / diarize + `stt.capability.*` KVs, which is exactly what read_capability_kv() in + src/transcribe-meta.cpp feeds to the public capability surface. + timestamps.granularities + NOT a KV. `max_timestamp_kind` is a family invariant applied in each + family's load(), so the ceiling is scraped out of + src/arch//capabilities.cpp and cannot drift from the C++. + Granite scopes it per variant off stt.capability.word_timestamps. + transcribe / batching + Left alone: batching sits behind the transcribe_model_supports() probe + rather than the capability struct, so reading it needs a loaded model. + +A GGUF is read locally when present, otherwise its header is range-fetched +from the published repo (a few MB, not the weights). + + uv run scripts/catalog/sync_capabilities.py --dry-run + uv run scripts/catalog/sync_capabilities.py --local-only + uv run scripts/catalog/sync_capabilities.py + uv run scripts/catalog/sync_capabilities.py --check # exit 1 on any disagreement + uv run scripts/catalog/sync_capabilities.py --check --models + # ship gate: every quant, every KV present, all agree +""" +from __future__ import annotations + +import argparse +import collections +import json +import pathlib +import re +import sys +import tempfile + +sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent)) +import common # noqa: E402 + +ARCH = common.REPO / "src" / "arch" + +# catalog `family` -> src/arch directory, where the two spellings diverge. +FAMILY_DIRS = { + "cohere_asr": "cohere", + "granite_speech": "granite", + "granite_speech_nar": "granite_nar", +} + +KIND_RE = re.compile(r"max_timestamp_kind\s*=\s*TRANSCRIBE_TIMESTAMPS_(\w+)") + +KV = { + "translate": "stt.capability.translate", + "lang_detect": "stt.capability.lang_detect", + "streaming": "stt.capability.streaming", + "diarize": "stt.capability.speaker_diarization", +} + + +def family_ceiling(family: str) -> str | None: + """The family's max_timestamp_kind, read straight out of the C++.""" + path = ARCH / FAMILY_DIRS.get(family, family) / "capabilities.cpp" + if not path.exists(): + return None + match = KIND_RE.search(path.read_text()) + return match.group(1).lower() if match else None + + +def granularities(ceiling: str | None) -> list[str]: + """Just the ceiling. max_timestamp_kind is the finest the family emits; + whether it also assembles coarser output is a run() question, so listing + the coarser kinds here would be a claim the C++ does not make.""" + return [] if not ceiling or ceiling == "none" else [ceiling] + + +def read_kvs(reader) -> dict: + """Every stt.capability.* / stt.translation.* KV the file carries.""" + out = {} + for key, field in reader.fields.items(): + if not (key.startswith("stt.capability.") or key.startswith("stt.translation.")): + continue + try: + parts = field.parts + if field.types and field.types[0].name == "ARRAY": + out[key] = [bytes(parts[i]).decode("utf-8", "replace") + if parts[i].dtype.kind in "iu" and len(parts[i]) > 1 + else parts[i].tolist() for i in field.data] + out[key] = [v if isinstance(v, str) else + bytes(bytearray(v)).decode("utf-8", "replace") + for v in out[key]] + else: + value = parts[field.data[0]] + out[key] = bool(value[0]) if field.types[0].name == "BOOL" else value.tolist()[0] + except (IndexError, KeyError, AttributeError, UnicodeDecodeError): + continue + return out + + +def local_path(filename: str) -> pathlib.Path | None: + for directory in sorted((common.REPO / "models").glob("*")): + if directory.is_dir() and (directory / filename).exists(): + return directory / filename + return None + + +def default_filename(record: dict) -> str: + """The one file the sweep reads per record: a local one if any is on + disk, else the smallest published quant (cheapest header fetch).""" + for item in record["downloads"]: + if local_path(item["filename"]): + return item["filename"] + return sorted(record["downloads"], key=lambda d: d["size_bytes"])[0]["filename"] + + +def open_gguf(record: dict, filename: str, local_only: bool): + """A GGUFReader over the named file: local if present, else its + range-fetched header from the published repo.""" + from gguf import GGUFReader + + path = local_path(filename) + if path: + return GGUFReader(str(path)), "local", None + if local_only or not record.get("published_repo"): + return None, None, f"{filename} not on disk" + + # Header-only read: pad the temp file out to the declared size so + # GGUFReader's memmap of the tensor region stays in bounds and is never + # touched. Same trick as scripts/audit_gguf_metadata.py. + import requests + from huggingface_hub import get_hf_file_metadata, hf_hub_url + from huggingface_hub.utils import build_hf_headers + + try: + url = hf_hub_url(record["published_repo"], filename) + total = get_hf_file_metadata(url).size + prefix = min(24 * 1024 * 1024, total) + headers = build_hf_headers() + headers["Range"] = f"bytes=0-{prefix - 1}" + response = requests.get(url, headers=headers, timeout=120) + response.raise_for_status() + except Exception as exc: # noqa: BLE001 - any transport failure is just "unavailable" + return None, None, f"{filename}: {type(exc).__name__}: {str(exc)[:70]}" + with tempfile.NamedTemporaryFile(suffix=".gguf", delete=False) as handle: + tmp = pathlib.Path(handle.name) + handle.write(response.content) + handle.truncate(total) + try: + return GGUFReader(str(tmp)), "hub", None + finally: + tmp.unlink(missing_ok=True) + + +def build(record: dict, kvs: dict) -> dict: + """The capabilities block this GGUF implies, keeping human-set flags.""" + previous = record.get("capabilities", {}) + + def carry(name: str, supported: bool, extra: dict | None = None) -> dict: + block = {"supported": supported} + if supported: + was = previous.get(name, {}) + block["verified"] = bool(was.get("verified")) if was.get("supported") else False + block.update(extra or {}) + if was.get("note"): + block["note"] = was["note"] + return block + + caps = {"transcribe": previous.get("transcribe", {"supported": True, "verified": False})} + + targets = kvs.get("stt.translation.target_languages") + pairs = kvs.get("stt.translation.pairs") + translate = bool(kvs.get(KV["translate"], False)) + extra = {} + if translate: + extra = {"targets": targets or None, "pairs": pairs or None} + caps["translate"] = carry("translate", translate, extra) + caps["lang_detect"] = carry("lang_detect", bool(kvs.get(KV["lang_detect"], False))) + + ceiling = family_ceiling(record["family"]) + if record["family"].startswith("granite_speech") and not kvs.get( + "stt.capability.word_timestamps", False): + # Granite scopes the ceiling per variant, in arch/granite/model.cpp. + ceiling = "none" + grans = granularities(ceiling) + caps["timestamps"] = carry("timestamps", bool(grans), {"granularities": grans} if grans else None) + + caps["streaming"] = carry("streaming", bool(kvs.get(KV["streaming"], False))) + if caps["streaming"]["supported"]: + for key in ("mode", "presets"): + if key in previous.get("streaming", {}): + caps["streaming"][key] = previous["streaming"][key] + caps["diarize"] = carry("diarize", bool(kvs.get(KV["diarize"], False))) + if caps["diarize"]["supported"]: + for key in ("max_speakers", "granularity", "markup"): + if key in previous.get("diarize", {}): + caps["diarize"][key] = previous["diarize"][key] + caps["batching"] = previous.get("batching", {"supported": False}) + for optional in ("punctuation", "casing", "itn"): + if optional in previous: + caps[optional] = previous[optional] + return caps + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--dry-run", action="store_true") + parser.add_argument("--local-only", action="store_true", + help="skip the hub fallback") + parser.add_argument("--models", default="", + help="comma-separated variants; every published quant " + "of each is inspected (default: all variants, one " + "file each)") + parser.add_argument("--check", action="store_true", + help="write nothing; exit 1 if any record disagrees " + "with its GGUF. With --models, a missing " + "capability KV, an unreadable file, or quants that " + "disagree with each other also fail.") + args = parser.parse_args() + args.dry_run = args.dry_run or args.check + + records = common.load_records() + selected = {item.strip() for item in args.models.split(",") if item.strip()} + unknown = selected - records.keys() + if unknown: + print(f"unknown catalog variant(s): {', '.join(sorted(unknown))}", file=sys.stderr) + return 2 + strict = args.check and bool(selected) + + changed, unreachable, missing, disagree = [], [], [], [] + sources = collections.Counter() + for variant, record in records.items(): + if selected and variant not in selected: + continue + filenames = ([item["filename"] for item in record["downloads"]] if selected + else [default_filename(record)]) + caps_by_file = {} + for filename in filenames: + reader, source, error = open_gguf(record, filename, args.local_only) + if reader is None: + unreachable.append((variant, error)) + continue + sources[source] += 1 + kvs = read_kvs(reader) + absent = [key for key in KV.values() if key not in kvs] + if absent: + missing.append((variant, filename, absent)) + caps_by_file[filename] = build(record, kvs) + if not caps_by_file: + continue + distinct = {json.dumps(caps, sort_keys=True) for caps in caps_by_file.values()} + if len(distinct) > 1: + disagree.append((variant, sorted(caps_by_file))) + continue + caps = next(iter(caps_by_file.values())) + before = record.get("capabilities", {}) + if caps == before: + continue + diff = [f"{name}: {before.get(name, {}).get('supported')} -> {block['supported']}" + for name, block in caps.items() + if before.get(name, {}).get("supported") != block.get("supported")] + grain_before = (before.get("timestamps") or {}).get("granularities") + grain_after = (caps.get("timestamps") or {}).get("granularities") + if grain_before != grain_after and not any(d.startswith("timestamps") for d in diff): + diff.append(f"timestamps: {grain_before} -> {grain_after}") + changed.append((variant, diff)) + if not args.dry_run: + record["capabilities"] = caps + common.write_record(common.CATALOG_DIR / f"{variant}.json", record) + + print(f"read {sum(sources.values())} GGUF(s): " + + ", ".join(f"{count} {where}" for where, count in sources.most_common())) + print(f"{len(changed)} record(s) corrected\n") + for variant, diff in changed: + print(f" {variant:42s} {'; '.join(diff) or 'payload only'}") + if disagree: + print(f"\n{len(disagree)} record(s) whose quants disagree with each other " + f"(not updated):") + for variant, files in disagree: + print(f" {variant:42s} {', '.join(files)}") + if missing: + # Every converter now writes every capability KV; a file without one + # predates that and inherits the family default, which is exactly the + # silence that let wrong flags ship. Fatal at ship time, a warning in + # the sweep until the published files are re-exported. + print(f"\n{len(missing)} file(s) missing capability KV(s)" + + (":" if strict else " (warning; --check --models makes this fatal):")) + for variant, filename, absent in missing: + print(f" {variant}/{filename}: {', '.join(absent)}") + if unreachable: + print(f"\n{len(unreachable)} file(s) not readable:") + for variant, error in unreachable: + print(f" {variant:42s} {error}") + if args.dry_run: + print("\ndry run: nothing written") + failed = bool(changed or disagree) if args.check else False + if strict and (missing or unreachable): + failed = True + return 1 if failed else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/catalog/test_db_mapping.py b/scripts/catalog/test_db_mapping.py new file mode 100644 index 00000000..44a23f23 --- /dev/null +++ b/scripts/catalog/test_db_mapping.py @@ -0,0 +1,48 @@ +"""Every catalog row property reaches the database, or is excluded on purpose. + + uv run --with pytest pytest scripts/catalog/test_db_mapping.py +""" +import json +import pathlib +import re +import sqlite3 +import sys + +HERE = pathlib.Path(__file__).resolve().parent +sys.path.insert(0, str(HERE)) +import db # noqa: E402 + +SCHEMA = json.loads((HERE.parents[1] / "catalog" / "_schema.json").read_text()) + +# Row properties that are flattened or renamed rather than stored one to one. +ACCURACY_MAPPED = {"dataset": "dataset_id", "split": "dataset_id", "language": "dataset_id", + "ci95": "ci_lo/ci_hi", "errors": "substitutions/deletions/insertions"} +SPEED_MAPPED = {} + + +def columns(table: str) -> set[str]: + con = sqlite3.connect(":memory:") + con.executescript(db.SCHEMA) + return {row[1] for row in con.execute(f"PRAGMA table_info({table})")} + + +def check(section: str, table: str, mapped: dict[str, str]) -> None: + props = set(SCHEMA["properties"][section]["items"]["properties"]) + cols = columns(table) + missing = sorted(p for p in props if p not in cols and p not in mapped) + assert not missing, f"{section} properties with no {table} column: {missing}" + for prop, target in mapped.items(): + for col in re.split(r"/", target): + assert col in cols, f"{section}.{prop} maps to missing column {col}" + + +def test_accuracy_rows_reach_the_database(): + check("accuracy_benchmarks", "accuracy", ACCURACY_MAPPED) + + +def test_speed_rows_reach_the_database(): + check("speed_benchmarks", "speed", SPEED_MAPPED) + + +def test_download_rows_reach_the_database(): + check("downloads", "downloads", {}) diff --git a/scripts/compose-multitalker-bundle.py b/scripts/compose-multitalker-bundle.py index d4314e80..1c21f2ee 100644 --- a/scripts/compose-multitalker-bundle.py +++ b/scripts/compose-multitalker-bundle.py @@ -70,6 +70,13 @@ # Keys the GGUFReader synthesizes (virtual) or the writer emits itself. SKIP_COPY_KEYS = {"general.architecture"} +# Capability KVs describe the FILE, and the file being written is not the ASR +# half: embedding a diarizer is precisely what makes the bundle diarize. The +# ASR half either says speaker_diarization=false or omits it, so copying it +# verbatim is how the bundle came to advertise diarize:false while the runtime +# diarize path was implemented and working. Drop it on copy and state it below. +SKIP_COPY_KEYS |= {"stt.capability.speaker_diarization"} + # Diarizer checkpoints the runtime's multitalker path is validated against. # run_multitalker pins the reference operating point (14-frame chunk # cadence, spkcache/FIFO/update 188, gating threshold 0.5, 2-chunk gating @@ -143,6 +150,10 @@ def main() -> int: continue copy_kv(writer, field) + # Model-level diarization capability: a property of the bundle, not of + # either half. read_capability_kv() and the catalog both read this. + writer.add_bool("stt.capability.speaker_diarization", True) + writer.add_bool("stt.parakeet.diarizer.embedded", True) writer.add_string("stt.parakeet.diarizer.variant", diar.fields["stt.variant"].contents()) writer.add_string("stt.parakeet.diarizer.tensor_prefix", DIAR_TENSOR_PREFIX) diff --git a/scripts/convert-cohere.py b/scripts/convert-cohere.py index 25db603c..d624e80f 100755 --- a/scripts/convert-cohere.py +++ b/scripts/convert-cohere.py @@ -73,6 +73,7 @@ import argparse import json +import re import sys from pathlib import Path @@ -203,7 +204,9 @@ def read_hparams(config: dict, gen_config: dict) -> dict: "dec_max_seq": int(dec["max_sequence_length"]), "dec_activation": str(dec["hidden_act"]).lower(), - "vocab_size": int(config["vocab_size"]), + # arabic-07-2026 omits the top-level vocab_size; head.num_classes + # carries the same value in every Cohere ASR config. + "vocab_size": int(config.get("vocab_size", config["head"]["num_classes"])), "decoder_start_token_id": int(gen_config["decoder_start_token_id"]), "bos_token_id": int(gen_config["bos_token_id"]), "eos_token_id": int(gen_config["eos_token_id"]), @@ -427,8 +430,14 @@ def convert(model_dir: Path, out_path: Path, repo_id: str | None = None) -> None hp = read_hparams(config, gen_config) + variant = slug_from_repo_id(repo_id) if repo_id else "cohere-transcribe-03-2026" + # Variants are named -; split the release tag off the + # end for general.version / general.basename. + m = re.match(r"^(.+)-(\d{2}-\d{4})$", variant) + basename, version = (m.group(1), m.group(2)) if m else (variant, None) + print(f"vocab_size = {hp['vocab_size']}") - print(f"Variant: cohere-transcribe-03-2026") + print(f"Variant: {variant}") print(f"Reading tokenizer from {tokenizer_path}") tok = extract_tokenizer(tokenizer_path) @@ -463,8 +472,8 @@ def convert(model_dir: Path, out_path: Path, repo_id: str | None = None) -> None add_general_identity( writer, name="Cohere Transcribe", - version="03-2026", - basename="cohere-transcribe", + version=version, + basename=basename, size_label=size_label, file_type=REFERENCE_FILE_TYPE, languages=hp["languages"], @@ -476,8 +485,11 @@ def convert(model_dir: Path, out_path: Path, repo_id: str | None = None) -> None repo_url=(f"https://huggingface.co/{repo_id}" if repo_id else None), ) - # ----- stt.variant ----- - writer.add_string("stt.variant", "cohere-transcribe-03-2026") + # ----- stt.variant + capability surface ----- + writer.add_string("stt.variant", variant) + writer.add_bool("stt.capability.translate", False) + writer.add_bool("stt.capability.lang_detect", False) + writer.add_bool("stt.capability.streaming", False) # ----- tokenizer.ggml.* ----- writer.add_string("tokenizer.ggml.model", "bpe") diff --git a/scripts/convert-funasr_nano.py b/scripts/convert-funasr_nano.py index a1baeb42..6057bde0 100644 --- a/scripts/convert-funasr_nano.py +++ b/scripts/convert-funasr_nano.py @@ -611,8 +611,11 @@ def convert(model_dir: Path, out_path: Path, variant: str, display_name: str, re ), ) - # ----- stt.variant ----- + # ----- stt.variant + capability surface ----- writer.add_string("stt.variant", variant) + writer.add_bool("stt.capability.translate", False) + writer.add_bool("stt.capability.lang_detect", False) + writer.add_bool("stt.capability.streaming", False) # ----- tokenizer.ggml.* ----- # Qwen3 byte-level BPE — llama.cpp tags this as "gpt2". diff --git a/scripts/convert-gigaam.py b/scripts/convert-gigaam.py index 5765d582..23b94635 100644 --- a/scripts/convert-gigaam.py +++ b/scripts/convert-gigaam.py @@ -622,8 +622,11 @@ def convert(variant_key: str, slug: str, out_path: Path, repo_id: str | None = N repo_url=(f"https://huggingface.co/{repo_id}" if repo_id else None), ) - # ----- stt.variant + head_kind ----- + # ----- stt.variant + capability surface + head_kind ----- writer.add_string("stt.variant", profile["variant"]) + writer.add_bool("stt.capability.translate", False) + writer.add_bool("stt.capability.lang_detect", False) + writer.add_bool("stt.capability.streaming", False) writer.add_string("stt.gigaam.head_kind", head_kind) # ----- tokenizer.ggml.* ----- diff --git a/scripts/convert-parakeet.py b/scripts/convert-parakeet.py index e9808525..7e946eb2 100644 --- a/scripts/convert-parakeet.py +++ b/scripts/convert-parakeet.py @@ -1462,8 +1462,8 @@ def convert(model_spec: str, out_path: Path, repo_id: str | None = None) -> None # ----- stt.variant + capability KV ----- writer.add_string("stt.variant", profile["variant"]) - if profile["lang_detect"]: - writer.add_bool("stt.capability.lang_detect", True) + writer.add_bool("stt.capability.translate", False) + writer.add_bool("stt.capability.lang_detect", profile["lang_detect"]) # Streaming capability. Derived from the encoder attention geometry so # the header bool can never disagree with what the C++ loader computes: diff --git a/scripts/diar/compare_compress_spkcache.py b/scripts/diar/compare_compress_spkcache.py deleted file mode 100644 index 711939e2..00000000 --- a/scripts/diar/compare_compress_spkcache.py +++ /dev/null @@ -1,238 +0,0 @@ -#!/usr/bin/env python3 -""" -compare_compress_spkcache.py - Verify the Sortformer AOSC `_compress_spkcache` -step bit-for-bit between the NeMo reference and the transcribe.cpp C++ port -(residual-uncertainty #1). - -Both sides run the SAME clip at the SAME streaming preset, so the sequence of -compression calls lines up 1:1. For each call we compare: - - * frame_idx the sorted selected frame indices (C++ `frame_idx` vs NeMo - `_get_topk_indices` -> `topk_indices_sorted`, post-remainder, - disabled entries set to 0). An EXACT match confirms the topk - tie-break hypothesis (ATen CPU topk: value desc, flat-index - asc; -inf picks -> max_index before the ascending sort) on real - drifted scores. - * is_disabled the disabled mask (silence-pad / -inf placeholder frames). - * spkcache_preds the gathered [spkcache_len, n_spk] preds (numeric; small - F32 drift expected, reported as max_abs). - -NeMo dump: `compress.{k:03d}.{topk_indices,is_disabled,spkcache_preds}.npy` - (scripts/dump_reference_sortformer_nemo.py diarize --dump-compress). -C++ dump: `compress.{k:03d}.{frame_idx,is_disabled,spkcache_preds}.{f32,json}` - (TRANSCRIBE_SORTFORMER_COMPRESS_DUMP=1 + TRANSCRIBE_DUMP_DIR). - -Usage: - uv run scripts/diar/compare_compress_spkcache.py \ - --ref-dir build/validate/sortformer/.../compress-ref \ - --cpp-dir build/validate/sortformer/.../compress-cpp -""" - -from __future__ import annotations - -import argparse -import json -import sys -from pathlib import Path - -import numpy as np - - -def _boost_topk(scores: np.ndarray, k: int, scale: float) -> np.ndarray: - """Increase the k highest scores per speaker column. Tie-break: value desc, - frame-index asc (ATen CPU topk). Mirrors _boost_topk_scores.""" - if k <= 0: - return scores - delta = scale * np.log(0.5) # negative -> subtracting boosts - n = scores.shape[0] - for s in range(scores.shape[1]): - col = scores[:, s] - order = np.lexsort((np.arange(n), -col)) # primary value desc, secondary idx asc - col[order[:min(k, n)]] -= delta - return scores - - -def select_indices(preds: np.ndarray, spkcache_len: int, sil_per_spk: int = 3, - pred_thr: float = 0.25, boost_latest: float = 0.05, - strong_rate: float = 0.75, weak_rate: float = 1.5, - min_pos_rate: float = 0.5, max_index: int = 99999): - """Independent numpy reimplementation of the Sortformer AOSC frame selection - (_get_log_pred_scores -> _disable_low_scores -> scores_boost_latest -> - _boost_topk_scores x2 -> silence pad -> _get_topk_indices). Returns - (frame_idx [spkcache_len], is_disabled [spkcache_len]). A third, neutral - arbiter for both the NeMo and C++ dumps.""" - N, n_spk = preds.shape - per_spk = spkcache_len // n_spk - sil_per_spk - import math - strong = math.floor(per_spk * strong_rate) - weak = math.floor(per_spk * weak_rate) - min_pos = math.floor(per_spk * min_pos_rate) - - lp = np.log(np.clip(preds, pred_thr, None)) - l1 = np.log(np.clip(1.0 - preds, pred_thr, None)) - scores = lp - l1 + l1.sum(1, keepdims=True) - math.log(0.5) - - is_speech = preds > 0.5 - scores = np.where(is_speech, scores, -np.inf) - is_pos = scores > 0 - n_pos = is_pos.sum(0, keepdims=True) - scores = np.where((~is_pos) & is_speech & (n_pos >= min_pos), -np.inf, scores) - - if boost_latest > 0: - scores[spkcache_len:, :] += boost_latest # -inf + x stays -inf - - scores = _boost_topk(scores, strong, 2.0) - scores = _boost_topk(scores, weak, 1.0) - - if sil_per_spk > 0: - scores = np.concatenate([scores, np.full((sil_per_spk, n_spk), np.inf)], axis=0) - n_frames = scores.shape[0] - n_frames_no_sil = n_frames - sil_per_spk - - flat = scores.T.reshape(-1) # flat index f = s * n_frames + i - order = np.lexsort((np.arange(flat.size), -flat)) # value desc, flat-idx asc - topk = order[:spkcache_len].copy() - topk = np.where(flat[topk] != -np.inf, topk, max_index) - topk_sorted = np.sort(topk) - is_disabled = topk_sorted == max_index - frame = np.remainder(topk_sorted, n_frames) - is_disabled = is_disabled | (frame >= n_frames_no_sil) - frame = np.where(is_disabled, 0, frame) - return frame.astype(np.int64), is_disabled.astype(bool) - - -def _load_cpp_f32(cpp_dir: Path, stem: str) -> np.ndarray: - """Load a C++ dump_host_f32 artifact (.f32 payload + .json shape).""" - meta = json.loads((cpp_dir / f"{stem}.json").read_text()) - shape = tuple(int(x) for x in meta["shape"]) # slow-to-fast - data = np.fromfile(cpp_dir / f"{stem}.f32", dtype=" int: - ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - ap.add_argument("--ref-dir", required=True, help="NeMo --dump-compress output dir") - ap.add_argument("--cpp-dir", required=True, help="C++ TRANSCRIBE_DUMP_DIR (compress.* artifacts)") - ap.add_argument("--preds-tol", type=float, default=1e-2, - help="Max allowed |Δ| on gathered spkcache_preds (F32 drift).") - ap.add_argument("--spkcache-len", type=int, default=188, - help="spkcache_len for the numpy adjudicator (188 for the shipped presets, " - "24 for `small`). Only used when input_preds dumps are present.") - args = ap.parse_args() - - ref_dir, cpp_dir = Path(args.ref_dir), Path(args.cpp_dir) - ref_calls = sorted(ref_dir.glob("compress.*.topk_indices.npy")) - cpp_calls = sorted(cpp_dir.glob("compress.*.frame_idx.json")) - n_ref, n_cpp = len(ref_calls), len(cpp_calls) - print(f"compression calls: ref={n_ref} cpp={n_cpp}") - if n_ref == 0: - raise SystemExit("error: no reference compression dumps found " - "(did the preset actually trigger AOSC compression?)") - if n_ref != n_cpp: - print(f"MISMATCH: call count differs (ref {n_ref} vs cpp {n_cpp}) -> streaming " - "trajectories diverged before/around compression", file=sys.stderr) - return 1 - - all_idx_ok = True - all_dis_ok = True - max_preds = 0.0 - # Adjudicator bookkeeping: does each side's own dumped selection match an - # independent numpy reimplementation run on that side's own input preds? - ref_self_ok = cpp_self_ok = True - ref_self_seen = cpp_self_seen = False - first_traj_div = None # first call where ref/cpp selections diverge - for k in range(n_ref): - ref_idx = np.load(ref_dir / f"compress.{k:03d}.topk_indices.npy").astype(np.int64) - ref_dis = np.load(ref_dir / f"compress.{k:03d}.is_disabled.npy").astype(bool) - cpp_idx = _load_cpp_f32(cpp_dir, f"compress.{k:03d}.frame_idx").astype(np.int64) - cpp_dis = _load_cpp_f32(cpp_dir, f"compress.{k:03d}.is_disabled").astype(bool) - - # Independent-arbiter check: run the numpy selection on each side's own - # input preds and confirm it reproduces that side's dumped selection. - ref_ipf = ref_dir / f"compress.{k:03d}.input_preds.npy" - cpp_ipf = cpp_dir / f"compress.{k:03d}.input_preds.json" - if ref_ipf.exists(): - ref_self_seen = True - nf, nd = select_indices(np.load(ref_ipf).astype(np.float32), args.spkcache_len) - if int(np.sum(nf != ref_idx)) or int(np.sum(nd != ref_dis)): - ref_self_ok = False - print(f" [{k:03d}] numpy-arbiter != NeMo dump " - f"(idx {int(np.sum(nf != ref_idx))}, dis {int(np.sum(nd != ref_dis))})", file=sys.stderr) - if cpp_ipf.exists(): - cpp_self_seen = True - nf, nd = select_indices(_load_cpp_f32(cpp_dir, f"compress.{k:03d}.input_preds"), args.spkcache_len) - if int(np.sum(nf != cpp_idx)) or int(np.sum(nd != cpp_dis)): - cpp_self_ok = False - print(f" [{k:03d}] numpy-arbiter != C++ dump " - f"(idx {int(np.sum(nf != cpp_idx))}, dis {int(np.sum(nd != cpp_dis))})", file=sys.stderr) - - if ref_idx.shape != cpp_idx.shape: - print(f" [{k:03d}] SHAPE MISMATCH idx ref{ref_idx.shape} cpp{cpp_idx.shape}", file=sys.stderr) - all_idx_ok = False - continue - idx_mism = int(np.sum(ref_idx != cpp_idx)) - dis_mism = int(np.sum(ref_dis != cpp_dis)) - all_idx_ok = all_idx_ok and idx_mism == 0 - all_dis_ok = all_dis_ok and dis_mism == 0 - if idx_mism and first_traj_div is None: - first_traj_div = k - - preds_line = "" - ref_pf = ref_dir / f"compress.{k:03d}.spkcache_preds.npy" - cpp_pf = cpp_dir / f"compress.{k:03d}.spkcache_preds.json" - if ref_pf.exists() and cpp_pf.exists(): - ref_p = np.load(ref_pf).astype(np.float32) - cpp_p = _load_cpp_f32(cpp_dir, f"compress.{k:03d}.spkcache_preds") - if ref_p.shape == cpp_p.shape: - m = float(np.max(np.abs(ref_p - cpp_p))) if ref_p.size else 0.0 - max_preds = max(max_preds, m) - preds_line = f" preds_max_abs {m:.2e}" - else: - preds_line = f" preds SHAPE MISMATCH ref{ref_p.shape} cpp{cpp_p.shape}" - - flag = "OK " if (idx_mism == 0 and dis_mism == 0) else "*** " - print(f" [{k:03d}] {flag}n={ref_idx.shape[0]:4d} idx_mismatch={idx_mism:4d} " - f"dis_mismatch={dis_mism:4d}{preds_line}") - - print() - print("== end-to-end trajectory (each side fed its OWN drifted preds) ==") - print(f"frame_idx : {'MATCH (bit-exact)' if all_idx_ok else 'MISMATCH'}" - + ("" if all_idx_ok else f" (first divergence at call {first_traj_div})")) - print(f"is_disabled : {'MATCH (bit-exact)' if all_dis_ok else 'MISMATCH'}") - print(f"spkcache_preds max_abs: {max_preds:.2e} (tol {args.preds_tol:.1e})") - - # The definitive residual-#1 check: is the compression FUNCTION correct? - # Independent numpy arbiter vs each side's own dump on that side's own input. - print() - print("== compression function (independent numpy arbiter, per-side own input) ==") - if ref_self_seen: - print(f"NeMo selection == numpy arbiter : {'MATCH' if ref_self_ok else 'MISMATCH'}") - if cpp_self_seen: - print(f"C++ selection == numpy arbiter : {'MATCH' if cpp_self_ok else 'MISMATCH'}") - if not (ref_self_seen and cpp_self_seen): - print("(input_preds dumps absent on one/both sides -> function-level arbiter skipped; " - "re-dump with the updated hooks to enable it)") - - # PASS criteria: if input_preds are present, the compression function is - # proven bit-exact against a neutral arbiter on both sides' real inputs - # (trajectory drift is expected and does NOT fail the check). If they are - # absent, fall back to end-to-end bit-exactness. - print() - if ref_self_seen and cpp_self_seen: - ok = ref_self_ok and cpp_self_ok - note = ("compression function bit-exact on both sides vs neutral arbiter" - if ok else "compression function disagrees with arbiter") - if ok and not all_idx_ok: - note += (f"; end-to-end trajectories diverge from call {first_traj_div} " - "purely via upstream F32 preds drift (near-tie topk flips)") - else: - ok = all_idx_ok and all_dis_ok and max_preds <= args.preds_tol - note = "compression internal state matches" if ok else "internal state mismatch" - print("VERDICT:", ("PASS - " + note) if ok else ("FAIL - " + note)) - return 0 if ok else 1 - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/scripts/hf_cards/README.md b/scripts/hf_cards/README.md new file mode 100644 index 00000000..4158afe1 --- /dev/null +++ b/scripts/hf_cards/README.md @@ -0,0 +1,27 @@ +# HF cards + +`.yaml` is prose only: summary, tags, validation pin, notes. Every +number, repo, licence, language, and capability comes from +`catalog/.json`, as do the metric column label and the link back to +the model page. `generate.py` accepts only the editorial keys and refuses +everything else, so a catalog-owned field cannot creep back in and a +misspelled key fails instead of silently rendering nothing. +`summary` and `wer.notes` are also the source for `docs/models/.md`, +rendered into its `catalog:intro` and `catalog:prose` markers by +`scripts/catalog/render.py`. The mechanical WER sentence (dataset, size, +batch, timestamps, build) is generated from the headline rows; `wer.notes` +holds only editorial caveats: state a number there only to compare against +something the catalog does not hold, such as an upstream self-reported +figure. A second download-table column is +`wer.source2` plus `wer.secondary: __` naming a +result set the catalog holds. + +```bash +uv run scripts/hf_cards/check_release.py # pin + validation date +uv run scripts/hf_cards/generate.py scripts/hf_cards/.yaml + # -> models//README.md +hf upload handy-computer/-gguf models/ . --repo-type model +``` + +Re-render and re-upload whenever the catalog record changes (new WER sweep, +re-bench, capability fix). Repos stay private until a maintainer flips them. diff --git a/scripts/hf_cards/breeze-asr-25.yaml b/scripts/hf_cards/breeze-asr-25.yaml index d639861d..3dcf24b6 100644 --- a/scripts/hf_cards/breeze-asr-25.yaml +++ b/scripts/hf_cards/breeze-asr-25.yaml @@ -1,14 +1,8 @@ # Spec for the HF README of handy-computer/Breeze-ASR-25-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: MediaTek-Research/Breeze-ASR-25 -target_repo: handy-computer/Breeze-ASR-25-gguf # Pure fine-tune of whisper-large-v2 (identical architecture), so the model # page points at the whisper-large-v2 doc rather than a Breeze-specific one. -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-large-v2.md - -# Pinned at download (the snapshot the GGUFs were converted from). -upstream_commit: cffe7ccb404d025296a00758d0a33468bec3a9d0 pin_date: 2026-06-29 validation: @@ -16,18 +10,7 @@ validation: commit: 3848875 date: 2026-06-29 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition - -# Breeze-ASR-25 is a Whisper-large-v2 fine-tune. It inherits Whisper's -# 99-language tokenizer, but MediaTek optimized and validated it for -# Taiwanese Mandarin (Traditional Chinese) and English only, so we advertise -# just those two — listing all 99 would overclaim coverage the fine-tune -# does not target. -languages: - - zh - - en tags: - gguf - transcribe.cpp @@ -51,39 +34,12 @@ summary: | tokenizer, but only Mandarin and English are optimized and validated — other languages remain technically accessible but out of scope. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. Same Whisper -# code path as whisper-large-v2 (the library exposes translation, language -# detection and segment timestamps for any Whisper GGUF), though this -# fine-tune is specialized for zh/en transcription. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, published raw as rtf_ -# in the metadata block. Breeze-ASR-25 is byte-for-byte the same architecture -# and tensor shapes as Whisper large-v2 (1.5B params, 32+32 layers, -# d_model 1280); RTF is a function of shapes/quant/backend, not weight values, -# so these are inherited verbatim from whisper-large-v2 rather than re-benched. -perf: - m4-max: - metal: 23.7 - cpu: 1.5 - ryzen-4750u: - vulkan: 2.1 - cpu: 0.6 - wer: - source: LibriSpeech test-clean - metadata_key: librispeech_test_clean - source2: FLEURS zh + source2: FLEURS zh (OpenCC t2s-folded) + secondary: cer_fleurs_zh_opencc_t2s notes: | - Two benchmarks, both full test splits, decoded on a Modal L40S with the - transcribe.cpp default recipe (greedy + temperature fallback, - suppress_tokens, segment timestamps), batch 1. + Both benchmarks use the transcribe.cpp default recipe (greedy with temperature + fallback, suppress_tokens) on a Modal L40S. **English — WER, LibriSpeech test-clean (2620 utterances).** Standard Whisper/`EnglishTextNormalizer` scoring. @@ -98,44 +54,3 @@ wer: Quantization is effectively free on both languages: every quant down to Q4_K_M (1.0 GB) sits within run-to-run noise of the BF16 reference (English 2.25-2.29%, Chinese 8.08-8.12%). - - # Secondary per-quant metric → wer_fleurs_zh in the metadata block. - fleurs_zh: - bf16: 8.12 - f16: 8.11 - q8_0: 8.10 - q6_k: 8.12 - q5_k_m: 8.12 - q4_k_m: 8.08 - -quants: - - name: BF16 - filename: Breeze-ASR-25-BF16.gguf - size: 3.10 GB - wer: 2.29% - wer2: 8.12% - - name: F16 - filename: Breeze-ASR-25-F16.gguf - size: 3.11 GB - wer: 2.29% - wer2: 8.11% - - name: Q8_0 - filename: Breeze-ASR-25-Q8_0.gguf - size: 1.67 GB - wer: 2.27% - wer2: 8.10% - - name: Q6_K - filename: Breeze-ASR-25-Q6_K.gguf - size: 1.30 GB - wer: 2.29% - wer2: 8.12% - - name: Q5_K_M - filename: Breeze-ASR-25-Q5_K_M.gguf - size: 1.16 GB - wer: 2.25% - wer2: 8.12% - - name: Q4_K_M - filename: Breeze-ASR-25-Q4_K_M.gguf - size: 1.00 GB - wer: 2.26% - wer2: 8.08% diff --git a/scripts/hf_cards/canary-180m-flash.yaml b/scripts/hf_cards/canary-180m-flash.yaml index 0af40950..d425f6e2 100644 --- a/scripts/hf_cards/canary-180m-flash.yaml +++ b/scripts/hf_cards/canary-180m-flash.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/canary-180m-flash-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/canary-180m-flash -target_repo: handy-computer/canary-180m-flash-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/canary-180m-flash.md - -upstream_commit: b12ab41 pin_date: 2026-05-08 # Validation pin for the most recent upload. Updated on each release — @@ -15,14 +10,7 @@ validation: commit: db53eda date: 2026-05-08 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - de - - es - - fr tags: - gguf - transcribe.cpp @@ -34,66 +22,16 @@ tags: - translation summary: | - Offline multilingual speech-to-text and translation. A 182M-parameter - multitask AED with a 17-layer FastConformer encoder and a 4-layer - Transformer decoder. Supports automatic speech recognition in English, - German, Spanish, and French, and bidirectional EN↔{DE, ES, FR} - translation. Takes a - 16 kHz mono WAV and produces a transcript. Not a streaming model; - word/segment timestamps are upstream-experimental and not exposed in - the v1 port. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 139.6 - cpu: 74 - ryzen-4750u: - vulkan: 31.9 - cpu: 21.3 + Offline multilingual speech-to-text and translation. A multitask AED + with a 17-layer FastConformer encoder and a 4-layer Transformer decoder. + Supports automatic speech recognition in English, German, Spanish, and + French, and bidirectional EN↔{DE, ES, FR} translation. Takes a 16 kHz + mono WAV and produces a transcript. Not a streaming model; word/segment + timestamps are upstream-experimental and not exposed in the v1 port. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding and no external LM. F32 reference baseline: 1.94%. - On the same wavs, NeMo's reference run produces 1.93% (one substitution - difference out of ~27k reference words), so the F32 port matches the - reference framework at the noise floor. NVIDIA's self-reported number - on the upstream model card is 1.87%. - -quants: - - name: F32 - filename: canary-180m-flash-F32.gguf - size: 721 MB - wer: 1.94% - - name: F16 - filename: canary-180m-flash-F16.gguf - size: 364 MB - wer: 1.94% - - name: Q8_0 - filename: canary-180m-flash-Q8_0.gguf - size: 208 MB - wer: 1.93% - - name: Q6_K - filename: canary-180m-flash-Q6_K.gguf - size: 168 MB - wer: 1.93% - - name: Q5_K_M - filename: canary-180m-flash-Q5_K_M.gguf - size: 151 MB - wer: 1.90% - - name: Q4_K_M - filename: canary-180m-flash-Q4_K_M.gguf - size: 133 MB - wer: 1.93% + Greedy decoding, no external LM. F32 reference baseline: 1.94%. On the same wavs, + NeMo's reference run produces 1.93% (one substitution difference out of ~27k + reference words), so the F32 port matches the reference framework at the noise + floor. NVIDIA's self-reported number on the upstream model card is 1.87%. diff --git a/scripts/hf_cards/canary-1b-flash.yaml b/scripts/hf_cards/canary-1b-flash.yaml index 135debe2..fa76912d 100644 --- a/scripts/hf_cards/canary-1b-flash.yaml +++ b/scripts/hf_cards/canary-1b-flash.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/canary-1b-flash-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/canary-1b-flash -target_repo: handy-computer/canary-1b-flash-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/canary-1b-flash.md - -upstream_commit: a9a55e0 pin_date: 2026-05-08 validation: @@ -13,14 +8,7 @@ validation: commit: db53eda date: 2026-05-08 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - de - - es - - fr tags: - gguf - transcribe.cpp @@ -32,63 +20,14 @@ tags: - translation summary: | - Offline multilingual speech-to-text and translation. An 883M-parameter - multitask AED with a 32-layer FastConformer encoder and a 4-layer - Transformer decoder. Supports automatic speech recognition in English, - German, Spanish, and French, and bidirectional EN↔{DE, ES, FR} - translation. Takes a - 16 kHz mono WAV and produces a transcript. Not a streaming model; - word/segment timestamps are upstream-experimental and not exposed in - the v1 port. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 99.6 - cpu: 20.4 - ryzen-4750u: - vulkan: 14.4 - cpu: 6.9 + Offline multilingual speech-to-text and translation. A multitask AED + with a 32-layer FastConformer encoder and a 4-layer Transformer decoder. + Supports automatic speech recognition in English, German, Spanish, and + French, and bidirectional EN↔{DE, ES, FR} translation. Takes a 16 kHz + mono WAV and produces a transcript. Not a streaming model; word/segment + timestamps are upstream-experimental and not exposed in the v1 port. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding and no external LM. F32 reference baseline: 1.62%. - NVIDIA's self-reported number on the upstream model card is 1.48%. - -quants: - - name: F32 - filename: canary-1b-flash-F32.gguf - size: 3.3 GB - wer: 1.62% - - name: F16 - filename: canary-1b-flash-F16.gguf - size: 1.7 GB - wer: 1.62% - - name: Q8_0 - filename: canary-1b-flash-Q8_0.gguf - size: 1.0 GB - wer: 1.62% - - name: Q6_K - filename: canary-1b-flash-Q6_K.gguf - size: 818 MB - wer: 1.65% - - name: Q5_K_M - filename: canary-1b-flash-Q5_K_M.gguf - size: 734 MB - wer: 1.64% - - name: Q4_K_M - filename: canary-1b-flash-Q4_K_M.gguf - size: 646 MB - wer: 1.59% + Greedy decoding, no external LM. F32 reference baseline: 1.62%. NVIDIA's + self-reported number on the upstream model card is 1.48%. diff --git a/scripts/hf_cards/canary-1b-v2.yaml b/scripts/hf_cards/canary-1b-v2.yaml index 6e76e333..3c2e8342 100644 --- a/scripts/hf_cards/canary-1b-v2.yaml +++ b/scripts/hf_cards/canary-1b-v2.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/canary-1b-v2-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/canary-1b-v2 -target_repo: handy-computer/canary-1b-v2-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/canary-1b-v2.md - -upstream_commit: 87bc526 pin_date: 2026-05-08 validation: @@ -13,35 +8,7 @@ validation: commit: db53eda date: 2026-05-08 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - bg - - hr - - cs - - da - - nl - - en - - et - - fi - - fr - - de - - el - - hu - - it - - lv - - lt - - mt - - pl - - pt - - ro - - sk - - sl - - es - - sv - - ru - - uk tags: - gguf - transcribe.cpp @@ -55,64 +22,16 @@ tags: summary: | Offline multilingual speech-to-text and translation across 25 European - languages. A 978M-parameter multitask AED with a 32-layer FastConformer - encoder and an 8-layer Transformer decoder. Supports automatic speech - recognition for any of the 25 supported languages, plus translation - between supported language pairs (per the upstream model card). Takes - a 16 kHz mono WAV and produces a transcript. Not a streaming model; - word and segment timestamps from the upstream model are not exposed in - the v1 port. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 85.2 - cpu: 18.9 - ryzen-4750u: - vulkan: 13.2 - cpu: 6.7 + languages. A multitask AED with a 32-layer FastConformer encoder and an + 8-layer Transformer decoder. Supports automatic speech recognition for + any of the 25 supported languages, plus translation between supported + language pairs (per the upstream model card). Takes a 16 kHz mono WAV and + produces a transcript. Not a streaming model; word and segment timestamps + from the upstream model are not exposed in the v1 port. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding and no external LM. F32 reference baseline: 1.92%. - NVIDIA's self-reported number on the upstream model card is 2.18%; - our F32 port comes in slightly under the upstream-reported number - (Δ −0.26pp) and is likely down to scoring differences. - -quants: - - name: F32 - filename: canary-1b-v2-F32.gguf - size: 3.7 GB - wer: 1.92% - - name: F16 - filename: canary-1b-v2-F16.gguf - size: 1.8 GB - wer: 1.92% - - name: Q8_0 - filename: canary-1b-v2-Q8_0.gguf - size: 1.1 GB - wer: 1.91% - - name: Q6_K - filename: canary-1b-v2-Q6_K.gguf - size: 889 MB - wer: 1.94% - - name: Q5_K_M - filename: canary-1b-v2-Q5_K_M.gguf - size: 798 MB - wer: 1.93% - - name: Q4_K_M - filename: canary-1b-v2-Q4_K_M.gguf - size: 701 MB - wer: 1.91% + Greedy decoding, no external LM. F32 reference baseline: 1.92%. NVIDIA's + self-reported number on the upstream model card is 2.18%; our F32 port comes in + slightly under the upstream-reported number (Δ −0.26pp) and is likely down to + scoring differences. diff --git a/scripts/hf_cards/canary-1b.yaml b/scripts/hf_cards/canary-1b.yaml index f560caec..57013e85 100644 --- a/scripts/hf_cards/canary-1b.yaml +++ b/scripts/hf_cards/canary-1b.yaml @@ -1,5 +1,5 @@ # Spec for the HF README of handy-computer/canary-1b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. # # canary-1b is the only family member under a non-commercial license. # The converter writes general.license="CC-BY-NC-4.0" and @@ -7,11 +7,6 @@ # into every preset; the YAML below mirrors that distinction so the HF # card surfaces the constraint to anyone clicking through to download. -hf_repo: nvidia/canary-1b -target_repo: handy-computer/canary-1b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/canary-1b.md - -upstream_commit: 1698acf pin_date: 2026-05-08 validation: @@ -19,14 +14,7 @@ validation: commit: db53eda date: 2026-05-08 -license: cc-by-nc-4.0 -license_display: CC-BY-NC-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - de - - es - - fr tags: - gguf - transcribe.cpp @@ -39,65 +27,18 @@ tags: - non-commercial summary: | - Offline multilingual speech-to-text and translation. A 1B-parameter - multitask AED with a 24-layer FastConformer encoder and a 24-layer - Transformer decoder — the original canary release. Supports automatic - speech recognition in English, German, Spanish, and French, and - translation between supported pairs. Takes a 16 kHz mono WAV and - produces a transcript. Not a streaming model. - **License: CC-BY-NC-4.0 (non-commercial only)** — the only canary - variant under a non-commercial license. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 42.5 - cpu: 19.4 - ryzen-4750u: - vulkan: 8.9 - cpu: 5.6 + Offline multilingual speech-to-text and translation. A multitask AED + with a 24-layer FastConformer encoder and a 24-layer Transformer + decoder — the original canary release. Supports automatic speech + recognition in English, German, Spanish, and French, and translation + between supported pairs. Takes a 16 kHz mono WAV and produces a + transcript. Not a streaming model. **License: CC-BY-NC-4.0 + (non-commercial only)** — the only canary variant under a + non-commercial license. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding and no external LM. F32 reference baseline: 1.55%. - NVIDIA's self-reported number on the upstream model card is 1.48%; - likely this is due to differences in how we score WER, based on the - results we have from canary-180m-flash. - -quants: - - name: F32 - filename: canary-1b-F32.gguf - size: 3.8 GB - wer: 1.55% - - name: F16 - filename: canary-1b-F16.gguf - size: 1.9 GB - wer: 1.55% - - name: Q8_0 - filename: canary-1b-Q8_0.gguf - size: 1.1 GB - wer: 1.55% - - name: Q6_K - filename: canary-1b-Q6_K.gguf - size: 891 MB - wer: 1.57% - - name: Q5_K_M - filename: canary-1b-Q5_K_M.gguf - size: 799 MB - wer: 1.57% - - name: Q4_K_M - filename: canary-1b-Q4_K_M.gguf - size: 696 MB - wer: 1.55% + Greedy decoding, no external LM. F32 reference baseline: 1.55%. NVIDIA's + self-reported number on the upstream model card is 1.48%; likely this is due to + differences in how we score WER, based on the results we have from + canary-180m-flash. diff --git a/scripts/hf_cards/canary-qwen-2.5b.yaml b/scripts/hf_cards/canary-qwen-2.5b.yaml index d66fc243..6f2fe286 100644 --- a/scripts/hf_cards/canary-qwen-2.5b.yaml +++ b/scripts/hf_cards/canary-qwen-2.5b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/canary-qwen-2.5b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/canary-qwen-2.5b -target_repo: handy-computer/canary-qwen-2.5b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/canary-qwen-2.5b.md - -upstream_commit: b1469e1bba1cfe140205529c79c434ca47180960 pin_date: 2026-05-15 validation: @@ -13,11 +8,7 @@ validation: commit: 6f6c699 date: 2026-05-16 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -38,63 +29,14 @@ summary: | the prompt. English only. Takes a 16 kHz mono WAV and produces a transcript via greedy decoding. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 39.8 - cpu: 7.3 - ryzen-4750u: - vulkan: 4.1 - cpu: 2.1 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech `test-clean` split (2620 English - utterances) with the Whisper-style English text normalizer and jiwer - 3.x, on the metal backend of an Apple M4. The same-machine NeMo SALM - reference run (CPU torch, dither=0.0, greedy `model.generate`) lands - at **1.61%** with 95% bootstrap CI [1.47%, 1.75%]: `0.01` above - NVIDIA's published 1.60% but well within statistical noise. All six - GGUF presets land at exactly 1.63% (`+0.02` over our reference run, - same CI band). Investigation of the worst per-utterance differences - shows scattered token-level noise consistent with BF16 weight - precision (homophones, word-boundary flips, function-word - substitutions). Reproduce with `scripts/wer/run.py` + + Scored with the Whisper-style English text normalizer and jiwer 3.x on an Apple M4. + The same-machine NeMo SALM reference run (CPU torch, dither=0.0, greedy + `model.generate`) lands at **1.61%** with 95% bootstrap CI [1.47%, 1.75%]: `0.01` + above NVIDIA's published 1.60% but well within statistical noise. All six GGUF + presets land at exactly 1.63% (`+0.02` over our reference run, same CI band). + Investigation of the worst per-utterance differences shows scattered token-level + noise consistent with BF16 weight precision (homophones, word-boundary flips, + function-word substitutions). Reproduce with `scripts/wer/run.py` + `scripts/wer/score.py`. - -quants: - - name: BF16 - filename: canary-qwen-2.5b-BF16.gguf - size: 4.73 GB - wer: 1.63% - - name: F16 - filename: canary-qwen-2.5b-F16.gguf - size: 4.73 GB - wer: 1.63% - - name: Q8_0 - filename: canary-qwen-2.5b-Q8_0.gguf - size: 2.61 GB - wer: 1.63% - - name: Q6_K - filename: canary-qwen-2.5b-Q6_K.gguf - size: 2.06 GB - wer: 1.63% - - name: Q5_K_M - filename: canary-qwen-2.5b-Q5_K_M.gguf - size: 1.85 GB - wer: 1.63% - - name: Q4_K_M - filename: canary-qwen-2.5b-Q4_K_M.gguf - size: 1.62 GB - wer: 1.63% diff --git a/scripts/hf_cards/check_release.py b/scripts/hf_cards/check_release.py new file mode 100644 index 00000000..0f925017 --- /dev/null +++ b/scripts/hf_cards/check_release.py @@ -0,0 +1,79 @@ +#!/usr/bin/env -S uv run --script +# /// script +# requires-python = ">=3.11" +# dependencies = ["pyyaml"] +# /// +"""Check the date and validation pin in an HF card before upload.""" +from __future__ import annotations + +import argparse +import datetime as dt +import pathlib +import re +import subprocess +import sys + +import yaml + +REPO = pathlib.Path(__file__).resolve().parents[2] +HEX_SHA = re.compile(r"^[0-9a-fA-F]{7,40}$") + + +def as_date(value, field: str) -> dt.date: + if isinstance(value, dt.date): + return value + try: + return dt.date.fromisoformat(str(value)) + except ValueError as exc: + raise ValueError(f"{field} must be an ISO date, got {value!r}") from exc + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("variant") + parser.add_argument("--date", default=dt.datetime.now(dt.timezone.utc).date().isoformat(), + help="UTC ship date expected in validation.date (default: today)") + args = parser.parse_args() + + expected_date = as_date(args.date, "--date") + path = REPO / "scripts" / "hf_cards" / f"{args.variant}.yaml" + if not path.exists(): + print(f"FAIL: no HF card spec at {path}", file=sys.stderr) + return 2 + spec = yaml.safe_load(path.read_text()) or {} + errors = [] + try: + pin_date = as_date(spec.get("pin_date"), "pin_date") + if pin_date > expected_date: + errors.append(f"pin_date {pin_date} is after ship date {expected_date}") + except ValueError as exc: + errors.append(str(exc)) + + validation = spec.get("validation") or {} + try: + validation_date = as_date(validation.get("date"), "validation.date") + if validation_date != expected_date: + errors.append( + f"validation.date is {validation_date}, expected ship date {expected_date}" + ) + except ValueError as exc: + errors.append(str(exc)) + commit = str(validation.get("commit") or "") + if not HEX_SHA.fullmatch(commit): + errors.append(f"validation.commit is not a commit SHA: {commit!r}") + elif subprocess.run( + ["git", "cat-file", "-e", f"{commit}^{{commit}}"], cwd=REPO, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, + ).returncode != 0: + errors.append(f"validation.commit {commit} is not present in this checkout") + + if errors: + for error in errors: + print(f"FAIL {args.variant}: {error}", file=sys.stderr) + return 1 + print(f"OK {args.variant}: pin_date={pin_date}, validation={commit} on {expected_date}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/hf_cards/cohere-transcribe-03-2026.yaml b/scripts/hf_cards/cohere-transcribe-03-2026.yaml index c9f41060..f3a1245d 100644 --- a/scripts/hf_cards/cohere-transcribe-03-2026.yaml +++ b/scripts/hf_cards/cohere-transcribe-03-2026.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/cohere-transcribe-03-2026-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: CohereLabs/cohere-transcribe-03-2026 -target_repo: handy-computer/cohere-transcribe-03-2026-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/cohere-transcribe-03-2026.md - -upstream_commit: 76b8b23 pin_date: 2026-04-16 # Validation pin for the most recent upload. Updated on each release — @@ -15,24 +10,7 @@ validation: commit: bf0d0b7 date: 2026-04-18 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - de - - es - - it - - pt - - nl - - pl - - el - - ar - - ja - - zh - - vi - - ko tags: - gguf - transcribe.cpp @@ -50,57 +28,10 @@ summary: | encoder-decoder head (cross-attention, tied token embedding). Takes a 16 kHz mono WAV and produces a transcript. Decoding is autoregressive. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 73 - cpu: 9 - ryzen-4750u: - vulkan: 8 - cpu: 3 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with - greedy decoding and no external LM. BF16 reference baseline: 1.26%. - Cohere's self-reported number on the same split is 1.25% - (Open ASR Leaderboard, as of 2026-03-26). Both ours and Cohere's numbers - use the Whisper EnglishTextNormalizer, so the comparison is apples-to-apples - and our port matches the upstream reference within rounding. - -quants: - - name: BF16 - filename: cohere-transcribe-03-2026-BF16.gguf - size: 4.10 GB - wer: 1.26% - - name: F16 - filename: cohere-transcribe-03-2026-F16.gguf - size: 4.11 GB - wer: 1.26% - - name: Q8_0 - filename: cohere-transcribe-03-2026-Q8_0.gguf - size: 2.41 GB - wer: 1.27% - - name: Q6_K - filename: cohere-transcribe-03-2026-Q6_K.gguf - size: 1.97 GB - wer: 1.27% - - name: Q5_K_M - filename: cohere-transcribe-03-2026-Q5_K_M.gguf - size: 1.76 GB - wer: 1.25% - - name: Q4_K_M - filename: cohere-transcribe-03-2026-Q4_K_M.gguf - size: 1.55 GB - wer: 1.25% + Greedy decoding, no external LM. BF16 reference baseline: 1.26%. Cohere's + self-reported number on the same split is 1.25% (Open ASR Leaderboard, as of + 2026-03-26). Both ours and Cohere's numbers use the Whisper EnglishTextNormalizer, + so the comparison is apples-to-apples and our port matches the upstream reference + within rounding. diff --git a/scripts/hf_cards/cohere-transcribe-arabic-07-2026.yaml b/scripts/hf_cards/cohere-transcribe-arabic-07-2026.yaml new file mode 100644 index 00000000..dd37765c --- /dev/null +++ b/scripts/hf_cards/cohere-transcribe-arabic-07-2026.yaml @@ -0,0 +1,67 @@ +# Spec for the HF README of handy-computer/cohere-transcribe-arabic-07-2026-gguf. +# Prose only; numbers and metadata come from catalog/.json. See README.md. + +pin_date: 2026-07-07 + +# Validation pin for the most recent upload. Updated on each release — +# older HF revisions carry whatever value was current at their upload time. +validation: + reference: Transformers + commit: d89ecb7 + date: 2026-07-07 + +pipeline_tag: automatic-speech-recognition +tags: + - gguf + - transcribe.cpp + - asr + - speech-to-text + - cohere + - conformer + - encoder-decoder + - arabic + +summary: | + Offline Arabic speech-to-text, including dialectal Arabic and + Arabic-English code-switching, with English as a secondary language. An + Arabic-focused adaptation of the Cohere Transcribe 03-2026 architecture: + a Conformer encoder with a Transformer encoder-decoder head + (cross-attention, tied token embedding). Takes a 16 kHz mono WAV and a + language flag (`-l ar` or `-l en`) and produces a transcript. Decoding + is autoregressive. + +usage: | + Build transcribe.cpp from source: + + ```bash + git clone git@github.com:handy-computer/transcribe.cpp.git + cd transcribe.cpp + cmake -B build && cmake --build build + ``` + + Run on a 16 kHz mono WAV: + + ```bash + build/bin/transcribe-cli \ + -m cohere-transcribe-arabic-07-2026-Q8_0.gguf \ + -l ar \ + input.wav + ``` + + If your audio isn't already 16 kHz mono WAV, convert it first: + + ```bash + ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav + ``` + + See the [transcribe.cpp model page](https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/cohere-transcribe-arabic-07-2026.md) for performance + numbers, numerical validation, and reproduction steps. + +wer: + notes: | + Greedy decoding, no external LM, scored with the Whisper BasicTextNormalizer; the + FLEURS Arabic split is `ar_eg`, Egyptian-dialect speech. BF16 reference baseline, + measured with native Transformers on the same manifest: 11.00%; the BF16 port scores + 11.02%, and every quant falls inside the reference's 95% confidence interval. FLEURS + Arabic is Egyptian-dialect speech; upstream numbers published on other Arabic test + sets are not directly comparable. diff --git a/scripts/hf_cards/diar_streaming_sortformer_4spk-v2.1.yaml b/scripts/hf_cards/diar_streaming_sortformer_4spk-v2.1.yaml index 61a9ab43..d9d6365b 100644 --- a/scripts/hf_cards/diar_streaming_sortformer_4spk-v2.1.yaml +++ b/scripts/hf_cards/diar_streaming_sortformer_4spk-v2.1.yaml @@ -1,14 +1,9 @@ # Spec for the HF README of handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. # # Diarizer family: metric is DER (not WER); pipeline_tag follows the # upstream NVIDIA card's diarization tagging rather than ASR. -hf_repo: nvidia/diar_streaming_sortformer_4spk-v2.1 -target_repo: handy-computer/diar_streaming_sortformer_4spk-v2.1-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/diar_streaming_sortformer_4spk-v2.1.md - -upstream_commit: fafaab5 pin_date: 2026-07-19 # Validation pin for the most recent upload. Updated on each release — @@ -18,11 +13,7 @@ validation: commit: d42c3bb date: 2026-07-22 -license: other -license_display: NVIDIA Open Model License pipeline_tag: voice-activity-detection -languages: - - en tags: - gguf - transcribe.cpp @@ -32,8 +23,6 @@ tags: - sortformer - fastconformer -metric: DER - summary: | Streaming speaker diarization: who spoke when, for up to 4 speakers. A FastConformer encoder with an 18-layer Transformer head emitting @@ -42,45 +31,46 @@ summary: | a run produces speaker segments (start, end, speaker id in arrival order), no text. Takes 16 kHz mono WAV. -default_quant_index: 2 # Q8_0 +usage: | + Build transcribe.cpp from source: + + ```bash + git clone git@github.com:handy-computer/transcribe.cpp.git + cd transcribe.cpp + cmake -B build && cmake --build build + ``` + + Run on a 16 kHz mono WAV. This is a diarizer, not a transcription model — + use the JSONL output to get the speaker segments (the plain output's + `text:` line is empty by design): -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: false - timestamps: none # segment times are diarization output, not transcript timestamps + ```bash + echo input.wav > files.txt + build/bin/transcribe-cli \ + -m diar_streaming_sortformer_4spk-v2.1-Q8_0.gguf \ + --batch files.txt --batch-jsonl + # {"file":"input.wav","text":"","speakers":[{"t0_ms":320,"t1_ms":2400,"speaker_id":1},...]} + ``` -# Speedup-over-realtime (×RT) per rig/backend, hand-authored from the bench; -# published raw as rtf_ in the metadata block. -perf: - m4: - cpu: 51 - metal: 110 + From the C API, read segments via `transcribe_n_speaker_segments` / + `transcribe_get_speaker_segment`, and select the streaming operating + point (latency / accuracy trade-off) with the run extension in + `include/transcribe/sortformer.h`. + + If your audio isn't already 16 kHz mono WAV, convert it first: + + ```bash + ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav + ``` + + See the [transcribe.cpp model page](https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/diar_streaming_sortformer_4spk-v2.1.md) for performance + numbers, numerical validation, and reproduction steps. wer: - source: AMI IHM test - metadata_key: ami_ihm_test notes: | - DER measured on the full AMI IHM test set (16 meetings, ~9 h) against - forced-alignment RTTMs with dihard3-dev post-processing, collar 0.0, - overlap scored, at the very_high_latency operating point. Measured NeMo - reference under the identical protocol: 14.83% DER / 19.89% JER; the - C++ F32 port scores 14.59% / 19.51%. Published DER numbers vary with - RTTM source and post-processing; compare like with like. Only - near-reference tiers ship for this family (k-quant tiers withdrawn; - see the transcribe.cpp family doc, "Quant policy (Stage 7)"). - -quants: - - name: F32 - filename: diar_streaming_sortformer_4spk-v2.1-F32.gguf - size: 471 MB - wer: 14.59% - - name: F16 - filename: diar_streaming_sortformer_4spk-v2.1-F16.gguf - size: 237 MB - wer: 14.23% - - name: Q8_0 - filename: diar_streaming_sortformer_4spk-v2.1-Q8_0.gguf - size: 139 MB - wer: 14.73% + Scored against forced-alignment RTTMs with dihard3-dev post-processing, collar 0.0, + overlap scored, at the very_high_latency operating point. Measured NeMo reference + under the identical protocol: 14.83% DER / 19.89% JER; the C++ F32 port scores + 14.59% / 19.51%. Published DER numbers vary with RTTM source and post-processing; + compare like with like. Only near-reference tiers ship for this family (k-quant + tiers withdrawn; see the transcribe.cpp family doc, "Quant policy (Stage 7)"). diff --git a/scripts/hf_cards/fun-asr-mlt-nano-2512.yaml b/scripts/hf_cards/fun-asr-mlt-nano-2512.yaml index b124dff4..6f1543b7 100644 --- a/scripts/hf_cards/fun-asr-mlt-nano-2512.yaml +++ b/scripts/hf_cards/fun-asr-mlt-nano-2512.yaml @@ -1,12 +1,6 @@ # Spec for the HF README of handy-computer/Fun-ASR-MLT-Nano-2512-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: FunAudioLLM/Fun-ASR-MLT-Nano-2512 -target_repo: handy-computer/Fun-ASR-MLT-Nano-2512-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/fun-asr-mlt-nano-2512.md - -# Pinned at intake (Stage 1). -upstream_commit: cf67a938bf2829959d08fdfb84e186eff02a67ff pin_date: 2026-05-06 # Validation pin for the most recent upload. Updated on each release — @@ -16,45 +10,7 @@ validation: commit: f094d28 date: 2026-05-06 -# Upstream license is "FunASR-Model-License-1.1" — not a standard SPDX id. -# The HF YAML field accepts a free-form `other` plus a license_name / -# license_link override on the upstream card; the renderer emits -# `license: other` and surfaces the human-facing form via `license_display`. -license: other -license_display: FunASR Model Open Source License Agreement v1.1 pipeline_tag: automatic-speech-recognition -languages: - - zh - - en - - yue - - ja - - ko - - vi - - id - - th - - ms - - tl - - ar - - hi - - bg - - hr - - cs - - da - - nl - - et - - fi - - el - - hu - - ga - - lv - - lt - - mt - - pl - - pt - - ro - - sk - - sl - - sv tags: - gguf - transcribe.cpp @@ -84,61 +40,12 @@ summary: | `--itn` CLI flag and `transcribe_funasr_nano_params { use_itn }` in the library API. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 68 - cpu: 16 - ryzen-4750u: - vulkan: 9 - cpu: 4.5 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy LLM decoding via the bundled Qwen3-0.6B head. The publisher - does not report a numerical LibriSpeech WER for the MLT variant - specifically (the shared README's per-model table covers Fun-ASR-Nano - only). Gate baseline is our own FunASR 1.3.1 reference run on the same - manifest: 1.76% (95% CI [1.60%, 1.93%]). transcribe.cpp's BF16 port - matches that baseline within -0.02 percentage-points. LibriSpeech is - English only; the strength of the MLT variant is multilingual coverage, - not English accuracy. For the other 30 languages, run your own - representative manifest. - -quants: - - name: BF16 - filename: Fun-ASR-MLT-Nano-2512-BF16.gguf - size: 1590 MB - wer: 1.74% - - name: F16 - filename: Fun-ASR-MLT-Nano-2512-F16.gguf - size: 1590 MB - wer: 1.74% - - name: Q8_0 - filename: Fun-ASR-MLT-Nano-2512-Q8_0.gguf - size: 850 MB - wer: 1.74% - - name: Q6_K - filename: Fun-ASR-MLT-Nano-2512-Q6_K.gguf - size: 659 MB - wer: 1.69% - - name: Q5_K_M - filename: Fun-ASR-MLT-Nano-2512-Q5_K_M.gguf - size: 602 MB - wer: 1.77% - - name: Q4_K_M - filename: Fun-ASR-MLT-Nano-2512-Q4_K_M.gguf - size: 531 MB - wer: 1.89% + Greedy LLM decoding via the bundled Qwen3-0.6B head. The publisher does not report a + numerical LibriSpeech WER for the MLT variant specifically (the shared README's + per-model table covers Fun-ASR-Nano only). Gate baseline is our own FunASR 1.3.1 + reference run on the same manifest: 1.76% (95% CI [1.60%, 1.93%]). transcribe.cpp's + BF16 port matches that baseline within -0.02 percentage-points. LibriSpeech is + English only; the strength of the MLT variant is multilingual coverage, not English + accuracy. For the other 30 languages, run your own representative manifest. diff --git a/scripts/hf_cards/fun-asr-nano-2512.yaml b/scripts/hf_cards/fun-asr-nano-2512.yaml index 7e512abd..03f20313 100644 --- a/scripts/hf_cards/fun-asr-nano-2512.yaml +++ b/scripts/hf_cards/fun-asr-nano-2512.yaml @@ -1,12 +1,6 @@ # Spec for the HF README of handy-computer/Fun-ASR-Nano-2512-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: FunAudioLLM/Fun-ASR-Nano-2512 -target_repo: handy-computer/Fun-ASR-Nano-2512-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/fun-asr-nano-2512.md - -# Pinned at intake (Stage 1). -upstream_commit: a7088d620f755dcdca575b63db184c3ad55b2865 pin_date: 2026-05-06 # Validation pin for the most recent upload. Updated on each release — @@ -16,17 +10,7 @@ validation: commit: f094d28 date: 2026-05-06 -# Upstream license is "FunASR-Model-License-1.1" — not a standard SPDX id. -# The HF YAML field accepts a free-form `other` plus a license_name / -# license_link override on the upstream card; the renderer emits -# `license: other` and surfaces the human-facing form via `license_display`. -license: other -license_display: FunASR Model Open Source License Agreement v1.1 pipeline_tag: automatic-speech-recognition -languages: - - zh - - en - - ja tags: - gguf - transcribe.cpp @@ -50,60 +34,11 @@ summary: | via the `--itn` CLI flag and `transcribe_funasr_nano_params { use_itn }` in the library API. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 77.5 - cpu: 27 - ryzen-4750u: - vulkan: 10.5 - cpu: 6.5 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy LLM decoding via the bundled Qwen3-0.6B head. Publisher - reports 1.76% on this split (model card "Open-Source Dataset - Performance" table). Our FunASR 1.3.1 reference run scores 1.79% - (95% CI [1.63%, 1.95%]), within bootstrap noise of the publisher's - number. transcribe.cpp's BF16 port matches that baseline within - -0.01 percentage-points. LibriSpeech is an English-only benchmark; - Chinese (AISHELL-1, WenetSpeech) and Japanese (CommonVoice JA) are - the recommended complementary checks. - -quants: - - name: BF16 - filename: Fun-ASR-Nano-2512-BF16.gguf - size: 1590 MB - wer: 1.78% - - name: F16 - filename: Fun-ASR-Nano-2512-F16.gguf - size: 1590 MB - wer: 1.79% - - name: Q8_0 - filename: Fun-ASR-Nano-2512-Q8_0.gguf - size: 850 MB - wer: 1.79% - - name: Q6_K - filename: Fun-ASR-Nano-2512-Q6_K.gguf - size: 659 MB - wer: 1.78% - - name: Q5_K_M - filename: Fun-ASR-Nano-2512-Q5_K_M.gguf - size: 602 MB - wer: 1.82% - - name: Q4_K_M - filename: Fun-ASR-Nano-2512-Q4_K_M.gguf - size: 531 MB - wer: 1.92% + Greedy LLM decoding via the bundled Qwen3-0.6B head. Publisher reports 1.76% on this + split (model card "Open-Source Dataset Performance" table). Our FunASR 1.3.1 + reference run scores 1.79% (95% CI [1.63%, 1.95%]), within bootstrap noise of the + publisher's number. transcribe.cpp's BF16 port matches that baseline within -0.01 + percentage-points. LibriSpeech is an English-only benchmark; Chinese (AISHELL-1, + WenetSpeech) and Japanese (CommonVoice JA) are the recommended complementary checks. diff --git a/scripts/hf_cards/generate.py b/scripts/hf_cards/generate.py index 5a027fc0..1741ac06 100755 --- a/scripts/hf_cards/generate.py +++ b/scripts/hf_cards/generate.py @@ -9,11 +9,20 @@ # /// """Generate the HuggingFace README.md for a transcribe.cpp GGUF repo. -Reads a YAML spec (see parakeet-tdt-0.6b-v2.yaml for an example), fetches the -upstream model card at the pinned commit, and renders template.md.j2. +Two inputs, disjoint by construction: -Default output is models//README.md alongside the GGUFs, so -`hf upload models/ .` picks it up in the same call. + catalog/.json identity, repos, licence, languages, + capabilities, downloads, benchmark numbers + scripts/hf_cards/.yaml editorial copy and release state only: + summary, tags, pipeline tag, validation pin, + prose notes, optional usage override + +Nothing numeric or mechanical is read from the YAML; a number that belongs +on the card belongs in the catalog first. Fetches the upstream model card at +the pinned commit and renders template.md.j2. + +Default output is models//README.md alongside the GGUFs, so +`hf upload models/ .` picks it up in the same call. Usage: uv run scripts/hf_cards/generate.py scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml @@ -24,6 +33,8 @@ from __future__ import annotations import argparse +import re +import statistics import sys from pathlib import Path @@ -33,45 +44,228 @@ HERE = Path(__file__).parent REPO_ROOT = HERE.parent.parent +sys.path.insert(0, str(REPO_ROOT / "scripts" / "catalog")) +import common # noqa: E402 + +# The editorial surface of a card spec, and the whole of it. Anything else is +# either owned by catalog/.json or a typo; both are refused, so the +# two can never quietly diverge again and a misspelled key cannot silently +# render an empty section. Adding a field here is a deliberate act. +SPEC_KEYS = { + "pin_date", # date the upstream revision was pinned + "validation", # reference framework, transcribe.cpp commit, date + "pipeline_tag", # HF Hub pipeline tag + "tags", # HF Hub tags + "summary", # the card's opening paragraph + "wer", # editorial caveats; see README.md + "usage", # extra usage prose for an unusual model + "upstream_card_commit", # revision of the upstream card that was quoted + "default_quant", # override the Q8_0 default +} +CAP_FLAGS = ("streaming", "translate", "lang_detect") +DEFAULT_QUANT = "Q8_0" def load_spec(path: Path) -> dict: + """The editorial half of a card. Fails on any catalog-owned key.""" with path.open() as f: - return yaml.safe_load(f) + spec = yaml.safe_load(f) or {} + unknown = sorted(spec.keys() - SPEC_KEYS) + if unknown: + raise SystemExit( + f"{path.name}: {', '.join(unknown)} is not an editorial field. It is " + f"either derived from catalog/{path.stem}.json or misspelled; remove " + f"it. Editorial fields: {', '.join(sorted(SPEC_KEYS))}") + if "default_quant_index" in spec: + raise SystemExit(f"{path.name}: default_quant_index is gone; the default is " + f"{DEFAULT_QUANT}, override with default_quant: ") + return spec + + +# -------------------------------------------------------------------------- +# catalog -> card context + + +def derive_capabilities(record: dict) -> dict: + """The boolean flags the `transcribe_cpp:` metadata block carries.""" + caps = record.get("capabilities", {}) + out = {flag: bool(caps.get(flag, {}).get("supported")) for flag in CAP_FLAGS} + if caps.get("diarize", {}).get("supported"): + out["diarize"] = True + granularities = caps.get("timestamps", {}).get("granularities") or [] + # Advertise the finest granularity the port actually emits. + out["timestamps"] = next((g for g in ("token", "word", "segment") + if g in granularities), "none") + return out + + +def derive_perf(record: dict, default_quant: str | None) -> dict: + """Speedup over realtime per rig/backend at the card's default quant, + averaged over the benchmark samples.""" + cells: dict[tuple[str, str], list[float]] = {} + for row in record.get("speed_benchmarks", []): + if row["quant"] != default_quant: + continue + cells.setdefault((row["machine"], row["backend"]), []).append(row["xrt_compute"]) + perf: dict[str, dict[str, float]] = {} + for (machine, backend), values in sorted(cells.items()): + perf.setdefault(machine, {})[backend] = round(statistics.fmean(values), 2) + return perf + + +def metric_key(row: dict) -> str: + """`__[_][_]`, the name of + the per-quant map this row belongs to in the metadata block.""" + key = f"{row['metric']}_{row['dataset']}_{common.dataset_tail(row)}" + for extra in ("scoring", "mode"): + if row.get(extra): + key += f"_{row[extra]}" + return re.sub(r"[^a-z0-9]+", "_", key.lower()) + + +def derive_metric_blocks(record: dict) -> dict[str, dict[str, float]]: + """Every per-quant error map the catalog holds, keyed by metric_key. + Where a cell was measured under several recipes (batch size, timestamps) + the profile-stamped row wins, else the first listed.""" + chosen: dict[tuple[str, str], dict] = {} + for row in record.get("accuracy_benchmarks", []): + cell = (metric_key(row), row["quant"].lower()) + if cell not in chosen or (row.get("engine_sha") and not chosen[cell].get("engine_sha")): + chosen[cell] = row + blocks: dict[str, dict[str, float]] = {} + for (key, quant), row in chosen.items(): + blocks.setdefault(key, {})[quant] = row["err_pct"] + return blocks + + +def derive_quants(record: dict, secondary: dict | None) -> list[dict]: + """One row per published GGUF: size, headline error rate, and the optional + second metric column named by `wer.secondary`.""" + errors = common.headline_rows(record) + quants = [] + for item in record.get("downloads", []): + entry = {"name": item["quant"], "filename": item["filename"], + "size": common.fmt_size(item["size_bytes"])} + row = errors.get(item["quant"]) + if row is not None: + entry["wer"] = common.fmt_err(row) + if secondary is not None: + value = secondary.get(item["quant"].lower()) + if value is not None: + entry["wer2"] = f"{float(value):.2f}%" + quants.append(entry) + return quants + -def build_transcribe_cpp_block(spec: dict) -> str: - """Serialize the `transcribe_cpp:` block (raw WER/RTF + capability flags). +def hub_language_tags(languages) -> tuple[list[str], list[str]]: + """Split catalog language tags the way the Hub's metadata validator wants. - See docs/tools/hf-metadata-schema.md. Returns "" when a spec omits `perf`, - opting out of the block. + `language:` accepts ISO 639 codes only, so a model whose GGUF advertises + locales (nemotron tags `en-US`) keeps the full tags in `language_bcp47` + and contributes each primary subtag, deduped, to `language`. """ - if "perf" not in spec: - return "" + tags = [str(lang) for lang in languages] + base, seen = [], set() + for tag in tags: + primary = tag.split("-")[0].lower() + if primary not in seen: + seen.add(primary) + base.append(primary) + return base, [tag for tag in tags if "-" in tag] + +DOCS_BASE = "https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models" + + +def docs_url(record: dict) -> str: + """The model's page on GitHub: its own, or the family page it shares.""" + page = f"{record['variant']}.md" + if not (common.DOCS_DIR / page).exists(): + page = record.get("docs_page") or page + return f"{DOCS_BASE}/{page}" - caps = spec.get("capabilities", {}) - wer = spec["wer"] - dataset_key = wer.get("metadata_key", "librispeech_test_clean") - block: dict = {} - # Headline dataset: per-quant WER taken from the `quants:` column. - block[f"wer_{dataset_key}"] = { - q["name"].lower(): float(str(q["wer"]).rstrip("%")) for q in spec["quants"] + +def build_context(record: dict, spec: dict) -> dict: + """Everything the template needs: catalog facts plus the editorial spec.""" + downloads = {item["quant"]: item for item in record.get("downloads", [])} + default_quant = spec.get("default_quant", DEFAULT_QUANT) + if default_quant not in downloads: + raise SystemExit(f"{record['variant']}: default quant {default_quant!r} is not " + f"a published download ({', '.join(downloads) or 'none'})") + wer = dict(spec.get("wer") or {}) + stale = [k for k, v in wer.items() if isinstance(v, dict)] + \ + [k for k in ("metadata_key", "metadata_key2") if k in wer] + if stale or "metrics" in spec: + raise SystemExit(f"{record['variant']}: per-quant numbers and metadata keys are " + f"derived from the catalog; remove wer.{'/'.join(stale)}" + + (" and metrics" if "metrics" in spec else "")) + if not wer.get("source"): + wer["source"] = common.headline_label(record) + elif wer["source"] == common.headline_label(record): + raise SystemExit(f"{record['variant']}: wer.source restates the catalog's " + f"headline label; remove it") + wer["recipe"] = common.headline_recipe(record) + blocks = derive_metric_blocks(record) + secondary = None + if "source2" in wer: + key2 = wer.get("secondary") + if key2 not in blocks: + raise SystemExit(f"{record['variant']}: wer.secondary must name one of " + f"{sorted(blocks)}") + secondary = blocks[key2] + headline = common.headline(record) or {} + hub_languages, languages_bcp47 = hub_language_tags(record.get("languages", [])) + ctx = { + **spec, + "hf_repo": record["upstream_repo"], + "target_repo": record.get("published_repo"), + "upstream_commit": record["upstream_commit"], + "license": record["license"]["spdx"], + "license_display": record["license"]["display"], + "languages": hub_languages, + "languages_bcp47": languages_bcp47, + "capabilities": derive_capabilities(record), + "perf": derive_perf(record, default_quant), + "quants": derive_quants(record, secondary), + "default_quant_filename": downloads[default_quant]["filename"], + "wer": wer, + "metric_blocks": blocks, + "transcribe_docs_url": docs_url(record), } - # Any additional per-quant WER maps listed inline under `wer:` (keyed by - # dataset name, e.g. `librispeech_test_clean:`) are emitted as their own - # `wer_` blocks. Only dict values count as datasets; scalar keys - # (metadata_key, source, notes) are skipped. - for key, per_quant in wer.items(): - if isinstance(per_quant, dict): - block[f"wer_{key}"] = { - str(q).lower(): float(str(v).rstrip("%")) for q, v in per_quant.items() - } - for machine, backends in spec["perf"].items(): + if headline.get("metric"): + ctx["metric"] = headline["metric"].upper() + for key in ("name", "link"): + if record["license"].get(key): + ctx[f"license_{key}"] = record["license"][key] + if not ctx["target_repo"]: + raise SystemExit(f"{record['variant']}: catalog has no published_repo") + return ctx + + +# -------------------------------------------------------------------------- +# rendering + + +def build_transcribe_cpp_block(ctx: dict) -> str: + """Serialize the `transcribe_cpp:` block (raw error rates, RTF, and + capability flags), entirely from the catalog record. + + See docs/tools/hf-metadata-schema.md. Returns "" when the catalog holds no + speed rows for the default quant, opting out of the block. + """ + if not ctx["perf"]: + return "" + + caps = ctx["capabilities"] + # Bumped when key names or shapes change. 2: every result set the catalog + # holds is emitted, keyed __[_scoring][_mode]; + # earlier cards emitted a hand-named headline map and up to one extra. + block: dict = {"schema_version": 2} + # Every per-quant error map the catalog holds, headline first. + for key, per_quant in ctx["metric_blocks"].items(): + block[key] = dict(per_quant) + for machine, backends in ctx["perf"].items(): block[f"rtf_{machine.replace('-', '_')}"] = backends - # Optional non-WER task metrics (for example cpWER for - # speaker-attributed ASR). Values are emitted verbatim so the spec keeps - # the metric's natural shape and units. - block.update(spec.get("metrics", {})) block["streaming"] = bool(caps.get("streaming", False)) if "diarize" in caps: block["diarize"] = bool(caps["diarize"]) @@ -99,7 +293,7 @@ def fetch_upstream_card(repo_id: str, revision: str) -> str: return content.strip() -def render(spec: dict, upstream_card: str) -> str: +def render(ctx: dict, upstream_card: str) -> str: env = Environment( loader=FileSystemLoader(HERE), undefined=StrictUndefined, @@ -108,19 +302,19 @@ def render(spec: dict, upstream_card: str) -> str: template = env.get_template("template.md.j2") return template.render( upstream_card=upstream_card, - transcribe_cpp_yaml=build_transcribe_cpp_block(spec), - **spec, + transcribe_cpp_yaml=build_transcribe_cpp_block(ctx), + **ctx, ) def main() -> int: ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - ap.add_argument("spec", type=Path, help="Path to the YAML spec file") + ap.add_argument("spec", type=Path, help="Path to the editorial YAML spec") ap.add_argument( "-o", "--output", type=Path, - help="Write to this path. Defaults to models//README.md.", + help="Write to this path. Defaults to models//README.md.", ) ap.add_argument( "--stdout", @@ -135,18 +329,20 @@ def main() -> int: args = ap.parse_args() spec = load_spec(args.spec) + record = common.load_record(args.spec.stem) + ctx = build_context(record, spec) # Most families pin the upstream card to the same SHA as the ported # weights. Multi-branch upstream repos (gigaam) ship the family card # only on `main` while per-variant branches have empty README stubs; # `upstream_card_commit` lets a spec point the card-fetch at a - # different revision than `upstream_commit`. - card_commit = spec.get("upstream_card_commit", spec["upstream_commit"]) + # different revision than the catalog's upstream_commit. + card_commit = spec.get("upstream_card_commit", ctx["upstream_commit"]) upstream = ( "_(upstream card not fetched — run without --skip-upstream to include it)_" if args.skip_upstream - else fetch_upstream_card(spec["hf_repo"], card_commit) + else fetch_upstream_card(ctx["hf_repo"], card_commit) ) - out = render(spec, upstream) + out = render(ctx, upstream) if args.stdout: sys.stdout.write(out) @@ -157,7 +353,7 @@ def main() -> int: # directory `hf upload` will publish. The kebab-cased spec stem is # the internal handle; the filesystem dir mirrors upstream casing # (matches the converter's output dir convention). - upstream_slug = spec["hf_repo"].rsplit("/", 1)[-1] + upstream_slug = ctx["hf_repo"].rsplit("/", 1)[-1] output = args.output or (REPO_ROOT / "models" / upstream_slug / "README.md") output.parent.mkdir(parents=True, exist_ok=True) output.write_text(out) diff --git a/scripts/hf_cards/gigaam-v3-ctc.yaml b/scripts/hf_cards/gigaam-v3-ctc.yaml index 178cbd00..5f0d34f4 100644 --- a/scripts/hf_cards/gigaam-v3-ctc.yaml +++ b/scripts/hf_cards/gigaam-v3-ctc.yaml @@ -1,13 +1,6 @@ # Spec for the HF README of handy-computer/gigaam-v3-ctc-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ai-sage/GigaAM-v3 -target_repo: handy-computer/gigaam-v3-ctc-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/gigaam-v3-ctc.md - -# Upstream branch: ctc. Each gigaam-v3 variant lives on its own -# HF branch; the SHA below pins this variant's branch revision. -upstream_commit: 15ef3b5 pin_date: 2026-05-12 # Upstream README lives only on the main branch (per-variant branches # ship empty README stubs); fetch the family card from main. @@ -20,11 +13,7 @@ validation: commit: 42b96d9 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ru tags: - gguf - transcribe.cpp @@ -38,53 +27,11 @@ tags: summary: | Offline Russian speech-to-text with greedy CTC decoding. 16-layer Conformer encoder with a 1×1 Conv1d CTC head. Output is lowercased Russian, no punctuation; 33-entry character vocabulary. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 146 - cpu: 29 - ryzen-4750u: - vulkan: 33 - cpu: 9 - wer: - metadata_key: fleurs_ru - source: FLEURS ru notes: | - WER measured on the full FLEURS ru test split (775 utterances) with greedy decoding and no external LM. F32 reference baseline: 8.42%. Upstream `gigaam` author package measured on the same manifest: 9.81%; the 1.4 pp gap is upstream rejecting 5 long (>25 s) utterances with `Too long wav file, use 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not publish a FLEURS ru WER; this number is measured here. - -quants: - - name: F32 - filename: gigaam-v3-ctc-F32.gguf - size: 842 MB - wer: 8.42% - - name: F16 - filename: gigaam-v3-ctc-F16.gguf - size: 428 MB - wer: 8.42% - - name: Q8_0 - filename: gigaam-v3-ctc-Q8_0.gguf - size: 259 MB - wer: 8.40% - - name: Q6_K - filename: gigaam-v3-ctc-Q6_K.gguf - size: 216 MB - wer: 8.38% - - name: Q5_K_M - filename: gigaam-v3-ctc-Q5_K_M.gguf - size: 195 MB - wer: 8.29% - - name: Q4_K_M - filename: gigaam-v3-ctc-Q4_K_M.gguf - size: 174 MB - wer: 8.42% + Greedy decoding, no external LM. F32 reference baseline: 8.42%. Upstream `gigaam` + author package measured on the same manifest: 9.81%; the 1.4 pp gap is upstream + rejecting 5 long (>25 s) utterances with `Too long wav file, use + 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt + subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not + publish a FLEURS ru WER; this number is measured here. diff --git a/scripts/hf_cards/gigaam-v3-e2e-ctc.yaml b/scripts/hf_cards/gigaam-v3-e2e-ctc.yaml index 0e4d9d0c..36c755f0 100644 --- a/scripts/hf_cards/gigaam-v3-e2e-ctc.yaml +++ b/scripts/hf_cards/gigaam-v3-e2e-ctc.yaml @@ -1,13 +1,6 @@ # Spec for the HF README of handy-computer/gigaam-v3-e2e-ctc-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ai-sage/GigaAM-v3 -target_repo: handy-computer/gigaam-v3-e2e-ctc-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/gigaam-v3-e2e-ctc.md - -# Upstream branch: e2e_ctc. Each gigaam-v3 variant lives on its own -# HF branch; the SHA below pins this variant's branch revision. -upstream_commit: cec030b pin_date: 2026-05-12 # Upstream README lives only on the main branch (per-variant branches # ship empty README stubs); fetch the family card from main. @@ -20,11 +13,7 @@ validation: commit: 42b96d9 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ru tags: - gguf - transcribe.cpp @@ -38,53 +27,11 @@ tags: summary: | Offline Russian speech-to-text with greedy CTC decoding. 16-layer Conformer encoder with a 1×1 Conv1d CTC head. Output is cased Russian with punctuation, decoded from a 256-piece SentencePiece tokenizer. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 112 - cpu: 27 - ryzen-4750u: - vulkan: 30 - cpu: 9 - wer: - metadata_key: fleurs_ru - source: FLEURS ru notes: | - WER measured on the full FLEURS ru test split (775 utterances) with greedy decoding and no external LM. F32 reference baseline: 5.50%. Upstream `gigaam` author package measured on the same manifest: 6.93%; the 1.4 pp gap is upstream rejecting 5 long (>25 s) utterances with `Too long wav file, use 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not publish a FLEURS ru WER; this number is measured here. - -quants: - - name: F32 - filename: gigaam-v3-e2e-ctc-F32.gguf - size: 843 MB - wer: 5.50% - - name: F16 - filename: gigaam-v3-e2e-ctc-F16.gguf - size: 428 MB - wer: 5.50% - - name: Q8_0 - filename: gigaam-v3-e2e-ctc-Q8_0.gguf - size: 260 MB - wer: 5.50% - - name: Q6_K - filename: gigaam-v3-e2e-ctc-Q6_K.gguf - size: 216 MB - wer: 5.56% - - name: Q5_K_M - filename: gigaam-v3-e2e-ctc-Q5_K_M.gguf - size: 195 MB - wer: 5.58% - - name: Q4_K_M - filename: gigaam-v3-e2e-ctc-Q4_K_M.gguf - size: 174 MB - wer: 5.57% + Greedy decoding, no external LM. F32 reference baseline: 5.50%. Upstream `gigaam` + author package measured on the same manifest: 6.93%; the 1.4 pp gap is upstream + rejecting 5 long (>25 s) utterances with `Too long wav file, use + 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt + subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not + publish a FLEURS ru WER; this number is measured here. diff --git a/scripts/hf_cards/gigaam-v3-e2e-rnnt.yaml b/scripts/hf_cards/gigaam-v3-e2e-rnnt.yaml index 63d5dafc..ad059e02 100644 --- a/scripts/hf_cards/gigaam-v3-e2e-rnnt.yaml +++ b/scripts/hf_cards/gigaam-v3-e2e-rnnt.yaml @@ -1,13 +1,6 @@ # Spec for the HF README of handy-computer/gigaam-v3-e2e-rnnt-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ai-sage/GigaAM-v3 -target_repo: handy-computer/gigaam-v3-e2e-rnnt-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/gigaam-v3-e2e-rnnt.md - -# Upstream branch: main (= v3_e2e_rnnt). Each gigaam-v3 variant lives on its own -# HF branch; the SHA below pins this variant's branch revision. -upstream_commit: ec1dc1f pin_date: 2026-05-12 # Upstream README lives only on the main branch; this variant IS main, so # upstream_card_commit matches upstream_commit. Made explicit for symmetry @@ -21,11 +14,7 @@ validation: commit: 42b96d9 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ru tags: - gguf - transcribe.cpp @@ -39,53 +28,11 @@ tags: summary: | Offline Russian speech-to-text with greedy RNN-T decoding. 16-layer Conformer encoder paired with an RNN-T transducer head. Output is cased Russian with punctuation, decoded from a 1024-piece SentencePiece tokenizer. Not a streaming model and does not translate. Short-form only (≤25 s per utterance). -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 88 - cpu: 25 - ryzen-4750u: - vulkan: 22 - cpu: 8 - wer: - metadata_key: fleurs_ru - source: FLEURS ru notes: | - WER measured on the full FLEURS ru test split (775 utterances) with greedy decoding and no external LM. F32 reference baseline: 5.35%. Upstream `gigaam` author package measured on the same manifest: 6.78%; the 1.4 pp gap is upstream rejecting 5 long (>25 s) utterances with `Too long wav file, use 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not publish a FLEURS ru WER; this number is measured here. - -quants: - - name: F32 - filename: gigaam-v3-e2e-rnnt-F32.gguf - size: 849 MB - wer: 5.35% - - name: F16 - filename: gigaam-v3-e2e-rnnt-F16.gguf - size: 431 MB - wer: 5.35% - - name: Q8_0 - filename: gigaam-v3-e2e-rnnt-Q8_0.gguf - size: 261 MB - wer: 5.36% - - name: Q6_K - filename: gigaam-v3-e2e-rnnt-Q6_K.gguf - size: 217 MB - wer: 5.37% - - name: Q5_K_M - filename: gigaam-v3-e2e-rnnt-Q5_K_M.gguf - size: 197 MB - wer: 5.42% - - name: Q4_K_M - filename: gigaam-v3-e2e-rnnt-Q4_K_M.gguf - size: 175 MB - wer: 5.36% + Greedy decoding, no external LM. F32 reference baseline: 5.35%. Upstream `gigaam` + author package measured on the same manifest: 6.78%; the 1.4 pp gap is upstream + rejecting 5 long (>25 s) utterances with `Too long wav file, use + 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt + subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not + publish a FLEURS ru WER; this number is measured here. diff --git a/scripts/hf_cards/gigaam-v3-rnnt.yaml b/scripts/hf_cards/gigaam-v3-rnnt.yaml index 0adca61d..afe84616 100644 --- a/scripts/hf_cards/gigaam-v3-rnnt.yaml +++ b/scripts/hf_cards/gigaam-v3-rnnt.yaml @@ -1,13 +1,6 @@ # Spec for the HF README of handy-computer/gigaam-v3-rnnt-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ai-sage/GigaAM-v3 -target_repo: handy-computer/gigaam-v3-rnnt-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/gigaam-v3-rnnt.md - -# Upstream branch: rnnt. Each gigaam-v3 variant lives on its own -# HF branch; the SHA below pins this variant's branch revision. -upstream_commit: c7f128b pin_date: 2026-05-12 # Upstream README lives only on the main branch (per-variant branches # ship empty README stubs); fetch the family card from main. @@ -20,11 +13,7 @@ validation: commit: 42b96d9 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ru tags: - gguf - transcribe.cpp @@ -38,53 +27,11 @@ tags: summary: | Offline Russian speech-to-text with greedy RNN-T decoding. Same 16-layer Conformer encoder as the e2e variant, fine-tuned to emit lowercased Russian with no punctuation; 33-entry character vocabulary. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 110 - cpu: 27 - ryzen-4750u: - vulkan: 25 - cpu: 9 - wer: - metadata_key: fleurs_ru - source: FLEURS ru notes: | - WER measured on the full FLEURS ru test split (775 utterances) with greedy decoding and no external LM. F32 reference baseline: 8.08%. Upstream `gigaam` author package measured on the same manifest: 9.46%; the 1.4 pp gap is upstream rejecting 5 long (>25 s) utterances with `Too long wav file, use 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not publish a FLEURS ru WER; this number is measured here. - -quants: - - name: F32 - filename: gigaam-v3-rnnt-F32.gguf - size: 846 MB - wer: 8.08% - - name: F16 - filename: gigaam-v3-rnnt-F16.gguf - size: 430 MB - wer: 8.08% - - name: Q8_0 - filename: gigaam-v3-rnnt-Q8_0.gguf - size: 260 MB - wer: 8.08% - - name: Q6_K - filename: gigaam-v3-rnnt-Q6_K.gguf - size: 217 MB - wer: 8.07% - - name: Q5_K_M - filename: gigaam-v3-rnnt-Q5_K_M.gguf - size: 196 MB - wer: 8.12% - - name: Q4_K_M - filename: gigaam-v3-rnnt-Q4_K_M.gguf - size: 175 MB - wer: 8.12% + Greedy decoding, no external LM. F32 reference baseline: 8.08%. Upstream `gigaam` + author package measured on the same manifest: 9.46%; the 1.4 pp gap is upstream + rejecting 5 long (>25 s) utterances with `Too long wav file, use + 'transcribe_longform' method.` (counted as 100% deletion errors). On the 770-utt + subset both sides decode, transcribe.cpp matches upstream exactly. ai-sage does not + publish a FLEURS ru WER; this number is measured here. diff --git a/scripts/hf_cards/granite-4.0-1b-speech.yaml b/scripts/hf_cards/granite-4.0-1b-speech.yaml index 24b3ac96..8d2c4eee 100644 --- a/scripts/hf_cards/granite-4.0-1b-speech.yaml +++ b/scripts/hf_cards/granite-4.0-1b-speech.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/granite-4.0-1b-speech-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ibm-granite/granite-4.0-1b-speech -target_repo: handy-computer/granite-4.0-1b-speech-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/granite-4.0-1b-speech.md - -upstream_commit: bd87ab8 pin_date: 2026-05-17 validation: @@ -13,16 +8,7 @@ validation: commit: 275332d date: 2026-05-17 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - de - - es - - pt - - ja tags: - gguf - transcribe.cpp @@ -43,59 +29,9 @@ summary: | five other languages in either direction (en ↔ fr, en ↔ de, en ↔ es, en ↔ pt, en ↔ ja) — always via English, no direct fr↔de etc. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 36.5 - cpu: 6 - m4: - metal: 11 - cpu: 5 - ryzen-4750u: - vulkan: 2.95 - cpu: 1.6 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding. BF16 reference baseline (re-run locally with the - model card's exact prompt): 1.42% — matches the upstream Open ASR - Leaderboard number exactly. Text normalizer: Whisper - `EnglishTextNormalizer`, the same normalizer Open ASR Leaderboard uses. - -quants: - - name: BF16 - filename: granite-4.0-1b-speech-BF16.gguf - size: 4.63 GB - wer: 1.42% - - name: F16 - filename: granite-4.0-1b-speech-F16.gguf - size: 4.63 GB - wer: 1.42% - - name: Q8_0 - filename: granite-4.0-1b-speech-Q8_0.gguf - size: 2.56 GB - wer: 1.44% - - name: Q6_K - filename: granite-4.0-1b-speech-Q6_K.gguf - size: 2.02 GB - wer: 1.41% - - name: Q5_K_M - filename: granite-4.0-1b-speech-Q5_K_M.gguf - size: 1.83 GB - wer: 1.42% - - name: Q4_K_M - filename: granite-4.0-1b-speech-Q4_K_M.gguf - size: 1.60 GB - wer: 1.48% + Greedy decoding. BF16 reference baseline (re-run locally with the model card's exact + prompt): 1.42% — matches the upstream Open ASR Leaderboard number exactly. Text + normalizer: Whisper `EnglishTextNormalizer`, the same normalizer Open ASR + Leaderboard uses. diff --git a/scripts/hf_cards/granite-speech-4.1-2b-nar.yaml b/scripts/hf_cards/granite-speech-4.1-2b-nar.yaml index d624036f..2d6d801b 100644 --- a/scripts/hf_cards/granite-speech-4.1-2b-nar.yaml +++ b/scripts/hf_cards/granite-speech-4.1-2b-nar.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/granite-speech-4.1-2b-nar-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ibm-granite/granite-speech-4.1-2b-nar -target_repo: handy-computer/granite-speech-4.1-2b-nar-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/granite-speech-4.1-2b-nar.md - -upstream_commit: 99a4df9 pin_date: 2026-05-24 validation: @@ -13,15 +8,7 @@ validation: commit: c53af2c date: 2026-05-24 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - de - - es - - pt tags: - gguf - transcribe.cpp @@ -44,66 +31,14 @@ summary: | German, Spanish, and Portuguese; ASR only (no translation, no timestamps). -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 56 - cpu: 5 - m4: - metal: 18 - cpu: 4 - ryzen-4750u: - vulkan: 3.65 - cpu: 1.5 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances). - BF16 reference baseline (transformers `model.transcribe`, MPS, re-run - locally): 1.28% — matches the upstream model card's 1.29% to within - sampling noise. Text normalizer: Whisper `EnglishTextNormalizer`, the - same normalizer Open ASR Leaderboard uses. Reference reproduction - follows the model card path verbatim (`AutoProcessor` + - `AutoModel.transcribe` + `processor.batch_decode`) at HF revision - `99a4df9` (single-file `modeling_granite_speech_nar.py` snapshot, the - README's canonical target); no mask patching is required because the - NAR LM uses `create_bidirectional_mask()` natively. F16, Q8_0, and - Q6_K all match BF16's 1.29%; Q5_K_M dips slightly to 1.25% (within - overlapping CIs). - -quants: - - name: BF16 - filename: granite-speech-4.1-2b-nar-BF16.gguf - size: 4.20 GB - wer: 1.29% - - name: F16 - filename: granite-speech-4.1-2b-nar-F16.gguf - size: 4.21 GB - wer: 1.29% - - name: Q8_0 - filename: granite-speech-4.1-2b-nar-Q8_0.gguf - size: 2.33 GB - wer: 1.29% - - name: Q6_K - filename: granite-speech-4.1-2b-nar-Q6_K.gguf - size: 1.84 GB - wer: 1.29% - - name: Q5_K_M - filename: granite-speech-4.1-2b-nar-Q5_K_M.gguf - size: 1.66 GB - wer: 1.25% - - name: Q4_K_M - filename: granite-speech-4.1-2b-nar-Q4_K_M.gguf - size: 1.45 GB - wer: 1.35% + BF16 reference baseline (transformers `model.transcribe`, MPS, re-run locally): + 1.28% — matches the upstream model card's 1.29% to within sampling noise. Text + normalizer: Whisper `EnglishTextNormalizer`, the same normalizer Open ASR + Leaderboard uses. Reference reproduction follows the model card path verbatim + (`AutoProcessor` + `AutoModel.transcribe` + `processor.batch_decode`) at HF revision + `99a4df9` (single-file `modeling_granite_speech_nar.py` snapshot, the README's + canonical target); no mask patching is required because the NAR LM uses + `create_bidirectional_mask()` natively. F16, Q8_0, and Q6_K all match BF16's 1.29%; + Q5_K_M dips slightly to 1.25% (within overlapping CIs). diff --git a/scripts/hf_cards/granite-speech-4.1-2b-plus.yaml b/scripts/hf_cards/granite-speech-4.1-2b-plus.yaml index e37d0d03..3f7b143e 100644 --- a/scripts/hf_cards/granite-speech-4.1-2b-plus.yaml +++ b/scripts/hf_cards/granite-speech-4.1-2b-plus.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/granite-speech-4.1-2b-plus-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ibm-granite/granite-speech-4.1-2b-plus -target_repo: handy-computer/granite-speech-4.1-2b-plus-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/granite-speech-4.1-2b-plus.md - -upstream_commit: edd3bf5 pin_date: 2026-05-17 validation: @@ -13,15 +8,7 @@ validation: commit: 275332d date: 2026-05-17 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - de - - es - - pt tags: - gguf - transcribe.cpp @@ -51,65 +38,14 @@ summary: | This variant is transcription-only: unlike the base granite-speech-4.1-2b, it does not perform speech translation. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: word # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 33 - cpu: 5.2 - m4: - metal: 11 - cpu: 5 - ryzen-4750u: - vulkan: 2.75 - cpu: 1.4 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding and the model-card chat template (system prompt + - leading-space user instruction + `add_generation_prompt=True`). BF16 - reference baseline (re-run locally with that exact prompt): 1.48%; 0.04pp - above upstream's published 1.44%, within bootstrap CI overlap and likely - a chat-template / normalization difference on the publisher side. Text - normalizer: Whisper `EnglishTextNormalizer`, the same normalizer Open ASR - Leaderboard uses. The `add_generation_prompt=True` is load-bearing — - without it the model emits 25-27 empty hypotheses on short test-clean - clips and WER blows up to ~26%. The transcribe.cpp runtime hard-codes the - prompt correctly; this note only matters if you reproduce the reference. - -quants: - - name: BF16 - filename: granite-speech-4.1-2b-plus-BF16.gguf - size: 4.23 GB - wer: 1.49% - - name: F16 - filename: granite-speech-4.1-2b-plus-F16.gguf - size: 4.23 GB - wer: 1.48% - - name: Q8_0 - filename: granite-speech-4.1-2b-plus-Q8_0.gguf - size: 2.35 GB - wer: 1.50% - - name: Q6_K - filename: granite-speech-4.1-2b-plus-Q6_K.gguf - size: 1.86 GB - wer: 1.46% - - name: Q5_K_M - filename: granite-speech-4.1-2b-plus-Q5_K_M.gguf - size: 1.69 GB - wer: 1.48% - - name: Q4_K_M - filename: granite-speech-4.1-2b-plus-Q4_K_M.gguf - size: 1.49 GB - wer: 1.56% + Greedy decoding with the model-card chat template (system prompt + leading-space + user instruction + `add_generation_prompt=True`). BF16 reference baseline (re-run + locally with that exact prompt): 1.48%; 0.04pp above upstream's published 1.44%, + within bootstrap CI overlap and likely a chat-template / normalization difference on + the publisher side. Text normalizer: Whisper `EnglishTextNormalizer`, the same + normalizer Open ASR Leaderboard uses. The `add_generation_prompt=True` is + load-bearing — without it the model emits 25-27 empty hypotheses on short test-clean + clips and WER blows up to ~26%. The transcribe.cpp runtime hard-codes the prompt + correctly; this note only matters if you reproduce the reference. diff --git a/scripts/hf_cards/granite-speech-4.1-2b.yaml b/scripts/hf_cards/granite-speech-4.1-2b.yaml index 94ce2399..5c848cec 100644 --- a/scripts/hf_cards/granite-speech-4.1-2b.yaml +++ b/scripts/hf_cards/granite-speech-4.1-2b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/granite-speech-4.1-2b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ibm-granite/granite-speech-4.1-2b -target_repo: handy-computer/granite-speech-4.1-2b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/granite-speech-4.1-2b.md - -upstream_commit: 8f4bb5f pin_date: 2026-05-17 validation: @@ -13,16 +8,7 @@ validation: commit: 275332d date: 2026-05-17 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - de - - es - - pt - - ja tags: - gguf - transcribe.cpp @@ -44,60 +30,10 @@ summary: | (en ↔ fr, en ↔ de, en ↔ es, en ↔ pt, en ↔ ja) — always via English, no direct fr↔de etc. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 33 - cpu: 5.8 - m4: - metal: 12 - cpu: 4 - ryzen-4750u: - vulkan: 2.80 - cpu: 1.4 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding. BF16 reference baseline (re-run locally with the - model card's exact prompt): 1.31% — 0.02pp below upstream's published - 1.33%, likely a minor normalization difference on the publisher side and - well within bootstrap CI overlap. Text normalizer: Whisper - `EnglishTextNormalizer`, the same normalizer Open ASR Leaderboard uses. - -quants: - - name: BF16 - filename: granite-speech-4.1-2b-BF16.gguf - size: 4.63 GB - wer: 1.31% - - name: F16 - filename: granite-speech-4.1-2b-F16.gguf - size: 4.63 GB - wer: 1.32% - - name: Q8_0 - filename: granite-speech-4.1-2b-Q8_0.gguf - size: 2.56 GB - wer: 1.32% - - name: Q6_K - filename: granite-speech-4.1-2b-Q6_K.gguf - size: 2.02 GB - wer: 1.29% - - name: Q5_K_M - filename: granite-speech-4.1-2b-Q5_K_M.gguf - size: 1.83 GB - wer: 1.33% - - name: Q4_K_M - filename: granite-speech-4.1-2b-Q4_K_M.gguf - size: 1.60 GB - wer: 1.37% + Greedy decoding. BF16 reference baseline (re-run locally with the model card's exact + prompt): 1.31% — 0.02pp below upstream's published 1.33%, likely a minor + normalization difference on the publisher side and well within bootstrap CI overlap. + Text normalizer: Whisper `EnglishTextNormalizer`, the same normalizer Open ASR + Leaderboard uses. diff --git a/scripts/hf_cards/granite-speech-5.0-470m-turboctc-nc.yaml b/scripts/hf_cards/granite-speech-5.0-470m-turboctc-nc.yaml index 7da78bf3..a905592f 100644 --- a/scripts/hf_cards/granite-speech-5.0-470m-turboctc-nc.yaml +++ b/scripts/hf_cards/granite-speech-5.0-470m-turboctc-nc.yaml @@ -1,15 +1,10 @@ # Spec for the HF README of handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. # # NON-COMMERCIAL. Upstream is CC-BY-NC-SA-4.0, so these GGUFs are derivatives # under the same terms (ShareAlike). Do not copy the Apache-2.0 sibling's # license fields into this file. -hf_repo: ibm-granite/granite-speech-5.0-470m-turboctc-nc -target_repo: handy-computer/granite-speech-5.0-470m-turboctc-nc-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/granite-speech-5.0-470m-turboctc-nc.md - -upstream_commit: 0eb7b4f pin_date: 2026-09-12 # Validation pin for the most recent upload. Updated on each release — @@ -19,11 +14,7 @@ validation: commit: f1d0e10 date: 2026-09-12 -license: cc-by-nc-sa-4.0 -license_display: CC-BY-NC-SA-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -35,62 +26,11 @@ tags: - non-commercial summary: | - Offline English speech-to-text, research and non-commercial use only. A 470M - parameter Granite Conformer encoder with a self-conditioned CTC head. - Takes a 16 kHz mono WAV and produces a transcript. Not a streaming model. English only. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, hand-authored from the bench as -# the mean of the four publication cells (q8_0/q4_k_m × jfk/dots); published raw -# as rtf_ in the metadata block. -perf: - m4-max: - cpu: 49 - metal: 351 - m4: - cpu: 28.5 - metal: 119 - ryzen-4750u: - cpu: 15.9 - vulkan: 20.3 + Offline English speech-to-text, research and non-commercial use only. A Granite Conformer + encoder with a self-conditioned CTC head. Takes a 16 kHz mono WAV and produces a + transcript. Not a streaming model. English only. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances), - greedy CTC decoding, no external LM. Measured reference baseline - (transformers 5.17.0, F32, CPU): 1.29%, 95% CI [1.15, 1.42]. - -quants: - - name: BF16 - filename: granite-speech-5.0-470m-turboctc-nc-BF16.gguf - size: 948 MB - wer: 1.29% - - name: F16 - filename: granite-speech-5.0-470m-turboctc-nc-F16.gguf - size: 949 MB - wer: 1.29% - - name: Q8_0 - filename: granite-speech-5.0-470m-turboctc-nc-Q8_0.gguf - size: 506 MB - wer: 1.30% - - name: Q6_K - filename: granite-speech-5.0-470m-turboctc-nc-Q6_K.gguf - size: 392 MB - wer: 1.28% - - name: Q5_K_M - filename: granite-speech-5.0-470m-turboctc-nc-Q5_K_M.gguf - size: 336 MB - wer: 1.29% - - name: Q4_K_M - filename: granite-speech-5.0-470m-turboctc-nc-Q4_K_M.gguf - size: 279 MB - wer: 1.33% + Greedy CTC decoding, no external LM. Measured reference baseline (transformers + 5.17.0, F32, CPU): 1.29%, 95% CI [1.15, 1.42]. diff --git a/scripts/hf_cards/granite-speech-5.0-470m-turboctc.yaml b/scripts/hf_cards/granite-speech-5.0-470m-turboctc.yaml index b538f413..1703642a 100644 --- a/scripts/hf_cards/granite-speech-5.0-470m-turboctc.yaml +++ b/scripts/hf_cards/granite-speech-5.0-470m-turboctc.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/granite-speech-5.0-470m-turboctc-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: ibm-granite/granite-speech-5.0-470m-turboctc -target_repo: handy-computer/granite-speech-5.0-470m-turboctc-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/granite-speech-5.0-470m-turboctc.md - -upstream_commit: 18ca3c1 pin_date: 2026-09-12 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: b9427cf date: 2026-09-12 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -30,61 +21,11 @@ tags: - ctc summary: | - Offline English speech-to-text. A 470M parameter Granite Conformer encoder - with a self-conditioned CTC head. - Takes a 16 kHz mono WAV and produces a transcript. Not a streaming model. English only. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, hand-authored from the bench; -# published raw as rtf_ in the metadata block. -perf: - m4-max: - cpu: 49 - metal: 353 - m4: - cpu: 29 - metal: 130 - ryzen-4750u: - cpu: 15.5 - vulkan: 20.5 + Offline English speech-to-text. A Granite Conformer encoder with a self-conditioned CTC + head. Takes a 16 kHz mono WAV and produces a transcript. Not a streaming model. English + only. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances), - greedy CTC decoding, no external LM. Measured reference baseline - (transformers 5.17.0, F32, CPU): 1.33%, 95% CI [1.20, 1.47]. - -quants: - - name: BF16 - filename: granite-speech-5.0-470m-turboctc-BF16.gguf - size: 948 MB - wer: 1.33% - - name: F16 - filename: granite-speech-5.0-470m-turboctc-F16.gguf - size: 948 MB - wer: 1.33% - - name: Q8_0 - filename: granite-speech-5.0-470m-turboctc-Q8_0.gguf - size: 506 MB - wer: 1.34% - - name: Q6_K - filename: granite-speech-5.0-470m-turboctc-Q6_K.gguf - size: 392 MB - wer: 1.33% - - name: Q5_K_M - filename: granite-speech-5.0-470m-turboctc-Q5_K_M.gguf - size: 336 MB - wer: 1.34% - - name: Q4_K_M - filename: granite-speech-5.0-470m-turboctc-Q4_K_M.gguf - size: 279 MB - wer: 1.34% + Greedy CTC decoding, no external LM. Measured reference baseline (transformers + 5.17.0, F32, CPU): 1.33%, 95% CI [1.20, 1.47]. diff --git a/scripts/hf_cards/medasr.yaml b/scripts/hf_cards/medasr.yaml index 4aad6245..683ae33f 100644 --- a/scripts/hf_cards/medasr.yaml +++ b/scripts/hf_cards/medasr.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/medasr-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: google/medasr -target_repo: handy-computer/medasr-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/medasr.md - -upstream_commit: ae1e484 pin_date: 2026-06-04 # Validation pin for the most recent upload. Updated on each release; @@ -15,11 +10,7 @@ validation: commit: 782abfd date: 2026-06-04 -license: other -license_display: Health AI Developer Foundations pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -34,52 +25,14 @@ tags: summary: | Offline English speech-to-text optimized for medical dictation (radiology, internal medicine, family medicine). 17-layer Conformer encoder with RoPE attention, macaron FFNs, and a 512-token SentencePiece CTC head. Greedy CTC decode; no language model, no beam search. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 354.5 - cpu: 65.5 - ryzen-4750u: - vulkan: 71 - cpu: 19.5 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2,620 utterances) with greedy CTC decoding and no external LM. F32 reference baseline (HuggingFace transformers, Mac MPS): 17.88%; transcribe.cpp F32 matches exactly. Absolute WER is higher than general-purpose ASR (e.g. Whisper-base ~5%) because the model is fine-tuned for medical dictation — on the publisher's internal RAD-DICT / GENERAL-DICT / FM-DICT datasets the model scores 6.6%–9.3%, but those datasets are not publicly reproducible. Q8_0 is the recommended default (smallest preset with no statistically detectable WER degradation); Q4_K_M shows a real +0.26 pp degradation and is shipped for completeness but not recommended — prefer Q5_K_M if you need smaller than Q8_0. - -quants: - - name: F32 - filename: medasr-F32.gguf - size: 417 MB - wer: 17.88% - - name: F16 - filename: medasr-F16.gguf - size: 202 MB - wer: 17.88% - - name: Q8_0 - filename: medasr-Q8_0.gguf - size: 122 MB - wer: 17.86% - - name: Q6_K - filename: medasr-Q6_K.gguf - size: 101 MB - wer: 17.93% - - name: Q5_K_M - filename: medasr-Q5_K_M.gguf - size: 90 MB - wer: 17.91% - - name: Q4_K_M - filename: medasr-Q4_K_M.gguf - size: 79 MB - wer: 18.14% + Greedy CTC decoding, no external LM. F32 reference baseline (HuggingFace + transformers, Mac MPS): 17.88%; transcribe.cpp F32 matches exactly. Absolute WER is + higher than general-purpose ASR (e.g. Whisper-base ~5%) because the model is + fine-tuned for medical dictation — on the publisher's internal RAD-DICT / + GENERAL-DICT / FM-DICT datasets the model scores 6.6%–9.3%, but those datasets are + not publicly reproducible. Q8_0 is the recommended default (smallest preset with no + statistically detectable WER degradation); Q4_K_M shows a real +0.26 pp degradation + and is shipped for completeness but not recommended — prefer Q5_K_M if you need + smaller than Q8_0. diff --git a/scripts/hf_cards/moonshine-base-ar.yaml b/scripts/hf_cards/moonshine-base-ar.yaml index bf3996d8..257efd15 100644 --- a/scripts/hf_cards/moonshine-base-ar.yaml +++ b/scripts/hf_cards/moonshine-base-ar.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-base-ar-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-base-ar -target_repo: handy-computer/moonshine-base-ar-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: 264cc18 pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ar tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (ar); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: WER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-base (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 79.5 - cpu: 80.5 - ryzen-4750u: - vulkan: 34.5 - cpu: 22 - wer: - metadata_key: fleurs_ar - source: FLEURS ar test notes: | - WER measured on the FLEURS-ar test split (428 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language WER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **24.51% WER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-base-ar-F32.gguf - size: 236 MB - wer: 24.45% - - name: F16 - filename: moonshine-base-ar-F16.gguf - size: 126 MB - wer: 24.45% - - name: Q8_0 - filename: moonshine-base-ar-Q8_0.gguf - size: 74 MB - wer: 24.50% diff --git a/scripts/hf_cards/moonshine-base-ja.yaml b/scripts/hf_cards/moonshine-base-ja.yaml index 6c00dd76..17998542 100644 --- a/scripts/hf_cards/moonshine-base-ja.yaml +++ b/scripts/hf_cards/moonshine-base-ja.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-base-ja-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-base-ja -target_repo: handy-computer/moonshine-base-ja-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: f9d4e6a pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ja tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (ja); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: CER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-base (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 79.5 - cpu: 80.5 - ryzen-4750u: - vulkan: 34.5 - cpu: 22 - wer: - metadata_key: fleurs_ja - source: FLEURS ja test notes: | - CER measured on the FLEURS-ja test split (650 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language CER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **10.69% CER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-base-ja-F32.gguf - size: 236 MB - wer: 10.84% - - name: F16 - filename: moonshine-base-ja-F16.gguf - size: 126 MB - wer: 10.81% - - name: Q8_0 - filename: moonshine-base-ja-Q8_0.gguf - size: 74 MB - wer: 10.53% diff --git a/scripts/hf_cards/moonshine-base-ko.yaml b/scripts/hf_cards/moonshine-base-ko.yaml index e6145987..976147f6 100644 --- a/scripts/hf_cards/moonshine-base-ko.yaml +++ b/scripts/hf_cards/moonshine-base-ko.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-base-ko-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-base-ko -target_repo: handy-computer/moonshine-base-ko-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: fff6035 pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ko tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (ko); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: CER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-base (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 79.5 - cpu: 80.5 - ryzen-4750u: - vulkan: 34.5 - cpu: 22 - wer: - metadata_key: fleurs_ko - source: FLEURS ko test notes: | - CER measured on the FLEURS-ko test split (382 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language CER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **8.19% CER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-base-ko-F32.gguf - size: 236 MB - wer: 8.00% - - name: F16 - filename: moonshine-base-ko-F16.gguf - size: 126 MB - wer: 8.00% - - name: Q8_0 - filename: moonshine-base-ko-Q8_0.gguf - size: 74 MB - wer: 8.13% diff --git a/scripts/hf_cards/moonshine-base-uk.yaml b/scripts/hf_cards/moonshine-base-uk.yaml index c4041fbe..2e3c8ed6 100644 --- a/scripts/hf_cards/moonshine-base-uk.yaml +++ b/scripts/hf_cards/moonshine-base-uk.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-base-uk-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-base-uk -target_repo: handy-computer/moonshine-base-uk-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: a52b2e9 pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - uk tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (uk); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: WER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-base (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 79.5 - cpu: 80.5 - ryzen-4750u: - vulkan: 34.5 - cpu: 22 - wer: - metadata_key: fleurs_uk - source: FLEURS uk test notes: | - WER measured on the FLEURS-uk test split (750 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language WER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **14.51% WER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-base-uk-F32.gguf - size: 236 MB - wer: 14.49% - - name: F16 - filename: moonshine-base-uk-F16.gguf - size: 126 MB - wer: 14.49% - - name: Q8_0 - filename: moonshine-base-uk-Q8_0.gguf - size: 74 MB - wer: 14.39% diff --git a/scripts/hf_cards/moonshine-base-vi.yaml b/scripts/hf_cards/moonshine-base-vi.yaml index 2b78e809..9b258aff 100644 --- a/scripts/hf_cards/moonshine-base-vi.yaml +++ b/scripts/hf_cards/moonshine-base-vi.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-base-vi-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-base-vi -target_repo: handy-computer/moonshine-base-vi-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: 0ee9ce9 pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - vi tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (vi); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: WER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-base (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 79.5 - cpu: 80.5 - ryzen-4750u: - vulkan: 34.5 - cpu: 22 - wer: - metadata_key: fleurs_vi - source: FLEURS vi test notes: | - WER measured on the FLEURS-vi test split (857 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language WER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **9.66% WER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-base-vi-F32.gguf - size: 236 MB - wer: 9.67% - - name: F16 - filename: moonshine-base-vi-F16.gguf - size: 126 MB - wer: 9.66% - - name: Q8_0 - filename: moonshine-base-vi-Q8_0.gguf - size: 74 MB - wer: 9.79% diff --git a/scripts/hf_cards/moonshine-base-zh.yaml b/scripts/hf_cards/moonshine-base-zh.yaml index 537e289d..724e7984 100644 --- a/scripts/hf_cards/moonshine-base-zh.yaml +++ b/scripts/hf_cards/moonshine-base-zh.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-base-zh-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-base-zh -target_repo: handy-computer/moonshine-base-zh-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: 1df4f95 pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - zh tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (zh); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: CER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-base (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 79.5 - cpu: 80.5 - ryzen-4750u: - vulkan: 34.5 - cpu: 22 - wer: - metadata_key: fleurs_zh - source: FLEURS zh test notes: | - CER measured on the FLEURS-zh test split (945 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language CER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **16.61% CER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-base-zh-F32.gguf - size: 236 MB - wer: 16.65% - - name: F16 - filename: moonshine-base-zh-F16.gguf - size: 126 MB - wer: 16.65% - - name: Q8_0 - filename: moonshine-base-zh-Q8_0.gguf - size: 74 MB - wer: 17.00% diff --git a/scripts/hf_cards/moonshine-base.yaml b/scripts/hf_cards/moonshine-base.yaml index d2a57521..753b00e8 100644 --- a/scripts/hf_cards/moonshine-base.yaml +++ b/scripts/hf_cards/moonshine-base.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-base-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-base -target_repo: handy-computer/moonshine-base-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine-base.md - -upstream_commit: 7a73d8d pin_date: 2026-05-05 validation: @@ -13,11 +8,7 @@ validation: commit: 07a8a84 date: 2026-05-05 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -28,54 +19,18 @@ tags: - encoder-decoder summary: | - Useful Sensors Moonshine base — a 61M-parameter encoder-decoder transformer - for English speech recognition. Consumes raw 16 kHz PCM directly via a - three-layer Conv1d stem (no STFT, no mel) and emits transcript-only output. - Wider and deeper than moonshine-tiny (8 encoder / 8 decoder layers, hidden - size 416, partial RoPE 0.62). English-only; no translation, no language - detection, no timestamps. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 79.5 - cpu: 80.5 - ryzen-4750u: - vulkan: 34.5 - cpu: 22 + Useful Sensors Moonshine base — an encoder-decoder transformer for English + speech recognition. Consumes raw 16 kHz PCM directly via a three-layer Conv1d + stem (no STFT, no mel) and emits transcript-only output. Wider and deeper + than moonshine-tiny (8 encoder / 8 decoder layers, hidden size 416, partial + RoPE 0.62). English-only; no translation, no language detection, no + timestamps. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with the transcribe.cpp default decode (greedy, num_beams=1, - max_length=194 — matching the upstream generation_config). Upstream - reports 3.27% on the same split (Moonshine paper, Table 2; also Open - ASR Leaderboard). Our F32 reference baseline lands at 3.28%, identical - to upstream within rounding and well within the ±1.00 pp Stage 7 - acceptance gate. Q8_0 lands at 3.26%, slightly under F32 — that delta - sits inside the 95% bootstrap CI and is noise, not a real improvement. - -quants: - - name: F32 - filename: moonshine-base-F32.gguf - size: 236 MB - wer: 3.28% - - name: F16 - filename: moonshine-base-F16.gguf - size: 126 MB - wer: 3.28% - - name: Q8_0 - filename: moonshine-base-Q8_0.gguf - size: 74 MB - wer: 3.26% + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=194, + matching the upstream generation_config). Upstream reports 3.27% on the same split + (Moonshine paper, Table 2; also Open ASR Leaderboard). Our F32 reference baseline + lands at 3.28%, identical to upstream within rounding and well within the ±1.00 pp + Stage 7 acceptance gate. Q8_0 lands at 3.26%, slightly under F32 — that delta sits + inside the 95% bootstrap CI and is noise, not a real improvement. diff --git a/scripts/hf_cards/moonshine-streaming-medium.yaml b/scripts/hf_cards/moonshine-streaming-medium.yaml index 59bc161a..62f8a12e 100644 --- a/scripts/hf_cards/moonshine-streaming-medium.yaml +++ b/scripts/hf_cards/moonshine-streaming-medium.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-streaming-medium-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-streaming-medium -target_repo: handy-computer/moonshine-streaming-medium-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine-streaming-medium.md - -upstream_commit: 57b8436 pin_date: 2026-05-06 validation: @@ -13,11 +8,7 @@ validation: commit: 0d312ce date: 2026-05-06 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -29,57 +20,19 @@ tags: - encoder-decoder summary: | - English speech-to-text in both one-shot and streaming modes. A 245M-parameter - encoder-decoder ASR model designed for streaming use (ergodic encoder + - sliding-window attention, 50 Hz time-domain frontend). Same family as - moonshine-streaming-tiny and moonshine-streaming-small; deepest of the three - (14 / 14 layers) and widest hidden dims (encoder 768 / decoder 640). Takes a - 16 kHz mono WAV and produces a transcript. No translation, no multilingual - capability, no timestamps. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 63 - cpu: 35.5 - ryzen-4750u: - vulkan: 14 - cpu: 7.5 + English speech-to-text in both one-shot and streaming modes. An encoder-decoder + ASR model designed for streaming use (ergodic encoder + sliding-window + attention, 50 Hz time-domain frontend). Same family as moonshine-streaming-tiny + and moonshine-streaming-small; deepest of the three (14 / 14 layers) and widest + hidden dims (encoder 768 / decoder 640). Takes a 16 kHz mono WAV and produces a + transcript. No translation, no multilingual capability, no timestamps. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference - baseline: 2.16%. Quants are numerically indistinguishable from F32 on - this manifest. Useful Sensors' self-reported number on this split is - 2.08% from the Open ASR Leaderboard table; the +0.08pp residual matches - the same scoring / text-normalization difference seen across the tiny - and small variants (cross-checked against HF Transformers on tiny and - found to be at 99.6% identical hypotheses to our port), and is not a - numerical drift. Q6_K / Q5_K_M / Q4_K_M GGUFs are not currently shipped - for this variant. - -quants: - - name: F32 - filename: moonshine-streaming-medium-F32.gguf - size: 1015 MB - wer: 2.16% - - name: F16 - filename: moonshine-streaming-medium-F16.gguf - size: 509 MB - wer: 2.16% - - name: Q8_0 - filename: moonshine-streaming-medium-Q8_0.gguf - size: 282 MB - wer: 2.16% + Greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference baseline: 2.16%. + Quants are numerically indistinguishable from F32 on this manifest. Useful Sensors' + self-reported number on this split is 2.08% from the Open ASR Leaderboard table; the + +0.08pp residual matches the same scoring / text-normalization difference seen + across the tiny and small variants (cross-checked against HF Transformers on tiny + and found to be at 99.6% identical hypotheses to our port), and is not a numerical + drift. Q6_K / Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. diff --git a/scripts/hf_cards/moonshine-streaming-small.yaml b/scripts/hf_cards/moonshine-streaming-small.yaml index a8557d40..e1180f14 100644 --- a/scripts/hf_cards/moonshine-streaming-small.yaml +++ b/scripts/hf_cards/moonshine-streaming-small.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-streaming-small-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-streaming-small -target_repo: handy-computer/moonshine-streaming-small-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine-streaming-small.md - -upstream_commit: 2c03650 pin_date: 2026-05-06 validation: @@ -13,11 +8,7 @@ validation: commit: 0d312ce date: 2026-05-06 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -29,55 +20,19 @@ tags: - encoder-decoder summary: | - English speech-to-text in both one-shot and streaming modes. A 123M-parameter - encoder-decoder ASR model designed for streaming use (ergodic encoder + - sliding-window attention, 50 Hz time-domain frontend). Same family as + English speech-to-text in both one-shot and streaming modes. An encoder-decoder + ASR model designed for streaming use (ergodic encoder + sliding-window + attention, 50 Hz time-domain frontend). Same family as moonshine-streaming-tiny; deeper encoder/decoder (10 / 10 layers) and wider - hidden dims (encoder 620 / decoder 512). Takes a 16 kHz mono WAV and produces - a transcript. No translation, no multilingual capability, no timestamps. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 96 - cpu: 57 - ryzen-4750u: - vulkan: 23.5 - cpu: 12 + hidden dims (encoder 620 / decoder 512). Takes a 16 kHz mono WAV and produces a + transcript. No translation, no multilingual capability, no timestamps. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference - baseline: 2.53%. Useful Sensors' self-reported number on this split is - 2.49% from the Open ASR Leaderboard table; the +0.04pp residual matches - the same scoring / text-normalization difference seen on the tiny variant - where we cross-checked against the HF Transformers reference (4.52% on - the same manifest, 99.6% identical hypotheses to our F32) and confirmed - it is not a numerical drift in the port. Q6_K / Q5_K_M / Q4_K_M GGUFs - are not currently shipped for this variant. - -quants: - - name: F32 - filename: moonshine-streaming-small-F32.gguf - size: 536 MB - wer: 2.53% - - name: F16 - filename: moonshine-streaming-small-F16.gguf - size: 269 MB - wer: 2.53% - - name: Q8_0 - filename: moonshine-streaming-small-Q8_0.gguf - size: 189 MB - wer: 2.54% + Greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference baseline: 2.53%. + Useful Sensors' self-reported number on this split is 2.49% from the Open ASR + Leaderboard table; the +0.04pp residual matches the same scoring / + text-normalization difference seen on the tiny variant where we cross-checked + against the HF Transformers reference (4.52% on the same manifest, 99.6% identical + hypotheses to our F32) and confirmed it is not a numerical drift in the port. Q6_K / + Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. diff --git a/scripts/hf_cards/moonshine-streaming-tiny.yaml b/scripts/hf_cards/moonshine-streaming-tiny.yaml index d92e2581..61723aca 100644 --- a/scripts/hf_cards/moonshine-streaming-tiny.yaml +++ b/scripts/hf_cards/moonshine-streaming-tiny.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-streaming-tiny-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-streaming-tiny -target_repo: handy-computer/moonshine-streaming-tiny-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine-streaming-tiny.md - -upstream_commit: f8e9dfd pin_date: 2026-05-06 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 0d312ce date: 2026-05-06 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -31,54 +22,18 @@ tags: - encoder-decoder summary: | - English speech-to-text in both one-shot and streaming modes. A 34M-parameter + English speech-to-text in both one-shot and streaming modes. An encoder-decoder ASR model designed for streaming use (ergodic encoder + - sliding-window attention, 50 Hz time-domain frontend). Takes a 16 kHz mono - WAV and produces a transcript. No translation, no multilingual capability, - no timestamps. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 159 - cpu: 211 - ryzen-4750u: - vulkan: 59.5 - cpu: 54.5 + sliding-window attention, 50 Hz time-domain frontend). Takes a 16 kHz mono WAV + and produces a transcript. No translation, no multilingual capability, no + timestamps. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference - baseline: 4.53%. The HF Transformers reference scored on the same manifest - in the same regime lands at 4.52% with 99.6% byte-identical hypotheses to - our F32, so the port is at exact parity with the reference. Useful Sensors' - self-reported number on this split is 4.49% from the Open ASR Leaderboard - table; the +0.04pp residual is a scoring / text-normalization difference vs - that methodology, not a numerical drift in the port. Q6_K / Q5_K_M / Q4_K_M - GGUFs are not currently shipped for this variant. - -quants: - - name: F32 - filename: moonshine-streaming-tiny-F32.gguf - size: 169 MB - wer: 4.53% - - name: F16 - filename: moonshine-streaming-tiny-F16.gguf - size: 85 MB - wer: 4.53% - - name: Q8_0 - filename: moonshine-streaming-tiny-Q8_0.gguf - size: 48 MB - wer: 4.52% + Greedy decoding (`num_beams=1`, `do_sample=False`). F32 reference baseline: 4.53%. + The HF Transformers reference scored on the same manifest in the same regime lands + at 4.52% with 99.6% byte-identical hypotheses to our F32, so the port is at exact + parity with the reference. Useful Sensors' self-reported number on this split is + 4.49% from the Open ASR Leaderboard table; the +0.04pp residual is a scoring / + text-normalization difference vs that methodology, not a numerical drift in the + port. Q6_K / Q5_K_M / Q4_K_M GGUFs are not currently shipped for this variant. diff --git a/scripts/hf_cards/moonshine-tiny-ar.yaml b/scripts/hf_cards/moonshine-tiny-ar.yaml index 55ddc3e4..bc4bc43f 100644 --- a/scripts/hf_cards/moonshine-tiny-ar.yaml +++ b/scripts/hf_cards/moonshine-tiny-ar.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-tiny-ar-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-tiny-ar -target_repo: handy-computer/moonshine-tiny-ar-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: 99e7fee pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ar tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (ar); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: WER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-tiny (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 127 - cpu: 153.5 - ryzen-4750u: - vulkan: 56 - cpu: 45.5 - wer: - metadata_key: fleurs_ar - source: FLEURS ar test notes: | - WER measured on the FLEURS-ar test split (428 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language WER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **27.13% WER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-tiny-ar-F32.gguf - size: 105 MB - wer: 27.11% - - name: F16 - filename: moonshine-tiny-ar-F16.gguf - size: 57 MB - wer: 27.11% - - name: Q8_0 - filename: moonshine-tiny-ar-Q8_0.gguf - size: 34 MB - wer: 26.79% diff --git a/scripts/hf_cards/moonshine-tiny-ja.yaml b/scripts/hf_cards/moonshine-tiny-ja.yaml index 2c206aaf..4219d1e1 100644 --- a/scripts/hf_cards/moonshine-tiny-ja.yaml +++ b/scripts/hf_cards/moonshine-tiny-ja.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-tiny-ja-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-tiny-ja -target_repo: handy-computer/moonshine-tiny-ja-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: 02ca41b pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ja tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (ja); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: CER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-tiny (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 127 - cpu: 153.5 - ryzen-4750u: - vulkan: 56 - cpu: 45.5 - wer: - metadata_key: fleurs_ja - source: FLEURS ja test notes: | - CER measured on the FLEURS-ja test split (650 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language CER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **13.23% CER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-tiny-ja-F32.gguf - size: 105 MB - wer: 13.20% - - name: F16 - filename: moonshine-tiny-ja-F16.gguf - size: 57 MB - wer: 13.20% - - name: Q8_0 - filename: moonshine-tiny-ja-Q8_0.gguf - size: 34 MB - wer: 13.36% diff --git a/scripts/hf_cards/moonshine-tiny-ko.yaml b/scripts/hf_cards/moonshine-tiny-ko.yaml index 23ffcff2..2f779818 100644 --- a/scripts/hf_cards/moonshine-tiny-ko.yaml +++ b/scripts/hf_cards/moonshine-tiny-ko.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-tiny-ko-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-tiny-ko -target_repo: handy-computer/moonshine-tiny-ko-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: 80995f0 pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - ko tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (ko); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: CER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-tiny (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 127 - cpu: 153.5 - ryzen-4750u: - vulkan: 56 - cpu: 45.5 - wer: - metadata_key: fleurs_ko - source: FLEURS ko test notes: | - CER measured on the FLEURS-ko test split (382 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language CER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **8.99% CER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-tiny-ko-F32.gguf - size: 105 MB - wer: 8.97% - - name: F16 - filename: moonshine-tiny-ko-F16.gguf - size: 57 MB - wer: 8.97% - - name: Q8_0 - filename: moonshine-tiny-ko-Q8_0.gguf - size: 34 MB - wer: 8.98% diff --git a/scripts/hf_cards/moonshine-tiny-uk.yaml b/scripts/hf_cards/moonshine-tiny-uk.yaml index 8a5f0620..09c8648e 100644 --- a/scripts/hf_cards/moonshine-tiny-uk.yaml +++ b/scripts/hf_cards/moonshine-tiny-uk.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-tiny-uk-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-tiny-uk -target_repo: handy-computer/moonshine-tiny-uk-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: 8cf70f4 pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - uk tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (uk); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: WER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-tiny (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 127 - cpu: 153.5 - ryzen-4750u: - vulkan: 56 - cpu: 45.5 - wer: - metadata_key: fleurs_uk - source: FLEURS uk test notes: | - WER measured on the FLEURS-uk test split (750 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language WER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **18.86% WER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-tiny-uk-F32.gguf - size: 105 MB - wer: 18.83% - - name: F16 - filename: moonshine-tiny-uk-F16.gguf - size: 57 MB - wer: 18.82% - - name: Q8_0 - filename: moonshine-tiny-uk-Q8_0.gguf - size: 34 MB - wer: 18.89% diff --git a/scripts/hf_cards/moonshine-tiny-vi.yaml b/scripts/hf_cards/moonshine-tiny-vi.yaml index b6356db9..8721b631 100644 --- a/scripts/hf_cards/moonshine-tiny-vi.yaml +++ b/scripts/hf_cards/moonshine-tiny-vi.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-tiny-vi-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-tiny-vi -target_repo: handy-computer/moonshine-tiny-vi-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: d4d20da pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - vi tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (vi); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: WER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-tiny (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 127 - cpu: 153.5 - ryzen-4750u: - vulkan: 56 - cpu: 45.5 - wer: - metadata_key: fleurs_vi - source: FLEURS vi test notes: | - WER measured on the FLEURS-vi test split (857 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language WER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **13.18% WER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-tiny-vi-F32.gguf - size: 105 MB - wer: 13.16% - - name: F16 - filename: moonshine-tiny-vi-F16.gguf - size: 57 MB - wer: 13.17% - - name: Q8_0 - filename: moonshine-tiny-vi-Q8_0.gguf - size: 34 MB - wer: 13.16% diff --git a/scripts/hf_cards/moonshine-tiny-zh.yaml b/scripts/hf_cards/moonshine-tiny-zh.yaml index c1780651..0ec78af5 100644 --- a/scripts/hf_cards/moonshine-tiny-zh.yaml +++ b/scripts/hf_cards/moonshine-tiny-zh.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-tiny-zh-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-tiny-zh -target_repo: handy-computer/moonshine-tiny-zh-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine.md - -upstream_commit: "3306270" pin_date: 2026-05-12 validation: @@ -13,11 +8,7 @@ validation: commit: 90bf720 date: 2026-05-12 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - zh tags: - gguf - transcribe.cpp @@ -34,33 +25,10 @@ summary: | STFT, no mel filterbank) and emits transcript-only output. Single-language (zh); no translation, no language detection, no timestamps. -default_quant_index: 2 # Q8_0 - -metric: CER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Perf inherited from moonshine-tiny (identical architecture; per- -# language RTF not separately benched). -perf: - m4-max: - metal: 127 - cpu: 153.5 - ryzen-4750u: - vulkan: 56 - cpu: 45.5 - wer: - metadata_key: fleurs_zh - source: FLEURS zh test notes: | - CER measured on the FLEURS-zh test split (945 - utterances) using the transcribe.cpp default decode (greedy, - num_beams=1, max_length=192 — matching the upstream generation_config). + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=192, + matching the upstream generation_config). UsefulSensors does not publish a per-language CER number for this variant. As a comparable baseline we ran the Transformers F32 @@ -68,17 +36,3 @@ wer: same manifest: **13.72% CER**. The C++ F32/F16 numbers above match the reference within bootstrap-CI noise; Q8_0 introduces a small additional drift from F16 (typically within 0.1pp). - -quants: - - name: F32 - filename: moonshine-tiny-zh-F32.gguf - size: 105 MB - wer: 13.70% - - name: F16 - filename: moonshine-tiny-zh-F16.gguf - size: 57 MB - wer: 13.70% - - name: Q8_0 - filename: moonshine-tiny-zh-Q8_0.gguf - size: 34 MB - wer: 13.78% diff --git a/scripts/hf_cards/moonshine-tiny.yaml b/scripts/hf_cards/moonshine-tiny.yaml index a777d281..191d8924 100644 --- a/scripts/hf_cards/moonshine-tiny.yaml +++ b/scripts/hf_cards/moonshine-tiny.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/moonshine-tiny-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: UsefulSensors/moonshine-tiny -target_repo: handy-computer/moonshine-tiny-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moonshine-tiny.md - -upstream_commit: 390624e pin_date: 2026-05-05 validation: @@ -13,11 +8,7 @@ validation: commit: 07a8a84 date: 2026-05-05 -license: mit -license_display: MIT pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -28,51 +19,15 @@ tags: - encoder-decoder summary: | - Useful Sensors Moonshine tiny — a 27M-parameter encoder-decoder transformer - for English speech recognition. Consumes raw 16 kHz PCM directly via a - three-layer Conv1d stem (no STFT, no mel) and emits transcript-only output. - English-only; no translation, no language detection, no timestamps. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 127 - cpu: 153.5 - ryzen-4750u: - vulkan: 56 - cpu: 45.5 + Useful Sensors Moonshine tiny — an encoder-decoder transformer for English + speech recognition. Consumes raw 16 kHz PCM directly via a three-layer Conv1d + stem (no STFT, no mel) and emits transcript-only output. English-only; no + translation, no language detection, no timestamps. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with the transcribe.cpp default decode (greedy, num_beams=1, - max_length=194 — matching the upstream generation_config). Useful - Sensors' self-reported number on the same split is 4.55% (model card). - Our F32 reference baseline lands at 4.58%, within rounding of upstream - and well within the ±1.00 pp Stage 7 acceptance gate. Q8_0 drift is - +0.02 pp vs F32 — within bootstrap CI noise. - -quants: - - name: F32 - filename: moonshine-tiny-F32.gguf - size: 105 MB - wer: 4.58% - - name: F16 - filename: moonshine-tiny-F16.gguf - size: 57 MB - wer: 4.58% - - name: Q8_0 - filename: moonshine-tiny-Q8_0.gguf - size: 34 MB - wer: 4.60% + Decoded with the transcribe.cpp defaults (greedy, num_beams=1, max_length=194, + matching the upstream generation_config). Useful Sensors' self-reported number on + the same split is 4.55% (model card). Our F32 reference baseline lands at 4.58%, + within rounding of upstream and well within the ±1.00 pp Stage 7 acceptance gate. + Q8_0 drift is +0.02 pp vs F32 — within bootstrap CI noise. diff --git a/scripts/hf_cards/moss-transcribe-diarize.yaml b/scripts/hf_cards/moss-transcribe-diarize.yaml index d9ceb748..ab097274 100644 --- a/scripts/hf_cards/moss-transcribe-diarize.yaml +++ b/scripts/hf_cards/moss-transcribe-diarize.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/MOSS-Transcribe-Diarize-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: OpenMOSS-Team/MOSS-Transcribe-Diarize -target_repo: handy-computer/MOSS-Transcribe-Diarize-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/moss-transcribe-diarize.md - -upstream_commit: d7231bb pin_date: 2026-07-12 validation: @@ -13,12 +8,7 @@ validation: commit: 3f5e15c date: 2026-07-12 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - zh tags: - gguf - transcribe.cpp @@ -39,71 +29,20 @@ summary: | Speaker attribution is opt-in (`--diarize`) and returns structured speaker ids/turns. Not a streaming model. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (xRT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. Values average the Q8_0 jfk and -# dots publication cells. -perf: - m4-max: - metal: 28.1 - cpu: 5.8 - ryzen-4750u: - vulkan: 3.0 - cpu: 1.6 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech `test-clean` split (2620 English - utterances) with the Whisper-style English text normalizer and jiwer - 3.x. MOSS emits the diarized format `[start][Sxx]text[end]`; the bracket - spans are metadata and are de-diarized to a space (for both hypothesis - and reference) before scoring, matching the author-repo reference runner. - These values describe this dataset only, not a general quality ranking: a - quant that scores slightly better here is not necessarily better in - real-world use, because dataset-specific decoding near-ties can make - quantization noise help or hurt individual utterances. The same-manifest - MOSS author-repo reference (bf16, greedy) lands at - **2.07%** with 95% bootstrap CI [1.82%, 2.40%]. The BF16 port lands at - 2.08% (within +0.01 of the reference, well inside the CI band); the - lower-bit presets sit between 1.93% and 1.99% (statistical noise) except - Q4_K_M at 2.59%, whose excess is a handful of 4-bit tail failures - (6 empty outputs, 5 English->Chinese language-drift utterances, 1 - timestamp-token repetition loop) rather than broad degradation. Prefer - Q5_K_M or higher if those tail failures matter. Reproduce with - `scripts/wer/run.py` + `scripts/wer/score.py --dediarize`; public - `full_text` applies equivalent marker removal. - -quants: - - name: BF16 - filename: MOSS-Transcribe-Diarize-BF16.gguf - size: 1.83 GB - wer: 2.08% - - name: F16 - filename: MOSS-Transcribe-Diarize-F16.gguf - size: 1.83 GB - wer: 2.07% - - name: Q8_0 - filename: MOSS-Transcribe-Diarize-Q8_0.gguf - size: 987 MB - wer: 1.93% - - name: Q6_K - filename: MOSS-Transcribe-Diarize-Q6_K.gguf - size: 768 MB - wer: 1.96% - - name: Q5_K_M - filename: MOSS-Transcribe-Diarize-Q5_K_M.gguf - size: 700 MB - wer: 1.99% - - name: Q4_K_M - filename: MOSS-Transcribe-Diarize-Q4_K_M.gguf - size: 617 MB - wer: 2.59% + Scored with the Whisper-style English text normalizer and jiwer 3.x. MOSS emits the + diarized format `[start][Sxx]text[end]`; the bracket spans are metadata and are + de-diarized to a space (for both hypothesis and reference) before scoring, matching + the author-repo reference runner. These values describe this dataset only, not a + general quality ranking: a quant that scores slightly better here is not necessarily + better in real-world use, because dataset-specific decoding near-ties can make + quantization noise help or hurt individual utterances. The same-manifest MOSS + author-repo reference (bf16, greedy) lands at **2.07%** with 95% bootstrap CI + [1.82%, 2.40%]. The BF16 port lands at 2.08% (within +0.01 of the reference, well + inside the CI band); the lower-bit presets sit between 1.93% and 1.99% (statistical + noise) except Q4_K_M at 2.59%, whose excess is a handful of 4-bit tail failures (6 + empty outputs, 5 English->Chinese language-drift utterances, 1 timestamp-token + repetition loop) rather than broad degradation. Prefer Q5_K_M or higher if those + tail failures matter. Reproduce with `scripts/wer/run.py` + `scripts/wer/score.py + --dediarize`; public `full_text` applies equivalent marker removal. diff --git a/scripts/hf_cards/multitalker-parakeet-streaming-0.6b-v1.yaml b/scripts/hf_cards/multitalker-parakeet-streaming-0.6b-v1.yaml index 0aee60b2..4be749b0 100644 --- a/scripts/hf_cards/multitalker-parakeet-streaming-0.6b-v1.yaml +++ b/scripts/hf_cards/multitalker-parakeet-streaming-0.6b-v1.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/multitalker-parakeet-streaming-0.6b-v1 -target_repo: handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/multitalker-parakeet-streaming-0.6b-v1.md - -upstream_commit: 8749fc7 pin_date: 2026-07-12 # Validation pin for the most recent upload. Updated on each release; @@ -15,18 +10,7 @@ validation: commit: 3083021 date: 2026-08-03 -# Upstream is the NVIDIA Open Model License (not a standard SPDX id), so -# the HF `license` field gets `other` and `license_name` / `license_link` -# carry the actual identifier + URL into the YAML frontmatter (HF prefers -# this when license=other). `license_display` is the human-facing form -# rendered in the README body. -license: other -license_name: nvidia-open-model-license -license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ -license_display: NVIDIA Open Model License pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -42,38 +26,15 @@ tags: - sortformer summary: | - Offline and cache-aware streaming English speech-to-text with punctuation and capitalization. A 0.6B-parameter cache-aware streaming FastConformer encoder with an RNN-T transducer decoder, fine-tuned from nvidia/nemotron-speech-streaming-en-0.6b. Plain GGUFs run the single_speaker_mode ASR path, while bundle GGUFs under `bundle/` embed nvidia/diar_streaming_sortformer_4spk-v2.1 and, with `--diarize`, transcribe up to four overlapping speakers into a speaker-tagged transcript. The encoder preserves the upstream att_context_size=[70, 13] (1.12s) cache-aware attention mask; all four latency lookahead settings are selectable. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - diarize: true - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (xRT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. Values average the Q8_0 jfk and -# dots publication cells. -perf: - m4-max: - metal: 178.3 - cpu: 34.6 - ryzen-4750u: - cpu: 13 - vulkan: 25 - -metrics: - cpwer_ami_ihm_test: - bundle_f32_kernel: 19.35 - bundle_f32_masked: 23.73 + Offline and cache-aware streaming English speech-to-text with punctuation and capitalization. A cache-aware streaming FastConformer encoder with an RNN-T transducer decoder, fine-tuned from nvidia/nemotron-speech-streaming-en-0.6b. Plain GGUFs run the single_speaker_mode ASR path, while bundle GGUFs under `bundle/` embed nvidia/diar_streaming_sortformer_4spk-v2.1 and, with `--diarize`, transcribe up to four overlapping speakers into a speaker-tagged transcript. The encoder preserves the upstream att_context_size=[70, 13] (1.12s) cache-aware attention mask; all four latency lookahead settings are selectable. wer: source: LibriSpeech test-clean, offline notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) in single_speaker_mode with greedy RNN-T decoding and whisper-normalizer (PnC-stripped) scoring. F32 reference baseline: 2.19%. The measured NeMo single_speaker_mode reference and NVIDIA's self-reported number on the same split are both 2.19%. + Run in single_speaker_mode with greedy RNN-T decoding and whisper-normalizer + (PnC-stripped) scoring. F32 reference baseline: 2.19%. The measured NeMo + single_speaker_mode reference and NVIDIA's self-reported number on the same split + are both 2.19%. ### Multitalker bundles (speaker-attributed ASR) @@ -95,29 +56,3 @@ wer: -m bundle/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf \ meeting.wav ``` - -quants: - - name: F32 - filename: multitalker-parakeet-streaming-0.6b-v1-F32.gguf - size: 2.49 GB - wer: 2.19% - - name: F16 - filename: multitalker-parakeet-streaming-0.6b-v1-F16.gguf - size: 1.25 GB - wer: 2.19% - - name: Q8_0 - filename: multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf - size: 734 MB - wer: 2.18% - - name: Q6_K - filename: multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf - size: 604 MB - wer: 2.20% - - name: Q5_K_M - filename: multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf - size: 542 MB - wer: 2.18% - - name: Q4_K_M - filename: multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf - size: 478 MB - wer: 2.18% diff --git a/scripts/hf_cards/nemotron-3.5-asr-streaming-0.6b.yaml b/scripts/hf_cards/nemotron-3.5-asr-streaming-0.6b.yaml index 660bfed1..af5266ac 100644 --- a/scripts/hf_cards/nemotron-3.5-asr-streaming-0.6b.yaml +++ b/scripts/hf_cards/nemotron-3.5-asr-streaming-0.6b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/nemotron-3.5-asr-streaming-0.6b -target_repo: handy-computer/nemotron-3.5-asr-streaming-0.6b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/nemotron-3.5-asr-streaming-0.6b.md - -upstream_commit: 24b151a pin_date: 2026-06-08 # Validation pin for the most recent upload. Updated on each release; @@ -15,43 +10,7 @@ validation: commit: 909e94e date: 2026-06-08 -# Upstream is OpenMDW-1.1, not a standard SPDX id, so the HF `license` -# field gets `other` and `license_name` / `license_link` carry the actual -# identifier + URL. `license_display` is the human-facing form. -license: other -license_name: openmdw-1.1 -license_link: https://huggingface.co/nvidia/nemotron-3.5-asr-streaming-0.6b -license_display: OpenMDW-1.1 pipeline_tag: automatic-speech-recognition -languages: - - en - - es - - fr - - it - - pt - - nl - - de - - tr - - ru - - ar - - hi - - ja - - ko - - vi - - uk - - pl - - sv - - cs - - nb - - da - - bg - - fi - - hr - - sk - - zh - - hu - - ro - - et tags: - gguf - transcribe.cpp @@ -65,62 +24,13 @@ tags: - multilingual summary: | - Multilingual speech-to-text across 32 supported language-locales (the model's tokenizer recognizes 40, but 8 are adaptation-ready and need fine-tuning) with punctuation and capitalization. A 0.6B-parameter cache-aware streaming FastConformer encoder with a prompt-conditioned RNN-T transducer decoder; the target language is selected per call (--language en-US, fr-FR, de-DE, ...) and an auto mode emits a tag. Ships both the offline path (att_context_size=[56, 13], 1.12s, headline accuracy) and runtime-selectable chunked streaming (--stream-chunk-ms 1120 --stream-att-right {0,3,6,13}). - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: true - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 98 - cpu: 29 - ryzen-4750u: - vulkan: 14.5 - cpu: 7.5 + Multilingual speech-to-text across 32 supported language-locales (the model's tokenizer recognizes 40, but 8 are adaptation-ready and need fine-tuning) with punctuation and capitalization. A cache-aware streaming FastConformer encoder with a prompt-conditioned RNN-T transducer decoder; the target language is selected per call (--language en-US, fr-FR, de-DE, ...) and an auto mode emits a tag. Ships both the offline path (att_context_size=[56, 13], 1.12s, headline accuracy) and runtime-selectable chunked streaming (--stream-chunk-ms 1120 --stream-att-right {0,3,6,13}). wer: - metadata_key: fleurs_en source: FLEURS test en (en-US), offline att_context_size=[56, 13] notes: | - WER measured on FLEURS test en (647 utterances), greedy RNN-T, --language en-US, whisper-normalizer scoring; the per-quant column is FLEURS en. NeMo reference baseline on the same manifest: 7.99% (NVIDIA self-reports 7.91% en-US). On LibriSpeech test-clean (2620 utterances) the same presets score F32 3.04 / F16 3.03 / Q8_0 3.06 / Q6_K 3.07 / Q5_K_M 3.10 / Q4_K_M 3.28, against a 3.03% NeMo reference. - librispeech_test_clean: - f32: 3.04 - f16: 3.03 - q8_0: 3.06 - q6_k: 3.07 - q5_k_m: 3.10 - q4_k_m: 3.28 - -quants: - - name: F32 - filename: nemotron-3.5-asr-streaming-0.6b-F32.gguf - size: 2.38 GB - wer: 7.97% - - name: F16 - filename: nemotron-3.5-asr-streaming-0.6b-F16.gguf - size: 1.19 GB - wer: 7.97% - - name: Q8_0 - filename: nemotron-3.5-asr-streaming-0.6b-Q8_0.gguf - size: 716 MB - wer: 7.88% - - name: Q6_K - filename: nemotron-3.5-asr-streaming-0.6b-Q6_K.gguf - size: 593 MB - wer: 8.02% - - name: Q5_K_M - filename: nemotron-3.5-asr-streaming-0.6b-Q5_K_M.gguf - size: 534 MB - wer: 8.15% - - name: Q4_K_M - filename: nemotron-3.5-asr-streaming-0.6b-Q4_K_M.gguf - size: 473 MB - wer: 8.49% + Greedy RNN-T decoding with whisper-normalizer scoring; the per-quant column is + FLEURS en. NeMo reference baseline on the same manifest: 7.99% (NVIDIA self-reports + 7.91% en-US). On LibriSpeech test-clean (2620 utterances) the same presets score F32 + 3.04 / F16 3.03 / Q8_0 3.06 / Q6_K 3.07 / Q5_K_M 3.10 / Q4_K_M 3.28, against a 3.03% + NeMo reference. diff --git a/scripts/hf_cards/nemotron-speech-streaming-en-0.6b.yaml b/scripts/hf_cards/nemotron-speech-streaming-en-0.6b.yaml index 67cb8fa0..38119404 100644 --- a/scripts/hf_cards/nemotron-speech-streaming-en-0.6b.yaml +++ b/scripts/hf_cards/nemotron-speech-streaming-en-0.6b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/nemotron-speech-streaming-en-0.6b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/nemotron-speech-streaming-en-0.6b -target_repo: handy-computer/nemotron-speech-streaming-en-0.6b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/nemotron-speech-streaming-en-0.6b.md - -upstream_commit: ef3bf40 pin_date: 2026-05-11 # Validation pin for the most recent upload. Updated on each release; @@ -15,18 +10,7 @@ validation: commit: 12f1076 date: 2026-05-11 -# Upstream is the NVIDIA Open Model License (not a standard SPDX id), so -# the HF `license` field gets `other` and `license_name` / `license_link` -# carry the actual identifier + URL into the YAML frontmatter (HF prefers -# this when license=other). `license_display` is the human-facing form -# rendered in the README body. -license: other -license_name: nvidia-open-model-license -license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ -license_display: NVIDIA Open Model License pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -39,54 +23,10 @@ tags: - cache-aware summary: | - English speech-to-text with punctuation and capitalization. A 0.6B-parameter cache-aware streaming FastConformer encoder with an RNN-T transducer decoder. Runs in both offline and cache-aware streaming modes. The encoder preserves the upstream att_context_size=[70, 13] (1.12s) cache-aware attention mask end-to-end. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 154.5 - cpu: 32 - ryzen-4750u: - vulkan: 13 - cpu: 7.5 + English speech-to-text with punctuation and capitalization. A cache-aware streaming FastConformer encoder with an RNN-T transducer decoder. Runs in both offline and cache-aware streaming modes. The encoder preserves the upstream att_context_size=[70, 13] (1.12s) cache-aware attention mask end-to-end. wer: source: LibriSpeech test-clean, offline notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy RNN-T decoding. F32 reference baseline: 2.31%. NVIDIA's self-reported number on the same split at att_context_size=[70, 13] (1.12s chunk, w/o PnC) is 2.32%. - -quants: - - name: F32 - filename: nemotron-speech-streaming-en-0.6b-F32.gguf - size: 2.30 GB - wer: 2.31% - - name: F16 - filename: nemotron-speech-streaming-en-0.6b-F16.gguf - size: 1.16 GB - wer: 2.31% - - name: Q8_0 - filename: nemotron-speech-streaming-en-0.6b-Q8_0.gguf - size: 696 MB - wer: 2.31% - - name: Q6_K - filename: nemotron-speech-streaming-en-0.6b-Q6_K.gguf - size: 573 MB - wer: 2.29% - - name: Q5_K_M - filename: nemotron-speech-streaming-en-0.6b-Q5_K_M.gguf - size: 514 MB - wer: 2.34% - - name: Q4_K_M - filename: nemotron-speech-streaming-en-0.6b-Q4_K_M.gguf - size: 453 MB - wer: 2.38% + Greedy RNN-T decoding. F32 reference baseline: 2.31%. NVIDIA's self-reported number + on the same split at att_context_size=[70, 13] (1.12s chunk, w/o PnC) is 2.32%. diff --git a/scripts/hf_cards/parakeet-ctc-0.6b.yaml b/scripts/hf_cards/parakeet-ctc-0.6b.yaml index 3b6c5cbe..4385f2f0 100644 --- a/scripts/hf_cards/parakeet-ctc-0.6b.yaml +++ b/scripts/hf_cards/parakeet-ctc-0.6b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-ctc-0.6b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-ctc-0.6b -target_repo: handy-computer/parakeet-ctc-0.6b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-ctc-0.6b.md - -upstream_commit: ad09ba1 pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -30,54 +21,9 @@ tags: - ctc summary: | - Offline English speech-to-text with greedy CTC decoding. A 0.6B-parameter FastConformer-Large encoder with a linear CTC head — the simplest and fastest decoder in the parakeet family. Output is lowercase, no punctuation. Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 218.5 - cpu: 30.5 - ryzen-4750u: - vulkan: 22.5 - cpu: 10 + Offline English speech-to-text with greedy CTC decoding. A FastConformer-Large encoder with a linear CTC head — the simplest and fastest decoder in the parakeet family. Output is lowercase, no punctuation. Not a streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy CTC decoding and no external LM. F32 reference baseline: 1.87%. NVIDIA's self-reported number on the same split is 1.87%. - -quants: - - name: F32 - filename: parakeet-ctc-0.6b-F32.gguf - size: 2.44 GB - wer: 1.87% - - name: F16 - filename: parakeet-ctc-0.6b-F16.gguf - size: 1.22 GB - wer: 1.87% - - name: Q8_0 - filename: parakeet-ctc-0.6b-Q8_0.gguf - size: 722 MB - wer: 1.87% - - name: Q6_K - filename: parakeet-ctc-0.6b-Q6_K.gguf - size: 594 MB - wer: 1.84% - - name: Q5_K_M - filename: parakeet-ctc-0.6b-Q5_K_M.gguf - size: 533 MB - wer: 1.87% - - name: Q4_K_M - filename: parakeet-ctc-0.6b-Q4_K_M.gguf - size: 469 MB - wer: 1.90% + Greedy CTC decoding, no external LM. F32 reference baseline: 1.87%. NVIDIA's + self-reported number on the same split is 1.87%. diff --git a/scripts/hf_cards/parakeet-ctc-1.1b.yaml b/scripts/hf_cards/parakeet-ctc-1.1b.yaml index dedc092b..a1dccfae 100644 --- a/scripts/hf_cards/parakeet-ctc-1.1b.yaml +++ b/scripts/hf_cards/parakeet-ctc-1.1b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-ctc-1.1b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-ctc-1.1b -target_repo: handy-computer/parakeet-ctc-1.1b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-ctc-1.1b.md - -upstream_commit: a707e81 pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -30,54 +21,9 @@ tags: - ctc summary: | - Offline English speech-to-text with greedy CTC decoding. A 1.1B-parameter FastConformer-XL encoder with a linear CTC head. Output is lowercase, no punctuation. Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 139.5 - cpu: 17.5 - ryzen-4750u: - vulkan: 14 - cpu: 6 + Offline English speech-to-text with greedy CTC decoding. A FastConformer-XL encoder with a linear CTC head. Output is lowercase, no punctuation. Not a streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy CTC decoding and no external LM. F32 reference baseline: 1.85%. NVIDIA's self-reported number on the same split is 1.83%. - -quants: - - name: F32 - filename: parakeet-ctc-1.1b-F32.gguf - size: 4.25 GB - wer: 1.85% - - name: F16 - filename: parakeet-ctc-1.1b-F16.gguf - size: 2.13 GB - wer: 1.85% - - name: Q8_0 - filename: parakeet-ctc-1.1b-Q8_0.gguf - size: 1.26 GB - wer: 1.85% - - name: Q6_K - filename: parakeet-ctc-1.1b-Q6_K.gguf - size: 1.04 GB - wer: 1.85% - - name: Q5_K_M - filename: parakeet-ctc-1.1b-Q5_K_M.gguf - size: 929 MB - wer: 1.84% - - name: Q4_K_M - filename: parakeet-ctc-1.1b-Q4_K_M.gguf - size: 818 MB - wer: 1.90% + Greedy CTC decoding, no external LM. F32 reference baseline: 1.85%. NVIDIA's + self-reported number on the same split is 1.83%. diff --git a/scripts/hf_cards/parakeet-primeline.yaml b/scripts/hf_cards/parakeet-primeline.yaml index 31c2812c..6f0c29d2 100644 --- a/scripts/hf_cards/parakeet-primeline.yaml +++ b/scripts/hf_cards/parakeet-primeline.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-primeline-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: primeline/parakeet-primeline -target_repo: handy-computer/parakeet-primeline-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-primeline.md - -upstream_commit: 3f1a9bc pin_date: 2026-08-16 validation: @@ -13,35 +8,7 @@ validation: commit: 856d7c1 date: 2026-08-16 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - de - - bg - - hr - - cs - - da - - nl - - en - - et - - fi - - fr - - el - - hu - - it - - lv - - lt - - mt - - pl - - pt - - ro - - ru - - sk - - sl - - es - - sv - - uk tags: - gguf - transcribe.cpp @@ -56,40 +23,16 @@ tags: summary: | primeLine's German fine-tune of NVIDIA's parakeet-tdt-0.6b-v3. A - 0.6B-parameter FastConformer encoder with a TDT/RNNT transducer decoder, - taking 16 kHz mono WAV and producing a punctuated, cased transcript with - optional token-level timestamps. Tuned for German, but the fine-tune did - not collapse the base model's multilingual ability: it still transcribes - the other 24 v3 languages with correct per-language casing and - punctuation. Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -metric: WER -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: true - timestamps: token # none | segment | word | token - -# Perf inherited from parakeet-tdt-0.6b-v3. The checkpoint is a -# weights-only fine-tune whose encoder/decoder/joint config is identical, -# so RTF is unchanged; not separately benched. -perf: - m4-max: - metal: 151 - cpu: 28 - ryzen-4750u: - vulkan: 12.5 - cpu: 7.5 + FastConformer encoder with a TDT/RNNT transducer decoder, taking 16 kHz + mono WAV and producing a punctuated, cased transcript with optional + token-level timestamps. Tuned for German, but the fine-tune did not + collapse the base model's multilingual ability: it still transcribes the + other 24 v3 languages with correct per-language casing and punctuation. + Not a streaming model and does not translate. wer: - metadata_key: fleurs_de - source: FLEURS de test notes: | - WER measured on the FLEURS German test split (862 utterances) with greedy - transducer decoding and no external LM. + Greedy transducer decoding, no external LM. primeLine's published figures (2.95% average over Tuda-De, Multilingual LibriSpeech, and Common Voice 19.0) are on different corpora and are not @@ -106,29 +49,3 @@ wer: produces the same spellings on the same utterances. FLEURS references use `ß` throughout, which costs roughly 1.05pp: folding `ß`→`ss` on both sides gives 4.92% for the reference and 4.94% for F32. - -quants: - - name: F32 - filename: parakeet-primeline-F32.gguf - size: 2.51 GB - wer: 6.00% - - name: F16 - filename: parakeet-primeline-F16.gguf - size: 1.26 GB - wer: 6.00% - - name: Q8_0 - filename: parakeet-primeline-Q8_0.gguf - size: 740 MB - wer: 6.00% - - name: Q6_K - filename: parakeet-primeline-Q6_K.gguf - size: 610 MB - wer: 5.96% - - name: Q5_K_M - filename: parakeet-primeline-Q5_K_M.gguf - size: 549 MB - wer: 5.99% - - name: Q4_K_M - filename: parakeet-primeline-Q4_K_M.gguf - size: 485 MB - wer: 5.98% diff --git a/scripts/hf_cards/parakeet-rnnt-0.6b.yaml b/scripts/hf_cards/parakeet-rnnt-0.6b.yaml index e423d54e..eb655071 100644 --- a/scripts/hf_cards/parakeet-rnnt-0.6b.yaml +++ b/scripts/hf_cards/parakeet-rnnt-0.6b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-rnnt-0.6b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-rnnt-0.6b -target_repo: handy-computer/parakeet-rnnt-0.6b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-rnnt-0.6b.md - -upstream_commit: c0c1f09 pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -30,54 +21,9 @@ tags: - rnnt summary: | - Offline English speech-to-text with greedy RNN-T decoding. A 0.6B-parameter FastConformer-Large encoder with an RNN-T transducer decoder. Output is lowercase, no punctuation. Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 185.5 - cpu: 30 - ryzen-4750u: - vulkan: 14.5 - cpu: 8 + Offline English speech-to-text with greedy RNN-T decoding. A FastConformer-Large encoder with an RNN-T transducer decoder. Output is lowercase, no punctuation. Not a streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy RNN-T decoding and no external LM. F32 reference baseline: 1.62%. NVIDIA's self-reported number on the same split is 1.63%. - -quants: - - name: F32 - filename: parakeet-rnnt-0.6b-F32.gguf - size: 2.47 GB - wer: 1.62% - - name: F16 - filename: parakeet-rnnt-0.6b-F16.gguf - size: 1.24 GB - wer: 1.62% - - name: Q8_0 - filename: parakeet-rnnt-0.6b-Q8_0.gguf - size: 730 MB - wer: 1.62% - - name: Q6_K - filename: parakeet-rnnt-0.6b-Q6_K.gguf - size: 601 MB - wer: 1.62% - - name: Q5_K_M - filename: parakeet-rnnt-0.6b-Q5_K_M.gguf - size: 540 MB - wer: 1.62% - - name: Q4_K_M - filename: parakeet-rnnt-0.6b-Q4_K_M.gguf - size: 476 MB - wer: 1.59% + Greedy RNN-T decoding, no external LM. F32 reference baseline: 1.62%. NVIDIA's + self-reported number on the same split is 1.63%. diff --git a/scripts/hf_cards/parakeet-rnnt-1.1b.yaml b/scripts/hf_cards/parakeet-rnnt-1.1b.yaml index 5a272b10..dde886cf 100644 --- a/scripts/hf_cards/parakeet-rnnt-1.1b.yaml +++ b/scripts/hf_cards/parakeet-rnnt-1.1b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-rnnt-1.1b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-rnnt-1.1b -target_repo: handy-computer/parakeet-rnnt-1.1b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-rnnt-1.1b.md - -upstream_commit: a07b19e pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -30,54 +21,9 @@ tags: - rnnt summary: | - Offline English speech-to-text with greedy RNN-T decoding. A 1.1B-parameter FastConformer-XL encoder with an RNN-T transducer decoder. Output is lowercase, no punctuation. Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 125.5 - cpu: 17.5 - ryzen-4750u: - vulkan: 11 - cpu: 5.5 + Offline English speech-to-text with greedy RNN-T decoding. A FastConformer-XL encoder with an RNN-T transducer decoder. Output is lowercase, no punctuation. Not a streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy RNN-T decoding and no external LM. F32 reference baseline: 1.45%. NVIDIA's self-reported number on the same split is 1.46%. - -quants: - - name: F32 - filename: parakeet-rnnt-1.1b-F32.gguf - size: 4.28 GB - wer: 1.45% - - name: F16 - filename: parakeet-rnnt-1.1b-F16.gguf - size: 2.15 GB - wer: 1.45% - - name: Q8_0 - filename: parakeet-rnnt-1.1b-Q8_0.gguf - size: 1.27 GB - wer: 1.46% - - name: Q6_K - filename: parakeet-rnnt-1.1b-Q6_K.gguf - size: 1.04 GB - wer: 1.43% - - name: Q5_K_M - filename: parakeet-rnnt-1.1b-Q5_K_M.gguf - size: 936 MB - wer: 1.43% - - name: Q4_K_M - filename: parakeet-rnnt-1.1b-Q4_K_M.gguf - size: 825 MB - wer: 1.41% + Greedy RNN-T decoding, no external LM. F32 reference baseline: 1.45%. NVIDIA's + self-reported number on the same split is 1.46%. diff --git a/scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml b/scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml index 68036aea..61c9af85 100644 --- a/scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml +++ b/scripts/hf_cards/parakeet-tdt-0.6b-v2.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-tdt-0.6b-v2-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-tdt-0.6b-v2 -target_repo: handy-computer/parakeet-tdt-0.6b-v2-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-tdt-0.6b-v2.md - -upstream_commit: 1b149a3 pin_date: 2026-04-15 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: bf0d0b7 date: 2026-04-18 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -31,61 +22,14 @@ tags: - tdt summary: | - Offline English speech-to-text. A 0.6B-parameter Conformer encoder with a - TDT/RNNT transducer decoder. Takes a 16 kHz mono WAV and produces a transcript - with optional token-level timestamps. Not a streaming model; no multilingual - capability (see v3 for that). - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, hand-authored from the bench; -# published raw as rtf_ in the metadata block. -perf: - ryzen-4750u: - cpu: 8 - vulkan: 15 - m4-max: - cpu: 29 - metal: 175 + Offline English speech-to-text. A Conformer encoder with a TDT/RNNT transducer + decoder. Takes a 16 kHz mono WAV and produces a transcript with optional + token-level timestamps. Not a streaming model; no multilingual capability (see + v3 for that). wer: - source: LibriSpeech test-clean # metadata_key: librispeech_test_clean # names the wer_ field; override when scored on another corpus notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with - greedy transducer decoding and no external LM. F32 reference baseline: 1.68%. - NVIDIA's self-reported number on the same split is 1.69%, so the F32 and Q8_0 - ports match the upstream reference within rounding. - -quants: - - name: F32 - filename: parakeet-tdt-0.6b-v2-F32.gguf - size: 2.47 GB - wer: 1.68% - - name: F16 - filename: parakeet-tdt-0.6b-v2-F16.gguf - size: 1.24 GB - wer: 1.68% - - name: Q8_0 - filename: parakeet-tdt-0.6b-v2-Q8_0.gguf - size: 730 MB - wer: 1.69% - - name: Q6_K - filename: parakeet-tdt-0.6b-v2-Q6_K.gguf - size: 608 MB - wer: 1.70% - - name: Q5_K_M - filename: parakeet-tdt-0.6b-v2-Q5_K_M.gguf - size: 547 MB - wer: 1.70% - - name: Q4_K_M - filename: parakeet-tdt-0.6b-v2-Q4_K_M.gguf - size: 483 MB - wer: 1.72% + Greedy transducer decoding, no external LM. F32 reference baseline: 1.68%. NVIDIA's + self-reported number on the same split is 1.69%, so the F32 and Q8_0 ports match the + upstream reference within rounding. diff --git a/scripts/hf_cards/parakeet-tdt-0.6b-v3.yaml b/scripts/hf_cards/parakeet-tdt-0.6b-v3.yaml index 77cc80f2..023160b1 100644 --- a/scripts/hf_cards/parakeet-tdt-0.6b-v3.yaml +++ b/scripts/hf_cards/parakeet-tdt-0.6b-v3.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-tdt-0.6b-v3-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-tdt-0.6b-v3 -target_repo: handy-computer/parakeet-tdt-0.6b-v3-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-tdt-0.6b-v3.md - -upstream_commit: 6d590f7 pin_date: 2026-04-16 # Validation pin for the most recent upload. Updated on each release — @@ -15,35 +10,7 @@ validation: commit: bf0d0b7 date: 2026-04-18 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - bg - - hr - - cs - - da - - nl - - en - - et - - fi - - fr - - de - - el - - hu - - it - - lv - - lt - - mt - - pl - - pt - - ro - - ru - - sk - - sl - - es - - sv - - uk tags: - gguf - transcribe.cpp @@ -57,58 +24,11 @@ tags: summary: | Offline multilingual speech-to-text covering 25 European languages. A - 0.6B-parameter Conformer encoder with a TDT/RNNT transducer decoder. Takes - a 16 kHz mono WAV and produces a transcript with optional token-level - timestamps. Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: true - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 151 - cpu: 28 - ryzen-4750u: - vulkan: 12.5 - cpu: 7.5 + Conformer encoder with a TDT/RNNT transducer decoder. Takes a 16 kHz mono + WAV and produces a transcript with optional token-level timestamps. Not a + streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with - greedy transducer decoding and no external LM. F32 reference baseline: 1.95%. - NVIDIA's self-reported number on the same split is 1.93%. - -quants: - - name: F32 - filename: parakeet-tdt-0.6b-v3-F32.gguf - size: 2.51 GB - wer: 1.95% - - name: F16 - filename: parakeet-tdt-0.6b-v3-F16.gguf - size: 1.26 GB - wer: 1.95% - - name: Q8_0 - filename: parakeet-tdt-0.6b-v3-Q8_0.gguf - size: 740 MB - wer: 1.94% - - name: Q6_K - filename: parakeet-tdt-0.6b-v3-Q6_K.gguf - size: 627 MB - wer: 1.93% - - name: Q5_K_M - filename: parakeet-tdt-0.6b-v3-Q5_K_M.gguf - size: 565 MB - wer: 1.92% - - name: Q4_K_M - filename: parakeet-tdt-0.6b-v3-Q4_K_M.gguf - size: 502 MB - wer: 1.98% + Greedy transducer decoding, no external LM. F32 reference baseline: 1.95%. NVIDIA's + self-reported number on the same split is 1.93%. diff --git a/scripts/hf_cards/parakeet-tdt-1.1b.yaml b/scripts/hf_cards/parakeet-tdt-1.1b.yaml index 2c9834b8..3311f92e 100644 --- a/scripts/hf_cards/parakeet-tdt-1.1b.yaml +++ b/scripts/hf_cards/parakeet-tdt-1.1b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-tdt-1.1b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-tdt-1.1b -target_repo: handy-computer/parakeet-tdt-1.1b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-tdt-1.1b.md - -upstream_commit: 53276c6 pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -31,54 +22,9 @@ tags: - tdt summary: | - Offline English speech-to-text. A 1.1B-parameter FastConformer-XL encoder with a TDT/RNNT transducer decoder. Takes a 16 kHz mono WAV and produces a transcript with optional token-level timestamps. Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 125.5 - cpu: 17.5 - ryzen-4750u: - vulkan: 11.5 - cpu: 5.5 + Offline English speech-to-text. A FastConformer-XL encoder with a TDT/RNNT transducer decoder. Takes a 16 kHz mono WAV and produces a transcript with optional token-level timestamps. Not a streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy TDT/RNN-T transducer decoding and no external LM. F32 reference baseline: 1.39%. NVIDIA's self-reported number on the same split is 1.39%. - -quants: - - name: F32 - filename: parakeet-tdt-1.1b-F32.gguf - size: 4.28 GB - wer: 1.39% - - name: F16 - filename: parakeet-tdt-1.1b-F16.gguf - size: 2.15 GB - wer: 1.39% - - name: Q8_0 - filename: parakeet-tdt-1.1b-Q8_0.gguf - size: 1.27 GB - wer: 1.38% - - name: Q6_K - filename: parakeet-tdt-1.1b-Q6_K.gguf - size: 1.04 GB - wer: 1.40% - - name: Q5_K_M - filename: parakeet-tdt-1.1b-Q5_K_M.gguf - size: 936 MB - wer: 1.39% - - name: Q4_K_M - filename: parakeet-tdt-1.1b-Q4_K_M.gguf - size: 825 MB - wer: 1.42% + Greedy TDT/RNN-T transducer decoding, no external LM. F32 reference baseline: 1.39%. + NVIDIA's self-reported number on the same split is 1.39%. diff --git a/scripts/hf_cards/parakeet-tdt_ctc-1.1b.yaml b/scripts/hf_cards/parakeet-tdt_ctc-1.1b.yaml index 9ff8ea1a..656c01d5 100644 --- a/scripts/hf_cards/parakeet-tdt_ctc-1.1b.yaml +++ b/scripts/hf_cards/parakeet-tdt_ctc-1.1b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-tdt_ctc-1.1b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-tdt_ctc-1.1b -target_repo: handy-computer/parakeet-tdt_ctc-1.1b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-tdt_ctc-1.1b.md - -upstream_commit: 675e786 pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -31,54 +22,9 @@ tags: - tdt summary: | - Offline English speech-to-text with punctuation and capitalization. A 1.1B-parameter FastConformer-XL encoder with a TDT/RNNT transducer decoder (the auxiliary CTC head from the upstream hybrid checkpoint is dropped at convert time). Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 125.5 - cpu: 18.5 - ryzen-4750u: - vulkan: 11 - cpu: 5.5 + Offline English speech-to-text with punctuation and capitalization. A FastConformer-XL encoder with a TDT/RNNT transducer decoder (the auxiliary CTC head from the upstream hybrid checkpoint is dropped at convert time). Not a streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy TDT/RNN-T transducer decoding and no external LM. F32 reference baseline: 1.87%. NVIDIA's self-reported number on the same split is 1.82%. - -quants: - - name: F32 - filename: parakeet-tdt_ctc-1.1b-F32.gguf - size: 4.28 GB - wer: 1.87% - - name: F16 - filename: parakeet-tdt_ctc-1.1b-F16.gguf - size: 2.15 GB - wer: 1.87% - - name: Q8_0 - filename: parakeet-tdt_ctc-1.1b-Q8_0.gguf - size: 1.27 GB - wer: 1.87% - - name: Q6_K - filename: parakeet-tdt_ctc-1.1b-Q6_K.gguf - size: 1.04 GB - wer: 1.87% - - name: Q5_K_M - filename: parakeet-tdt_ctc-1.1b-Q5_K_M.gguf - size: 936 MB - wer: 1.87% - - name: Q4_K_M - filename: parakeet-tdt_ctc-1.1b-Q4_K_M.gguf - size: 825 MB - wer: 1.91% + Greedy TDT/RNN-T transducer decoding, no external LM. F32 reference baseline: 1.87%. + NVIDIA's self-reported number on the same split is 1.82%. diff --git a/scripts/hf_cards/parakeet-tdt_ctc-110m.yaml b/scripts/hf_cards/parakeet-tdt_ctc-110m.yaml index ff44530e..5fc752f2 100644 --- a/scripts/hf_cards/parakeet-tdt_ctc-110m.yaml +++ b/scripts/hf_cards/parakeet-tdt_ctc-110m.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-tdt_ctc-110m-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-tdt_ctc-110m -target_repo: handy-computer/parakeet-tdt_ctc-110m-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-tdt_ctc-110m.md - -upstream_commit: 431a349 pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -31,54 +22,9 @@ tags: - tdt summary: | - Offline English speech-to-text with punctuation and capitalization. A 110M-parameter FastConformer encoder with a TDT/RNNT transducer decoder (the auxiliary CTC head from the upstream hybrid checkpoint is dropped at convert time). Not a streaming model and does not translate. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 347.5 - cpu: 116 - ryzen-4750u: - vulkan: 32.5 - cpu: 23.5 + Offline English speech-to-text with punctuation and capitalization. A FastConformer encoder with a TDT/RNNT transducer decoder (the auxiliary CTC head from the upstream hybrid checkpoint is dropped at convert time). Not a streaming model and does not translate. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy TDT/RNN-T transducer decoding and no external LM. F32 reference baseline: 2.43%. NVIDIA's self-reported number on the same split is 2.40%. - -quants: - - name: F32 - filename: parakeet-tdt_ctc-110m-F32.gguf - size: 457 MB - wer: 2.43% - - name: F16 - filename: parakeet-tdt_ctc-110m-F16.gguf - size: 229 MB - wer: 2.43% - - name: Q8_0 - filename: parakeet-tdt_ctc-110m-Q8_0.gguf - size: 135 MB - wer: 2.43% - - name: Q6_K - filename: parakeet-tdt_ctc-110m-Q6_K.gguf - size: 112 MB - wer: 2.44% - - name: Q5_K_M - filename: parakeet-tdt_ctc-110m-Q5_K_M.gguf - size: 101 MB - wer: 2.47% - - name: Q4_K_M - filename: parakeet-tdt_ctc-110m-Q4_K_M.gguf - size: 90 MB - wer: 2.53% + Greedy TDT/RNN-T transducer decoding, no external LM. F32 reference baseline: 2.43%. + NVIDIA's self-reported number on the same split is 2.40%. diff --git a/scripts/hf_cards/parakeet-unified-en-0.6b.yaml b/scripts/hf_cards/parakeet-unified-en-0.6b.yaml index 76827d67..35b9dada 100644 --- a/scripts/hf_cards/parakeet-unified-en-0.6b.yaml +++ b/scripts/hf_cards/parakeet-unified-en-0.6b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/parakeet-unified-en-0.6b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: nvidia/parakeet-unified-en-0.6b -target_repo: handy-computer/parakeet-unified-en-0.6b-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/parakeet-unified-en-0.6b.md - -upstream_commit: d4ac992 pin_date: 2026-05-10 # Validation pin for the most recent upload. Updated on each release — @@ -15,11 +10,7 @@ validation: commit: 42528dd date: 2026-05-10 -license: cc-by-4.0 -license_display: CC-BY-4.0 pipeline_tag: automatic-speech-recognition -languages: - - en tags: - gguf - transcribe.cpp @@ -30,54 +21,10 @@ tags: - rnnt summary: | - English speech-to-text with punctuation and capitalization. A 0.6B-parameter FastConformer encoder with an RNN-T transducer decoder, trained as a 'unified' streaming/offline model. This port runs the model in both offline and buffered streaming modes. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: false - timestamps: token # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 163 - cpu: 28.5 - ryzen-4750u: - vulkan: 12.5 - cpu: 7.5 + English speech-to-text with punctuation and capitalization. A FastConformer encoder with an RNN-T transducer decoder, trained as a 'unified' streaming/offline model. This port runs the model in both offline and buffered streaming modes. wer: source: LibriSpeech test-clean, offline notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with greedy RNN-T decoding and no external LM. F32 reference baseline: 1.59%. NVIDIA's self-reported number on the same split is 1.63%. - -quants: - - name: F32 - filename: parakeet-unified-en-0.6b-F32.gguf - size: 2.47 GB - wer: 1.59% - - name: F16 - filename: parakeet-unified-en-0.6b-F16.gguf - size: 1.24 GB - wer: 1.59% - - name: Q8_0 - filename: parakeet-unified-en-0.6b-Q8_0.gguf - size: 731 MB - wer: 1.60% - - name: Q6_K - filename: parakeet-unified-en-0.6b-Q6_K.gguf - size: 602 MB - wer: 1.61% - - name: Q5_K_M - filename: parakeet-unified-en-0.6b-Q5_K_M.gguf - size: 541 MB - wer: 1.58% - - name: Q4_K_M - filename: parakeet-unified-en-0.6b-Q4_K_M.gguf - size: 477 MB - wer: 1.62% + Greedy RNN-T decoding, no external LM. F32 reference baseline: 1.59%. NVIDIA's + self-reported number on the same split is 1.63%. diff --git a/scripts/hf_cards/qwen3-asr-0.6b.yaml b/scripts/hf_cards/qwen3-asr-0.6b.yaml index 230a585d..573a9fd4 100644 --- a/scripts/hf_cards/qwen3-asr-0.6b.yaml +++ b/scripts/hf_cards/qwen3-asr-0.6b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/qwen3-asr-0.6b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: Qwen/Qwen3-ASR-0.6B -target_repo: handy-computer/Qwen3-ASR-0.6B-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/qwen3-asr-0.6b.md - -upstream_commit: 5eb144179a02acc5e5ba31e748d22b0cf3e303b0 pin_date: 2026-04-19 # Validation pin for the most recent upload. Updated on each release — @@ -15,40 +10,7 @@ validation: commit: 3f61df7 date: 2026-04-20 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - zh - - en - - yue - - ar - - de - - fr - - es - - pt - - id - - it - - ko - - ru - - th - - vi - - ja - - tr - - hi - - ms - - nl - - sv - - da - - fi - - pl - - cs - - fil - - fa - - el - - ro - - hu - - mk tags: - gguf - transcribe.cpp @@ -65,58 +27,10 @@ summary: | across 30 languages and emits the transcript in that language. Takes a 16 kHz mono WAV; explicit language hints are not supported at this time. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: true - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 65 - cpu: 16.5 - ryzen-4750u: - vulkan: 8 - cpu: 4.3 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech `test-clean` split (2620 English - utterances) with the Whisper-style English text normalizer and jiwer 3.x, - on the metal backend of an Apple M4. Qwen3-ASR is a multilingual model — - this number characterizes the English case only. BF16 / F16 / Q8_0 / Q6_K - are all within bootstrap CI of each other; Q5_K_M and Q4_K_M show a small - but real regression driven by the tied token-embedding / head. Reproduce - with `scripts/wer/run.py` + `scripts/wer/score.py`. - -quants: - - name: BF16 - filename: Qwen3-ASR-0.6B-BF16.gguf - size: 1.46 GB - wer: 2.11% - - name: F16 - filename: Qwen3-ASR-0.6B-F16.gguf - size: 1.47 GB - wer: 2.12% - - name: Q8_0 - filename: Qwen3-ASR-0.6B-Q8_0.gguf - size: 811 MB - wer: 2.11% - - name: Q6_K - filename: Qwen3-ASR-0.6B-Q6_K.gguf - size: 763 MB - wer: 2.10% - - name: Q5_K_M - filename: Qwen3-ASR-0.6B-Q5_K_M.gguf - size: 707 MB - wer: 2.21% - - name: Q4_K_M - filename: Qwen3-ASR-0.6B-Q4_K_M.gguf - size: 654 MB - wer: 2.26% + Scored with the Whisper-style English text normalizer and jiwer 3.x on an Apple M4. + Qwen3-ASR is a multilingual model — this number characterizes the English case only. + BF16 / F16 / Q8_0 / Q6_K are all within bootstrap CI of each other; Q5_K_M and + Q4_K_M show a small but real regression driven by the tied token-embedding / head. + Reproduce with `scripts/wer/run.py` + `scripts/wer/score.py`. diff --git a/scripts/hf_cards/qwen3-asr-1.7b.yaml b/scripts/hf_cards/qwen3-asr-1.7b.yaml index 33cd6ebf..d37ae837 100644 --- a/scripts/hf_cards/qwen3-asr-1.7b.yaml +++ b/scripts/hf_cards/qwen3-asr-1.7b.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/qwen3-asr-1.7b-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: Qwen/Qwen3-ASR-1.7B -target_repo: handy-computer/Qwen3-ASR-1.7B-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/qwen3-asr-1.7b.md - -upstream_commit: 7278e1e70fe206f11671096ffdd38061171dd6e5 pin_date: 2026-04-19 # Validation pin for the most recent upload. Updated on each release — @@ -15,40 +10,7 @@ validation: commit: 3f61df7 date: 2026-04-20 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - zh - - en - - yue - - ar - - de - - fr - - es - - pt - - id - - it - - ko - - ru - - th - - vi - - ja - - tr - - hi - - ms - - nl - - sv - - da - - fi - - pl - - cs - - fil - - fa - - el - - ro - - hu - - mk tags: - gguf - transcribe.cpp @@ -67,59 +29,10 @@ summary: | Takes a 16 kHz mono WAV; explicit language hints are not supported at this time. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: true - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 39.5 - cpu: 8 - ryzen-4750u: - vulkan: 3.8 - cpu: 2 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech `test-clean` split (2620 English - utterances) with the Whisper-style English text normalizer and jiwer 3.x, - on the metal backend of an Apple M4. Qwen3-ASR is a multilingual model — - this number characterizes the English case only. The larger decoder - gives 1.7B more quantization headroom than the 0.6B; BF16 / F16 / Q8_0 / - Q6_K / Q5_K_M are all within bootstrap CI of each other, and Q4_K_M - regresses only ~0.2 WER points. Reproduce with `scripts/wer/run.py` + - `scripts/wer/score.py`. - -quants: - - name: BF16 - filename: Qwen3-ASR-1.7B-BF16.gguf - size: 3.80 GB - wer: 1.62% - - name: F16 - filename: Qwen3-ASR-1.7B-F16.gguf - size: 3.81 GB - wer: 1.62% - - name: Q8_0 - filename: Qwen3-ASR-1.7B-Q8_0.gguf - size: 2.04 GB - wer: 1.61% - - name: Q6_K - filename: Qwen3-ASR-1.7B-Q6_K.gguf - size: 1.58 GB - wer: 1.65% - - name: Q5_K_M - filename: Qwen3-ASR-1.7B-Q5_K_M.gguf - size: 1.41 GB - wer: 1.65% - - name: Q4_K_M - filename: Qwen3-ASR-1.7B-Q4_K_M.gguf - size: 1.23 GB - wer: 1.81% + Scored with the Whisper-style English text normalizer and jiwer 3.x on an Apple M4. + Qwen3-ASR is a multilingual model — this number characterizes the English case only. + The larger decoder gives 1.7B more quantization headroom than the 0.6B; BF16 / F16 / + Q8_0 / Q6_K / Q5_K_M are all within bootstrap CI of each other, and Q4_K_M regresses + only ~0.2 WER points. Reproduce with `scripts/wer/run.py` + `scripts/wer/score.py`. diff --git a/scripts/hf_cards/sensevoice-small.yaml b/scripts/hf_cards/sensevoice-small.yaml index c33a5119..e9df8062 100644 --- a/scripts/hf_cards/sensevoice-small.yaml +++ b/scripts/hf_cards/sensevoice-small.yaml @@ -1,12 +1,6 @@ # Spec for the HF README of handy-computer/SenseVoiceSmall-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: FunAudioLLM/SenseVoiceSmall -target_repo: handy-computer/SenseVoiceSmall-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/sensevoice-small.md - -# Pinned at intake (Stage 1). -upstream_commit: 3eb3b4eeffc2f2dde6051b853983753db33e35c3 pin_date: 2026-05-06 # Validation pin for the most recent upload. Updated on each release — @@ -16,20 +10,7 @@ validation: commit: f094d28 date: 2026-05-06 -# Upstream license is "model-license" (FunASR's MODEL_LICENSE), not a -# standard SPDX id. The HF YAML field accepts a free-form `other` plus a -# license_name / license_link override on the upstream card; the renderer -# emits `license: other` and surfaces the human-facing form via -# `license_display`. -license: other -license_display: model-license (FunASR MODEL_LICENSE) pipeline_tag: automatic-speech-recognition -languages: - - zh - - yue - - en - - ja - - ko tags: - gguf - transcribe.cpp @@ -42,71 +23,22 @@ tags: summary: | Offline multilingual speech-to-text in Chinese, Cantonese, English, Japanese, - and Korean. A 234M-parameter SAN-M encoder with a single CTC head over a - 25,055-token SentencePiece vocabulary. Takes a 16 kHz mono WAV (capped at - 30 seconds per call, per upstream's direct-inference contract) and produces - a transcript. Not a streaming model, no translation, no built-in long-form - chunking. The same CTC head also emits language-ID, simple emotion labels, - audio-event tags, and inverse-text-normalization control tags. These tags are - hidden unless `--raw-tokens` is passed. ITN is on by default for readable - casing, punctuation, and digits; pass `--no-itn` for upstream's spoken-form - output. - -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: true - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 289.5 - cpu: 51.5 - ryzen-4750u: - vulkan: 32.5 - cpu: 15.5 + and Korean. A SAN-M encoder with a single CTC head over a 25,055-token + SentencePiece vocabulary. Takes a 16 kHz mono WAV (capped at 30 seconds per + call, per upstream's direct-inference contract) and produces a transcript. Not + a streaming model, no translation, no built-in long-form chunking. The same CTC + head also emits language-ID, simple emotion labels, audio-event tags, and + inverse-text-normalization control tags. These tags are hidden unless + `--raw-tokens` is passed. ITN is on by default for readable casing, + punctuation, and digits; pass `--no-itn` for upstream's spoken-form output. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) - with greedy CTC decoding. The publisher does not report a numerical - LibriSpeech WER (the model card publishes scores only as PNG figures), so - the gate baseline is our own FunASR 1.3.1 reference run on the same - manifest: 3.13% (95% CI [2.93%, 3.34%]). transcribe.cpp's F32 port matches - that baseline within +0.002 percentage-points. LibriSpeech is an English - benchmark; SenseVoice's strongest case is Mandarin, and AISHELL-1 (CER) - is the recommended complementary check. These table values were measured - with ITN off, matching the FunASR reference; `scripts/wer/run.py` pins + Greedy CTC decoding. The publisher does not report a numerical LibriSpeech WER (the + model card publishes scores only as PNG figures), so the gate baseline is our own + FunASR 1.3.1 reference run on the same manifest: 3.13% (95% CI [2.93%, 3.34%]). + transcribe.cpp's F32 port matches that baseline within +0.002 percentage-points. + LibriSpeech is an English benchmark; SenseVoice's strongest case is Mandarin, and + AISHELL-1 (CER) is the recommended complementary check. These table values were + measured with ITN off, matching the FunASR reference; `scripts/wer/run.py` pins `--no-itn` so the benchmark does not inherit the runtime default. - -quants: - - name: F32 - filename: SenseVoiceSmall-F32.gguf - size: 893 MB - wer: 3.13% - - name: F16 - filename: SenseVoiceSmall-F16.gguf - size: 449 MB - wer: 3.13% - - name: Q8_0 - filename: SenseVoiceSmall-Q8_0.gguf - size: 241 MB - wer: 3.13% - - name: Q6_K - filename: SenseVoiceSmall-Q6_K.gguf - size: 187 MB - wer: 3.14% - - name: Q5_K_M - filename: SenseVoiceSmall-Q5_K_M.gguf - size: 164 MB - wer: 3.18% - - name: Q4_K_M - filename: SenseVoiceSmall-Q4_K_M.gguf - size: 139 MB - wer: 3.45% diff --git a/scripts/hf_cards/template.md.j2 b/scripts/hf_cards/template.md.j2 index cc165dbc..758be13a 100644 --- a/scripts/hf_cards/template.md.j2 +++ b/scripts/hf_cards/template.md.j2 @@ -14,6 +14,12 @@ language: {%- for lang in languages %} - {{ lang }} {%- endfor %} +{%- if languages_bcp47 %} +language_bcp47: +{%- for lang in languages_bcp47 %} + - {{ lang }} +{%- endfor %} +{%- endif %} tags: {%- for tag in tags %} - {{ tag }} @@ -42,13 +48,18 @@ on {{ validation.date }}. | Quantization | Download | Size | {{ metric|default("WER") }} ({{ wer.source }}){% if wer.source2 is defined %} | {{ metric2|default("CER") }} ({{ wer.source2 }}){% endif %} | | --- | --- | ---: | ---:{% if wer.source2 is defined %} | ---:{% endif %} | {%- for q in quants %} -| {{ q.name }} | [{{ q.filename }}](https://huggingface.co/{{ target_repo }}/resolve/main/{{ q.filename }}) | {{ q.size }} | {{ q.wer }}{% if wer.source2 is defined %} | {{ q.wer2 }}{% endif %} | +| {{ q.name }} | [{{ q.filename }}](https://huggingface.co/{{ target_repo }}/resolve/main/{{ q.filename }}) | {{ q.size }} | {{ q.wer | default("-") }}{% if wer.source2 is defined %} | {{ q.wer2 | default("-") }}{% endif %} | {%- endfor %} +{{ wer.recipe }} + {{ wer.notes }} ## Usage +{% if usage is defined -%} +{{ usage.rstrip() }} +{% else -%} Build transcribe.cpp from source: ```bash @@ -61,7 +72,7 @@ Run on a 16 kHz mono WAV: ```bash build/bin/transcribe-cli \ - -m {{ quants[default_quant_index].filename }} \ + -m {{ default_quant_filename }} \ input.wav ``` @@ -73,7 +84,7 @@ ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav See the [transcribe.cpp model page]({{ transcribe_docs_url }}) for performance numbers, numerical validation, and reproduction steps. - +{% endif %} ## License Inherited from the base model: **{{ license_display }}**. See the diff --git a/scripts/hf_cards/voxtral-mini-3b-2507.yaml b/scripts/hf_cards/voxtral-mini-3b-2507.yaml index 368b9808..443e5e1b 100644 --- a/scripts/hf_cards/voxtral-mini-3b-2507.yaml +++ b/scripts/hf_cards/voxtral-mini-3b-2507.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/Voxtral-Mini-3B-2507-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: mistralai/Voxtral-Mini-3B-2507 -target_repo: handy-computer/Voxtral-Mini-3B-2507-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/voxtral-mini-3b-2507.md - -upstream_commit: "3060fe3" pin_date: 2026-06-06 # Validation pin for the most recent upload. Updated on each release — @@ -15,18 +10,7 @@ validation: commit: 483c122 date: 2026-06-06 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - de - - es - - it - - pt - - nl - - hi tags: - gguf - transcribe.cpp @@ -46,58 +30,10 @@ summary: | same encoder, projector, log-mel frontend, and tekken tokenizer, with a 3B decoder in place of Mistral-Small-24B. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 14.9 - cpu: 2 - ryzen-4750u: - vulkan: 1.2 - cpu: 0.5 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with - the Whisper English text normalizer, batch size 8 on an NVIDIA L40S. - Same-machine HuggingFace transformers reference - (VoxtralForConditionalGeneration, BF16, attn_implementation=eager, greedy): - 1.87%; the BF16 GGUF matches within rounding. The BF16-vs-reference parity is - the family's tensor-level numerical gate — 43 checkpointed tensors within - tolerance, transcript byte-exact. - -quants: - - name: BF16 - filename: Voxtral-Mini-3B-2507-BF16.gguf - size: 9.37 GB - wer: 1.88% - - name: F16 - filename: Voxtral-Mini-3B-2507-F16.gguf - size: 9.38 GB - wer: 1.89% - - name: Q8_0 - filename: Voxtral-Mini-3B-2507-Q8_0.gguf - size: 5.00 GB - wer: 1.87% - - name: Q6_K - filename: Voxtral-Mini-3B-2507-Q6_K.gguf - size: 3.87 GB - wer: 1.87% - - name: Q5_K_M - filename: Voxtral-Mini-3B-2507-Q5_K_M.gguf - size: 3.46 GB - wer: 1.91% - - name: Q4_K_M - filename: Voxtral-Mini-3B-2507-Q4_K_M.gguf - size: 2.98 GB - wer: 1.94% + Scored with the Whisper English text normalizer on an NVIDIA L40S. Same-machine + HuggingFace transformers reference (VoxtralForConditionalGeneration, BF16, + attn_implementation=eager, greedy): 1.87%; the BF16 GGUF matches within rounding. + The BF16-vs-reference parity is the family's tensor-level numerical gate — 43 + checkpointed tensors within tolerance, transcript byte-exact. diff --git a/scripts/hf_cards/voxtral-mini-4b-realtime-2602.yaml b/scripts/hf_cards/voxtral-mini-4b-realtime-2602.yaml index 17ccce9c..8f6e7793 100644 --- a/scripts/hf_cards/voxtral-mini-4b-realtime-2602.yaml +++ b/scripts/hf_cards/voxtral-mini-4b-realtime-2602.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/Voxtral-Mini-4B-Realtime-2602-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: mistralai/Voxtral-Mini-4B-Realtime-2602 -target_repo: handy-computer/Voxtral-Mini-4B-Realtime-2602-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/voxtral-realtime.md - -upstream_commit: "2769294" pin_date: 2026-06-06 # Validation pin for the most recent upload. Updated on each release — @@ -15,23 +10,7 @@ validation: commit: 483c122 date: 2026-06-06 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - es - - de - - ru - - zh - - ja - - it - - pt - - nl - - ar - - hi - - ko tags: - gguf - transcribe.cpp @@ -54,60 +33,12 @@ summary: | from the offline Voxtral 2507 family — own arch, streaming frontend, causal encoder, additive audio fusion. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: true - translate: false - lang_detect: true - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 8.6 - cpu: 2.5 - ryzen-4750u: - vulkan: 0.9 - cpu: 0.6 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with - the Whisper English text normalizer, offline path at delay 6, batch size 8 - on an NVIDIA L40S. A delay-30 remeasurement is pending. Same-machine - HuggingFace transformers reference - (VoxtralRealtimeForConditionalGeneration, BF16, greedy): 2.08%; the BF16 GGUF - matches at 2.08%. Every shipped quant stays within bootstrap noise - (2.07-2.09%), so the quantization ladder is WER-neutral down to Q4_K_M. The - model is multilingual (13 languages, auto-detect); the published WER is - English only. - -quants: - - name: BF16 - filename: Voxtral-Mini-4B-Realtime-2602-BF16.gguf - size: 8.87 GB - wer: 2.08% - - name: F16 - filename: Voxtral-Mini-4B-Realtime-2602-F16.gguf - size: 8.88 GB - wer: 2.09% - - name: Q8_0 - filename: Voxtral-Mini-4B-Realtime-2602-Q8_0.gguf - size: 4.73 GB - wer: 2.07% - - name: Q6_K - filename: Voxtral-Mini-4B-Realtime-2602-Q6_K.gguf - size: 3.66 GB - wer: 2.08% - - name: Q5_K_M - filename: Voxtral-Mini-4B-Realtime-2602-Q5_K_M.gguf - size: 3.28 GB - wer: 2.08% - - name: Q4_K_M - filename: Voxtral-Mini-4B-Realtime-2602-Q4_K_M.gguf - size: 2.83 GB - wer: 2.08% + Scored with the Whisper English text normalizer, offline path at delay 6, on an + NVIDIA L40S. A delay-30 remeasurement is pending. Same-machine HuggingFace + transformers reference (VoxtralRealtimeForConditionalGeneration, BF16, greedy): + 2.08%; the BF16 GGUF matches at 2.08%. Every shipped quant stays within bootstrap + noise (2.07-2.09%), so the quantization ladder is WER-neutral down to Q4_K_M. The + model is multilingual (13 languages, auto-detect); the published WER is English + only. diff --git a/scripts/hf_cards/voxtral-small-24b-2507.yaml b/scripts/hf_cards/voxtral-small-24b-2507.yaml index 85b5f481..a8dcf8b8 100644 --- a/scripts/hf_cards/voxtral-small-24b-2507.yaml +++ b/scripts/hf_cards/voxtral-small-24b-2507.yaml @@ -1,11 +1,6 @@ # Spec for the HF README of handy-computer/Voxtral-Small-24B-2507-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: mistralai/Voxtral-Small-24B-2507 -target_repo: handy-computer/Voxtral-Small-24B-2507-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/voxtral-small-24b-2507.md - -upstream_commit: da5b424 pin_date: 2026-06-05 # Validation pin for the most recent upload. Updated on each release — @@ -15,18 +10,7 @@ validation: commit: dac22fa date: 2026-06-05 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - en - - fr - - de - - es - - it - - pt - - nl - - hi tags: - gguf - transcribe.cpp @@ -44,54 +28,11 @@ summary: | transcript via greedy decoding. The larger sibling of Voxtral Mini 3B — same encoder, projector, frontend, and tokenizer, with a scaled-up decoder. -default_quant_index: 2 # Q8_0 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: none # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 3.2 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with - the Whisper English text normalizer, greedy decoding, batch size 8 on an - NVIDIA A100 80 GB. Same-machine HuggingFace transformers reference - (VoxtralForConditionalGeneration, BF16, greedy): 1.57%; the BF16 GGUF matches - at 1.56%. Validation for this variant is end-to-end by WER — the family's - tensor-level numerical parity is established by the Voxtral Mini 3B sibling - (identical architecture). - -quants: - - name: BF16 - filename: Voxtral-Small-24B-2507-BF16.gguf - size: 48.54 GB - wer: 1.56% - - name: F16 - filename: Voxtral-Small-24B-2507-F16.gguf - size: 48.55 GB - wer: 1.57% - - name: Q8_0 - filename: Voxtral-Small-24B-2507-Q8_0.gguf - size: 25.81 GB - wer: 1.56% - - name: Q6_K - filename: Voxtral-Small-24B-2507-Q6_K.gguf - size: 19.94 GB - wer: 1.58% - - name: Q5_K_M - filename: Voxtral-Small-24B-2507-Q5_K_M.gguf - size: 17.14 GB - wer: 1.60% - - name: Q4_K_M - filename: Voxtral-Small-24B-2507-Q4_K_M.gguf - size: 14.30 GB - wer: 2.11% + Greedy decoding scored with the Whisper English text normalizer on an NVIDIA A100 80 + GB. Same-machine HuggingFace transformers reference + (VoxtralForConditionalGeneration, BF16, greedy): 1.57%; the BF16 GGUF matches at + 1.56%. Validation for this variant is end-to-end by WER — the family's tensor-level + numerical parity is established by the Voxtral Mini 3B sibling (identical + architecture). diff --git a/scripts/hf_cards/whisper-base.en.yaml b/scripts/hf_cards/whisper-base.en.yaml index 5a8d8489..31a0f6e2 100644 --- a/scripts/hf_cards/whisper-base.en.yaml +++ b/scripts/hf_cards/whisper-base.en.yaml @@ -1,23 +1,14 @@ # Spec for the HF README of handy-computer/whisper-base.en-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-base.en -target_repo: handy-computer/whisper-base.en-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-base.en.md - -upstream_commit: 911407f pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" tags: - gguf - transcribe.cpp @@ -30,52 +21,10 @@ tags: summary: | OpenAI Whisper base.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 211.9 - cpu: 38.2 - ryzen-4750u: - vulkan: 36.2 - cpu: 12.8 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 4.25%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-base.en-F32.gguf - size: 279 MB - wer: 4.14% - - name: F16 - filename: whisper-base.en-F16.gguf - size: 144 MB - wer: 4.13% - - name: Q8_0 - filename: whisper-base.en-Q8_0.gguf - size: 81 MB - wer: 4.16% - - name: Q6_K - filename: whisper-base.en-Q6_K.gguf - size: 65 MB - wer: 4.15% - - name: Q5_K_M - filename: whisper-base.en-Q5_K_M.gguf - size: 61 MB - wer: 4.16% - - name: Q4_K_M - filename: whisper-base.en-Q4_K_M.gguf - size: 56 MB - wer: 4.29% + OpenAI's self-reported number on the same split is 4.25%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-base.yaml b/scripts/hf_cards/whisper-base.yaml index 75cf6650..e834f9c7 100644 --- a/scripts/hf_cards/whisper-base.yaml +++ b/scripts/hf_cards/whisper-base.yaml @@ -1,121 +1,14 @@ # Spec for the HF README of handy-computer/whisper-base-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-base -target_repo: handy-computer/whisper-base-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-base.md - -upstream_commit: e37978b pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" tags: - gguf - transcribe.cpp @@ -127,52 +20,10 @@ tags: summary: | OpenAI Whisper base — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 209.4 - cpu: 36.6 - ryzen-4750u: - vulkan: 34.8 - cpu: 11.9 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 5.009%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-base-F32.gguf - size: 279 MB - wer: 5.10% - - name: F16 - filename: whisper-base-F16.gguf - size: 144 MB - wer: 5.10% - - name: Q8_0 - filename: whisper-base-Q8_0.gguf - size: 81 MB - wer: 5.12% - - name: Q6_K - filename: whisper-base-Q6_K.gguf - size: 65 MB - wer: 5.12% - - name: Q5_K_M - filename: whisper-base-Q5_K_M.gguf - size: 61 MB - wer: 5.19% - - name: Q4_K_M - filename: whisper-base-Q4_K_M.gguf - size: 56 MB - wer: 5.36% + OpenAI's self-reported number on the same split is 5.009%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-large-v2.yaml b/scripts/hf_cards/whisper-large-v2.yaml index 67f57d01..54f9e37c 100644 --- a/scripts/hf_cards/whisper-large-v2.yaml +++ b/scripts/hf_cards/whisper-large-v2.yaml @@ -1,121 +1,14 @@ # Spec for the HF README of handy-computer/whisper-large-v2-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-large-v2 -target_repo: handy-computer/whisper-large-v2-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-large-v2.md - -upstream_commit: ae46427 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" tags: - gguf - transcribe.cpp @@ -127,52 +20,10 @@ tags: summary: | OpenAI Whisper large-v2 — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 23.7 - cpu: 1.5 - ryzen-4750u: - vulkan: 2.1 - cpu: 0.6 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 2.83%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-large-v2-F32.gguf - size: 5.75 GB - wer: 2.68% - - name: F16 - filename: whisper-large-v2-F16.gguf - size: 2.89 GB - wer: 2.68% - - name: Q8_0 - filename: whisper-large-v2-Q8_0.gguf - size: 1.55 GB - wer: 2.65% - - name: Q6_K - filename: whisper-large-v2-Q6_K.gguf - size: 1.21 GB - wer: 2.83% - - name: Q5_K_M - filename: whisper-large-v2-Q5_K_M.gguf - size: 1.08 GB - wer: 2.72% - - name: Q4_K_M - filename: whisper-large-v2-Q4_K_M.gguf - size: 950 MB - wer: 2.46% + OpenAI's self-reported number on the same split is 2.83%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-large-v3-turbo.yaml b/scripts/hf_cards/whisper-large-v3-turbo.yaml index 7420cd9f..d103a946 100644 --- a/scripts/hf_cards/whisper-large-v3-turbo.yaml +++ b/scripts/hf_cards/whisper-large-v3-turbo.yaml @@ -1,122 +1,14 @@ # Spec for the HF README of handy-computer/whisper-large-v3-turbo-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-large-v3-turbo -target_repo: handy-computer/whisper-large-v3-turbo-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-large-v3-turbo.md - -upstream_commit: 41f01f3 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" - - "yue" tags: - gguf - transcribe.cpp @@ -126,50 +18,12 @@ tags: - openai summary: | - OpenAI Whisper large-v3-turbo — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). v3 family adds Cantonese (yue) and uses a 128-bin mel input. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. - -default_quant_index: 1 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 46.4 - cpu: 1.8 - ryzen-4750u: - vulkan: 3.4 - cpu: 0.8 + OpenAI Whisper large-v3-turbo — converted to GGUF for transcribe.cpp. Multilingual transcription and language detection; unlike the full large-v3 model, this turbo variant does not support speech translation. The v3 family adds Cantonese (yue) and uses a 128-bin mel input. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 2.10%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F16 - filename: whisper-large-v3-turbo-F16.gguf - size: 1.51 GB - wer: 2.01% - - name: Q8_0 - filename: whisper-large-v3-turbo-Q8_0.gguf - size: 845 MB - wer: 2.01% - - name: Q6_K - filename: whisper-large-v3-turbo-Q6_K.gguf - size: 660 MB - wer: 2.01% - - name: Q5_K_M - filename: whisper-large-v3-turbo-Q5_K_M.gguf - size: 591 MB - wer: 2.03% - - name: Q4_K_M - filename: whisper-large-v3-turbo-Q4_K_M.gguf - size: 511 MB - wer: 2.04% + OpenAI's self-reported number on the same split is 2.10%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-large-v3.yaml b/scripts/hf_cards/whisper-large-v3.yaml index 8ae870c8..dda6a5fd 100644 --- a/scripts/hf_cards/whisper-large-v3.yaml +++ b/scripts/hf_cards/whisper-large-v3.yaml @@ -1,122 +1,14 @@ # Spec for the HF README of handy-computer/whisper-large-v3-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-large-v3 -target_repo: handy-computer/whisper-large-v3-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-large-v3.md - -upstream_commit: 06f233f pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" - - "yue" tags: - gguf - transcribe.cpp @@ -128,48 +20,10 @@ tags: summary: | OpenAI Whisper large-v3 — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). v3 family adds Cantonese (yue) and uses a 128-bin mel input. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 1 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 23.6 - cpu: 1.5 - ryzen-4750u: - vulkan: 2.1 - cpu: 0.6 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 2.01%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F16 - filename: whisper-large-v3-F16.gguf - size: 2.88 GB - wer: 1.81% - - name: Q8_0 - filename: whisper-large-v3-Q8_0.gguf - size: 1.55 GB - wer: 1.82% - - name: Q6_K - filename: whisper-large-v3-Q6_K.gguf - size: 1.21 GB - wer: 1.83% - - name: Q5_K_M - filename: whisper-large-v3-Q5_K_M.gguf - size: 1.08 GB - wer: 1.84% - - name: Q4_K_M - filename: whisper-large-v3-Q4_K_M.gguf - size: 951 MB - wer: 1.86% + OpenAI's self-reported number on the same split is 2.01%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-large.yaml b/scripts/hf_cards/whisper-large.yaml index d990dc81..7fcea267 100644 --- a/scripts/hf_cards/whisper-large.yaml +++ b/scripts/hf_cards/whisper-large.yaml @@ -1,121 +1,14 @@ # Spec for the HF README of handy-computer/whisper-large-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-large -target_repo: handy-computer/whisper-large-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-large.md - -upstream_commit: 4ef9b41 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" tags: - gguf - transcribe.cpp @@ -127,52 +20,10 @@ tags: summary: | OpenAI Whisper large — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 24.9 - cpu: 1.5 - ryzen-4750u: - vulkan: 2.1 - cpu: 0.5 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 2.73%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-large-F32.gguf - size: 5.75 GB - wer: 2.72% - - name: F16 - filename: whisper-large-F16.gguf - size: 2.89 GB - wer: 2.74% - - name: Q8_0 - filename: whisper-large-Q8_0.gguf - size: 1.55 GB - wer: 2.74% - - name: Q6_K - filename: whisper-large-Q6_K.gguf - size: 1.21 GB - wer: 2.62% - - name: Q5_K_M - filename: whisper-large-Q5_K_M.gguf - size: 1.08 GB - wer: 2.70% - - name: Q4_K_M - filename: whisper-large-Q4_K_M.gguf - size: 950 MB - wer: 2.67% + OpenAI's self-reported number on the same split is 2.73%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-medium.en.yaml b/scripts/hf_cards/whisper-medium.en.yaml index be11f851..8b852483 100644 --- a/scripts/hf_cards/whisper-medium.en.yaml +++ b/scripts/hf_cards/whisper-medium.en.yaml @@ -1,23 +1,14 @@ # Spec for the HF README of handy-computer/whisper-medium.en-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-medium.en -target_repo: handy-computer/whisper-medium.en-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-medium.en.md - -upstream_commit: 2e98eb6 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" tags: - gguf - transcribe.cpp @@ -30,52 +21,10 @@ tags: summary: | OpenAI Whisper medium.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 45.1 - cpu: 3.2 - ryzen-4750u: - vulkan: 4.6 - cpu: 1.1 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 3.02%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-medium.en-F32.gguf - size: 2.85 GB - wer: 2.74% - - name: F16 - filename: whisper-medium.en-F16.gguf - size: 1.44 GB - wer: 2.73% - - name: Q8_0 - filename: whisper-medium.en-Q8_0.gguf - size: 793 MB - wer: 2.72% - - name: Q6_K - filename: whisper-medium.en-Q6_K.gguf - size: 618 MB - wer: 2.83% - - name: Q5_K_M - filename: whisper-medium.en-Q5_K_M.gguf - size: 556 MB - wer: 2.74% - - name: Q4_K_M - filename: whisper-medium.en-Q4_K_M.gguf - size: 481 MB - wer: 2.91% + OpenAI's self-reported number on the same split is 3.02%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-medium.yaml b/scripts/hf_cards/whisper-medium.yaml index 57a36d43..bdfb94fb 100644 --- a/scripts/hf_cards/whisper-medium.yaml +++ b/scripts/hf_cards/whisper-medium.yaml @@ -1,121 +1,14 @@ # Spec for the HF README of handy-computer/whisper-medium-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-medium -target_repo: handy-computer/whisper-medium-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-medium.md - -upstream_commit: abdf7c3 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" tags: - gguf - transcribe.cpp @@ -127,52 +20,10 @@ tags: summary: | OpenAI Whisper medium — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 41.6 - cpu: 3 - ryzen-4750u: - vulkan: 4.3 - cpu: 1.1 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 2.90%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-medium-F32.gguf - size: 2.85 GB - wer: 2.63% - - name: F16 - filename: whisper-medium-F16.gguf - size: 1.44 GB - wer: 2.63% - - name: Q8_0 - filename: whisper-medium-Q8_0.gguf - size: 793 MB - wer: 2.64% - - name: Q6_K - filename: whisper-medium-Q6_K.gguf - size: 618 MB - wer: 2.59% - - name: Q5_K_M - filename: whisper-medium-Q5_K_M.gguf - size: 556 MB - wer: 2.62% - - name: Q4_K_M - filename: whisper-medium-Q4_K_M.gguf - size: 481 MB - wer: 2.59% + OpenAI's self-reported number on the same split is 2.90%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-small.en.yaml b/scripts/hf_cards/whisper-small.en.yaml index 8d05e29c..e380745b 100644 --- a/scripts/hf_cards/whisper-small.en.yaml +++ b/scripts/hf_cards/whisper-small.en.yaml @@ -1,23 +1,14 @@ # Spec for the HF README of handy-computer/whisper-small.en-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-small.en -target_repo: handy-computer/whisper-small.en-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-small.en.md - -upstream_commit: e872752 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" tags: - gguf - transcribe.cpp @@ -30,52 +21,10 @@ tags: summary: | OpenAI Whisper small.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 104.4 - cpu: 10.3 - ryzen-4750u: - vulkan: 12.8 - cpu: 3.6 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 3.05%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-small.en-F32.gguf - size: 924 MB - wer: 3.09% - - name: F16 - filename: whisper-small.en-F16.gguf - size: 470 MB - wer: 2.97% - - name: Q8_0 - filename: whisper-small.en-Q8_0.gguf - size: 257 MB - wer: 3.09% - - name: Q6_K - filename: whisper-small.en-Q6_K.gguf - size: 202 MB - wer: 2.97% - - name: Q5_K_M - filename: whisper-small.en-Q5_K_M.gguf - size: 185 MB - wer: 3.12% - - name: Q4_K_M - filename: whisper-small.en-Q4_K_M.gguf - size: 164 MB - wer: 3.08% + OpenAI's self-reported number on the same split is 3.05%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-small.yaml b/scripts/hf_cards/whisper-small.yaml index cb8d920c..624651eb 100644 --- a/scripts/hf_cards/whisper-small.yaml +++ b/scripts/hf_cards/whisper-small.yaml @@ -1,121 +1,14 @@ # Spec for the HF README of handy-computer/whisper-small-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-small -target_repo: handy-computer/whisper-small-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-small.md - -upstream_commit: 973afd2 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" tags: - gguf - transcribe.cpp @@ -127,52 +20,10 @@ tags: summary: | OpenAI Whisper small — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 99.2 - cpu: 9.8 - ryzen-4750u: - vulkan: 12.1 - cpu: 3.4 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 3.432%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-small-F32.gguf - size: 924 MB - wer: 3.33% - - name: F16 - filename: whisper-small-F16.gguf - size: 470 MB - wer: 3.34% - - name: Q8_0 - filename: whisper-small-Q8_0.gguf - size: 257 MB - wer: 3.33% - - name: Q6_K - filename: whisper-small-Q6_K.gguf - size: 202 MB - wer: 3.33% - - name: Q5_K_M - filename: whisper-small-Q5_K_M.gguf - size: 185 MB - wer: 3.37% - - name: Q4_K_M - filename: whisper-small-Q4_K_M.gguf - size: 164 MB - wer: 3.40% + OpenAI's self-reported number on the same split is 3.432%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-tiny.en.yaml b/scripts/hf_cards/whisper-tiny.en.yaml index 29779cee..e877df78 100644 --- a/scripts/hf_cards/whisper-tiny.en.yaml +++ b/scripts/hf_cards/whisper-tiny.en.yaml @@ -1,23 +1,14 @@ # Spec for the HF README of handy-computer/whisper-tiny.en-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-tiny.en -target_repo: handy-computer/whisper-tiny.en-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-tiny.en.md - -upstream_commit: 87c7102 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" tags: - gguf - transcribe.cpp @@ -30,52 +21,10 @@ tags: summary: | OpenAI Whisper tiny.en — converted to GGUF for transcribe.cpp. English-only; faster than the multilingual model at the same size. Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: false - lang_detect: false - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 279.8 - cpu: 78.7 - ryzen-4750u: - vulkan: 60.7 - cpu: 26.1 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 5.66%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-tiny.en-F32.gguf - size: 146 MB - wer: 5.77% - - name: F16 - filename: whisper-tiny.en-F16.gguf - size: 76 MB - wer: 5.77% - - name: Q8_0 - filename: whisper-tiny.en-Q8_0.gguf - size: 44 MB - wer: 5.72% - - name: Q6_K - filename: whisper-tiny.en-Q6_K.gguf - size: 43 MB - wer: 5.80% - - name: Q5_K_M - filename: whisper-tiny.en-Q5_K_M.gguf - size: 42 MB - wer: 5.89% - - name: Q4_K_M - filename: whisper-tiny.en-Q4_K_M.gguf - size: 42 MB - wer: 5.99% + OpenAI's self-reported number on the same split is 5.66%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/hf_cards/whisper-tiny.yaml b/scripts/hf_cards/whisper-tiny.yaml index cbedb7f3..c612ced6 100644 --- a/scripts/hf_cards/whisper-tiny.yaml +++ b/scripts/hf_cards/whisper-tiny.yaml @@ -1,121 +1,14 @@ # Spec for the HF README of handy-computer/whisper-tiny-gguf. -# Consumed by scripts/hf_cards/generate.py. +# Prose only; numbers and metadata come from catalog/.json. See README.md. -hf_repo: openai/whisper-tiny -target_repo: handy-computer/whisper-tiny-gguf -transcribe_docs_url: https://github.com/handy-computer/transcribe.cpp/blob/main/docs/models/whisper-tiny.md - -upstream_commit: 169d4a4 pin_date: 2026-04-25 validation: reference: transformers - commit: 5.6.1 - date: 2026-04-26 + commit: 0a26478 + date: 2026-09-13 -license: apache-2.0 -license_display: Apache-2.0 pipeline_tag: automatic-speech-recognition -languages: - - "en" - - "zh" - - "de" - - "es" - - "ru" - - "ko" - - "fr" - - "ja" - - "pt" - - "tr" - - "pl" - - "ca" - - "nl" - - "ar" - - "sv" - - "it" - - "id" - - "hi" - - "fi" - - "vi" - - "he" - - "uk" - - "el" - - "ms" - - "cs" - - "ro" - - "da" - - "hu" - - "ta" - - "no" - - "th" - - "ur" - - "hr" - - "bg" - - "lt" - - "la" - - "mi" - - "ml" - - "cy" - - "sk" - - "te" - - "fa" - - "lv" - - "bn" - - "sr" - - "az" - - "sl" - - "kn" - - "et" - - "mk" - - "br" - - "eu" - - "is" - - "hy" - - "ne" - - "mn" - - "bs" - - "kk" - - "sq" - - "sw" - - "gl" - - "mr" - - "pa" - - "si" - - "km" - - "sn" - - "yo" - - "so" - - "af" - - "oc" - - "ka" - - "be" - - "tg" - - "sd" - - "gu" - - "am" - - "yi" - - "lo" - - "uz" - - "fo" - - "ht" - - "ps" - - "tk" - - "nn" - - "mt" - - "sa" - - "lb" - - "my" - - "bo" - - "tl" - - "mg" - - "as" - - "tt" - - "haw" - - "ln" - - "ha" - - "ba" - - "jw" - - "su" tags: - gguf - transcribe.cpp @@ -127,52 +20,10 @@ tags: summary: | OpenAI Whisper tiny — converted to GGUF for transcribe.cpp. Multilingual transcription, language detection, and speech translation (audio in any supported language → English text). Encoder-decoder transformer; 30-second windows with chunked long-form decoding. -default_quant_index: 2 - -# Capability flags for the transcribe_cpp metadata block. -capabilities: - streaming: false - translate: true - lang_detect: true - timestamps: segment # none | segment | word | token - -# Speedup-over-realtime (×RT) per rig/backend, from docs/models; published -# raw as rtf_ in the metadata block. -perf: - m4-max: - metal: 302.1 - cpu: 76.1 - ryzen-4750u: - vulkan: 61 - cpu: 24.8 - wer: - source: LibriSpeech test-clean notes: | - WER measured on the full LibriSpeech test-clean split (2620 utterances) with the transcribe.cpp default decode (greedy, suppress_tokens, temperature fallback, segment timestamps enabled). OpenAI's self-reported number on the same split is 7.54%. We don't know upstream's exact eval config, but the most likely cause of any divergence is that OpenAI's `model.generate()` defaults to `<|notimestamps|>` while transcribe.cpp's pipeline runs with timestamps enabled. Numbers come from a single Metal-backed run; Metal's non-deterministic parallel reductions can shift corpus WER by ~0.1pp between runs, mostly driven by short-clip hallucination outcomes on the noise floor. - -quants: - - name: F32 - filename: whisper-tiny-F32.gguf - size: 146 MB - wer: 7.54% - - name: F16 - filename: whisper-tiny-F16.gguf - size: 76 MB - wer: 7.49% - - name: Q8_0 - filename: whisper-tiny-Q8_0.gguf - size: 44 MB - wer: 7.53% - - name: Q6_K - filename: whisper-tiny-Q6_K.gguf - size: 43 MB - wer: 7.63% - - name: Q5_K_M - filename: whisper-tiny-Q5_K_M.gguf - size: 42 MB - wer: 7.63% - - name: Q4_K_M - filename: whisper-tiny-Q4_K_M.gguf - size: 42 MB - wer: 7.76% + OpenAI's self-reported number on the same split is 7.54%. Both are + short-form WER decoded without timestamps; OpenAI does not publish its exact + evaluation configuration, so small differences are expected. Single-run + figures: GPU reductions can shift corpus WER by about 0.1pp between runs, + mostly on short-clip hallucination outcomes at the noise floor. diff --git a/scripts/lib/gguf_common.py b/scripts/lib/gguf_common.py index 6b6b60d1..98091f6f 100644 --- a/scripts/lib/gguf_common.py +++ b/scripts/lib/gguf_common.py @@ -323,13 +323,47 @@ def move_bulk_metadata_last(writer) -> list[str]: return moved +# The capability KVs the loader reads (read_capability_kv in +# src/transcribe-meta.cpp). Absence is not falsity there: a missing key leaves +# the family default in place, which is how a model that does not translate +# came to advertise that it does. Every file therefore states all four. +CAPABILITY_KV_KEYS = ( + "stt.capability.translate", + "stt.capability.lang_detect", + "stt.capability.streaming", + "stt.capability.speaker_diarization", +) + + +def declare_missing_capabilities(writer) -> list[str]: + """Write `false` for any capability KV the converter did not set. + + False is the honest default: a converter that knows a model translates, + streams, or diarizes says so explicitly, and every family whose C++ default + is true already writes that key. Returns the keys filled in. + """ + filled = [] + for key in CAPABILITY_KV_KEYS: + if any(key in shard for shard in writer.kv_data): + continue + writer.kv_data[0][key] = gguf.GGUFValue(value=False, type=gguf.GGUFValueType.BOOL) + filled.append(key) + return filled + + class _BulkLastGGUFWriter(gguf.GGUFWriter): - """GGUFWriter that relocates the bulk tokenizer KVs to the trailer at write - time. Hooked at write_kv_data_to_file (not write_header_to_file): the header - pass calls add_shard_kv_data(), which appends split.* scalar KVs, so we must - reorder *after* those are present but immediately before the KV dict is - serialized. The header only writes the KV count, which reordering leaves - unchanged.""" + """GGUFWriter that declares every capability KV and relocates the bulk + tokenizer KVs to the trailer at write time. Hooked at write_kv_data_to_file + (not write_header_to_file): the header pass calls add_shard_kv_data(), which + appends split.* scalar KVs, so we must reorder *after* those are present but + immediately before the KV dict is serialized. The header only writes the KV + count, so the capability fill runs before it and the reorder after.""" + + def write_header_to_file(self, path=None) -> None: + # Before the header: it writes the KV count, so a key added later + # would be silently dropped by every reader. + declare_missing_capabilities(self) + super().write_header_to_file(path) def write_kv_data_to_file(self) -> None: move_bulk_metadata_last(self) diff --git a/scripts/wer/ingest.py b/scripts/wer/ingest.py index a6924b83..fe03c420 100644 --- a/scripts/wer/ingest.py +++ b/scripts/wer/ingest.py @@ -44,60 +44,7 @@ import numpy as np import soundfile as sf - -# -------- BCP-47 → FLEURS config mapping --------------------------------- -# -# FLEURS uses underscore-region codes. Region-ambiguous BCP-47 codes -# (zh, no) pick a sensible default; explicit regional aliases live -# alongside (zh-cn). FLEURS only ships one regional variant for -# most languages, so most entries are unambiguous. -# -# Full FLEURS coverage (102 languages). Add aliases here when a new -# BCP-47 → config shorthand is wanted. -FLEURS_LANGS: dict[str, str] = { - # African - "af": "af_za", "am": "am_et", "ff": "ff_sn", "ha": "ha_ng", - "ig": "ig_ng", "kam": "kam_ke", "kea": "kea_cv", "lg": "lg_ug", - "ln": "ln_cd", "luo": "luo_ke", "nso": "nso_za", "ny": "ny_mw", - "om": "om_et", "sn": "sn_zw", "so": "so_so", "sw": "sw_ke", - "umb": "umb_ao", "wo": "wo_sn", "xh": "xh_za", "yo": "yo_ng", - "zu": "zu_za", - # Arabic, Hebrew, Persian, Kurdish - "ar": "ar_eg", "he": "he_il", "fa": "fa_ir", "ckb": "ckb_iq", - "ps": "ps_af", "ur": "ur_pk", - # South Asian - "as": "as_in", "bn": "bn_in", "gu": "gu_in", "hi": "hi_in", - "kn": "kn_in", "ml": "ml_in", "mr": "mr_in", "ne": "ne_np", - "or": "or_in", "pa": "pa_in", "sd": "sd_in", "ta": "ta_in", - "te": "te_in", - # East / Southeast Asian - "my": "my_mm", "fil": "fil_ph", "tl": "fil_ph", # tl alias - "id": "id_id", "ja": "ja_jp", "jv": "jv_id", "km": "km_kh", - "ko": "ko_kr", "lo": "lo_la", "ms": "ms_my", "th": "th_th", - "vi": "vi_vn", "ceb": "ceb_ph", - # Chinese / Cantonese - "zh": "cmn_hans_cn", # simplified Mandarin (default) - "zh-cn": "cmn_hans_cn", # simplified Mandarin (explicit) - # FLEURS has no Traditional Mandarin, so zh-tw is intentionally unmapped. - # Traditional-script models are scored against cmn_hans_cn with both sides - # OpenCC-folded to one script; see docs/tools/wer.md. - "yue": "yue_hant_hk", # Cantonese (traditional) - # Central Asian - "az": "az_az", "kk": "kk_kz", "ky": "ky_kg", "mn": "mn_mn", - "tg": "tg_tj", "uz": "uz_uz", "hy": "hy_am", "ka": "ka_ge", - # European - "ast": "ast_es", "be": "be_by", "bg": "bg_bg", "bs": "bs_ba", - "ca": "ca_es", "cs": "cs_cz", "cy": "cy_gb", "da": "da_dk", - "de": "de_de", "el": "el_gr", "en": "en_us", "es": "es_419", - "et": "et_ee", "fi": "fi_fi", "fr": "fr_fr", "ga": "ga_ie", - "gl": "gl_es", "hr": "hr_hr", "hu": "hu_hu", "is": "is_is", - "it": "it_it", "lb": "lb_lu", "lt": "lt_lt", "lv": "lv_lv", - "mi": "mi_nz", "mk": "mk_mk", "mt": "mt_mt", "nb": "nb_no", - "no": "nb_no", # Norwegian macro → Bokmål - "nl": "nl_nl", "oc": "oc_fr", "pl": "pl_pl", "pt": "pt_br", - "ro": "ro_ro", "ru": "ru_ru", "sk": "sk_sk", "sl": "sl_si", - "sr": "sr_rs", "sv": "sv_se", "tr": "tr_tr", "uk": "uk_ua", -} +from languages import FLEURS_LANGS # -------- Shared helpers -------------------------------------------------- diff --git a/scripts/wer/languages.py b/scripts/wer/languages.py new file mode 100644 index 00000000..047c6e58 --- /dev/null +++ b/scripts/wer/languages.py @@ -0,0 +1,63 @@ +"""Lightweight language metadata shared by WER ingestion and catalog policy.""" +from __future__ import annotations + +# BCP-47 shorthand -> google/fleurs configuration. Keep this module stdlib-only: +# catalog publication planning imports it without installing dataset/audio deps. +FLEURS_LANGS: dict[str, str] = { + # African + "af": "af_za", "am": "am_et", "ff": "ff_sn", "ha": "ha_ng", + "ig": "ig_ng", "kam": "kam_ke", "kea": "kea_cv", "lg": "lg_ug", + "ln": "ln_cd", "luo": "luo_ke", "nso": "nso_za", "ny": "ny_mw", + "om": "om_et", "sn": "sn_zw", "so": "so_so", "sw": "sw_ke", + "umb": "umb_ao", "wo": "wo_sn", "xh": "xh_za", "yo": "yo_ng", + "zu": "zu_za", + # Arabic, Hebrew, Persian, Kurdish + "ar": "ar_eg", "he": "he_il", "fa": "fa_ir", "ckb": "ckb_iq", + "ps": "ps_af", "ur": "ur_pk", + # South Asian + "as": "as_in", "bn": "bn_in", "gu": "gu_in", "hi": "hi_in", + "kn": "kn_in", "ml": "ml_in", "mr": "mr_in", "ne": "ne_np", + "or": "or_in", "pa": "pa_in", "sd": "sd_in", "ta": "ta_in", + "te": "te_in", + # East / Southeast Asian + "my": "my_mm", "fil": "fil_ph", "tl": "fil_ph", + "id": "id_id", "ja": "ja_jp", "jv": "jv_id", "jw": "jv_id", + "km": "km_kh", + "ko": "ko_kr", "lo": "lo_la", "ms": "ms_my", "th": "th_th", + "vi": "vi_vn", "ceb": "ceb_ph", + # Chinese / Cantonese + "zh": "cmn_hans_cn", "zh-cn": "cmn_hans_cn", + # zh-tw is intentionally absent: FLEURS has no Traditional Mandarin. + "yue": "yue_hant_hk", + # Central Asian + "az": "az_az", "kk": "kk_kz", "ky": "ky_kg", "mn": "mn_mn", + "tg": "tg_tj", "uz": "uz_uz", "hy": "hy_am", "ka": "ka_ge", + # European + "ast": "ast_es", "be": "be_by", "bg": "bg_bg", "bs": "bs_ba", + "ca": "ca_es", "cs": "cs_cz", "cy": "cy_gb", "da": "da_dk", + "de": "de_de", "el": "el_gr", "en": "en_us", "es": "es_419", + "et": "et_ee", "fi": "fi_fi", "fr": "fr_fr", "ga": "ga_ie", + "gl": "gl_es", "hr": "hr_hr", "hu": "hu_hu", "is": "is_is", + "it": "it_it", "lb": "lb_lu", "lt": "lt_lt", "lv": "lv_lv", + "mi": "mi_nz", "mk": "mk_mk", "mt": "mt_mt", "nb": "nb_no", + "no": "nb_no", "nl": "nl_nl", "oc": "oc_fr", "pl": "pl_pl", + "pt": "pt_br", "ro": "ro_ro", "ru": "ru_ru", "sk": "sk_sk", + "sl": "sl_si", "sr": "sr_rs", "sv": "sv_se", "tr": "tr_tr", + "uk": "uk_ua", +} + +# First spelling for a configuration is the canonical catalog spelling. This +# makes aliases deterministic: jw -> jv, tl -> fil, no -> nb, zh-cn -> zh. +FLEURS_CANONICAL_BY_CONFIG: dict[str, str] = {} +for _language, _config in FLEURS_LANGS.items(): + FLEURS_CANONICAL_BY_CONFIG.setdefault(_config, _language) + +# Character-based scoring for scripts without reliable whitespace-delimited +# words in FLEURS. Khmer/Lao/Burmese spaces are phrase separators rather than +# dependable word boundaries, so WER would mostly measure orthography policy. +CER_LANGUAGES = {"zh", "yue", "ja", "ko", "th", "km", "lo", "my"} + +# Model-side spellings that name the same language as a dataset code. A +# language hint is checked against the manifest through this map, so a model +# that prompts with Whisper's legacy `jw` can be scored on FLEURS `jv`. +LANGUAGE_ALIASES = {"jw": "jv", "tl": "fil", "no": "nb", "zh-cn": "zh"} diff --git a/scripts/wer/remote/cache_paths.py b/scripts/wer/remote/cache_paths.py index 5093f4b9..6d4d1e37 100644 --- a/scripts/wer/remote/cache_paths.py +++ b/scripts/wer/remote/cache_paths.py @@ -16,6 +16,8 @@ def hyp_cache_paths( language: str = "", stream_chunk_ms: int = 0, stream_att_right: int = -1, + publication_profile: str = "", + backend: str = "", ) -> tuple[str, str]: """Deterministic Volume paths for the (model, dataset, subset, batch, sort) tuple. @@ -45,8 +47,10 @@ def hyp_cache_paths( # default R), no tag so it stays compatible with already-cached entries; # any explicit R gets its own slot so e.g. R=13 and R=0 never collide. r_tag = "" if stream_att_right < 0 else f".r{stream_att_right}" + profile_tag = "" if not publication_profile else f".profile-{publication_profile}" + backend_tag = "" if not backend else f".backend-{backend}" base = (f"/data/wer/hyps/{hyp_fp}/{slug}." - f"{dataset_id(dataset_spec)}.{subset_tag}{bs_tag}{sort_tag}{ts_tag}{lang_tag}{stream_tag}{r_tag}") + f"{dataset_id(dataset_spec)}.{subset_tag}{bs_tag}{sort_tag}{ts_tag}{lang_tag}{stream_tag}{r_tag}{profile_tag}{backend_tag}") return f"{base}.jsonl", f"{base}.summary.json" diff --git a/scripts/wer/remote/fingerprints.py b/scripts/wer/remote/fingerprints.py index aaa560aa..a9f7eff7 100644 --- a/scripts/wer/remote/fingerprints.py +++ b/scripts/wer/remote/fingerprints.py @@ -54,14 +54,14 @@ def source_fingerprint(root: Path) -> str: def hyp_extra_hash(root: Path) -> str: - """Hash of the Python pieces that affect hyp output: run.py + ingest.py. + """Hash of the Python pieces that affect hyp output and manifests. Folded with SRC_FP into the hyp cache key. Edits to the dispatcher (modal_sweep.py) or to local-only scripts (score.py) do NOT invalidate the hyp cache because they cannot change what the cell produces. """ h = hashlib.sha256() - for name in ("run.py", "ingest.py"): + for name in ("run.py", "ingest.py", "languages.py"): p = root / "scripts" / "wer" / name if p.is_file(): h.update(name.encode()); h.update(b"\0") diff --git a/scripts/wer/remote/modal_sweep.py b/scripts/wer/remote/modal_sweep.py index 503f62ba..438f55ca 100644 --- a/scripts/wer/remote/modal_sweep.py +++ b/scripts/wer/remote/modal_sweep.py @@ -93,6 +93,12 @@ from subprocess_io import run_subprocess_capturing_stderr from workdir import prepare_work +_CATALOG_HELPERS = pathlib.Path(REPO) / "scripts" / "catalog" +if _CATALOG_HELPERS.is_dir() and str(_CATALOG_HELPERS) not in sys.path: + sys.path.insert(0, str(_CATALOG_HELPERS)) +import common as catalog_common +import profiles as benchmark_profiles + # SRC_FP keys the build cache (C++ binary). HYP_FP keys the hyp cache and # folds SRC_FP in so a binary change invalidates hyps too. Splitting them @@ -388,6 +394,27 @@ def list_ggufs(repos: list[str]) -> list[tuple[str, list[str]]]: # streams to Modal logs. # --------------------------------------------------------------------------- +def _local_engine_sha() -> str: + """Short SHA of the dispatching checkout, or "" when the engine is dirty. + + Refuses to name a commit the binary does not correspond to: if src/ or + CMakeLists.txt carry uncommitted edits the build is not that commit, and + no claim beats a wrong one. + """ + import subprocess + root = pathlib.Path(__file__).resolve().parents[3] + try: + dirty = subprocess.run(["git", "status", "--porcelain", "src", "CMakeLists.txt"], + capture_output=True, text=True, timeout=5, cwd=root) + if dirty.returncode != 0 or dirty.stdout.strip(): + return "" + out = subprocess.run(["git", "rev-parse", "--short", "HEAD"], + capture_output=True, text=True, timeout=5, cwd=root) + return out.stdout.strip() if out.returncode == 0 else "" + except (OSError, subprocess.SubprocessError): + return "" + + def _run_wer_impl( model_repo: str, model_file: str, @@ -398,6 +425,9 @@ def _run_wer_impl( sort_by_length: bool = True, timestamps: str = "none", language: str = "", + engine_sha: str = "", + publication_profile: str = "", + backend: str = "", stream_chunk_ms: int = 0, stream_att_right: int = -1, dataset_status: dict | None = None, @@ -419,7 +449,8 @@ def _run_wer_impl( # when a hyp for this (fingerprint, model, dataset, subset) already exists. cache_hyp, cache_sum = hyp_cache_paths( HYP_FP, model_file, dataset_spec, n_utts, batch_size, sort_by_length, - timestamps, language, stream_chunk_ms, stream_att_right) + timestamps, language, stream_chunk_ms, stream_att_right, + publication_profile, backend) if os.path.exists(cache_hyp) and os.path.exists(cache_sum) \ and os.path.getsize(cache_hyp) > 0: _log_prepared_dataset("wer", dataset_status) @@ -459,6 +490,11 @@ def _run_wer_impl( # Force per-line stdout flushing so progress streams live to Modal logs. env = {**os.environ, "PYTHONUNBUFFERED": "1"} + # The container gets a source tree with no .git, so run.py's own + # `git rev-parse` finds nothing and every remote row would land + # unattributable. Hand it the sha of the tree this sweep built from. + if engine_sha: + env["TRANSCRIBE_ENGINE_SHA"] = engine_sha cmd = [ "uv", "run", "scripts/wer/run.py", "--cli", cli_path, @@ -474,6 +510,10 @@ def _run_wer_impl( cmd += ["--timestamps", timestamps] if language: cmd += ["--language", language] + if publication_profile: + cmd += ["--publication-profile", publication_profile] + if backend: + cmd += ["--backend", backend] if stream_chunk_ms and stream_chunk_ms > 0: cmd += ["--stream-chunk-ms", str(stream_chunk_ms)] if stream_att_right >= 0: @@ -577,6 +617,9 @@ def runner( stream_chunk_ms: int = 0, stream_att_right: int = -1, dataset_status: dict | None = None, + engine_sha: str = "", + publication_profile: str = "", + backend: str = "", ) -> dict: # Prefer the build_dir the local entrypoint computed and built into: # SRC_FP can drift between the laptop and the container, so recomputing @@ -586,7 +629,8 @@ def runner( model_repo, model_file, dataset_spec, n_utts, build_dir or default_build_dir, batch_size=batch_size, sort_by_length=sort_by_length, - timestamps=timestamps, language=language, + timestamps=timestamps, language=language, engine_sha=engine_sha, + publication_profile=publication_profile, backend=backend, stream_chunk_ms=stream_chunk_ms, stream_att_right=stream_att_right, dataset_status=dataset_status, @@ -816,6 +860,94 @@ def runner( # Local entrypoints. # --------------------------------------------------------------------------- +def _dispatch(cells: list[dict], gpu: str, *, clean: bool = False, n_utts: int = -1, + sort_by_length: bool = True, stream_chunk_ms: int = 0, + stream_att_right: int = -1) -> tuple[list[tuple], list[tuple]]: + """Run every cell at once on one GPU class and collect the results. + + A cell is {repo, file, dataset, bs, language, timestamps, + publication_profile, backend}. The binary is built once, every distinct + dataset is prefetched in parallel, then one container per cell is + spawned up front, so wall time is the slowest cell rather than the sum of + per-dataset rounds. Each hypothesis file is written the moment its cell + finishes. + """ + runner = _GPU_FNS.get(gpu) + if runner is None: + raise SystemExit(f"--gpu {gpu!r} not registered; choose one of: {sorted(_GPU_FNS)}") + repo_root = pathlib.Path(REPO) + arch = GPU_TO_ARCH[gpu] + build_dir = _build_dir(gpu) + print(f">>> build (arch sm_{arch} -> {build_dir})") + build.remote(arch=arch, build_dir=build_dir, clean=clean) + + datasets = sorted({c["dataset"] for c in cells}) + print(f">>> prefetch {len(datasets)} dataset(s) in parallel") + status_futs = {d: prefetch_dataset.spawn(d) for d in datasets} + statuses = {} + for d, fut in status_futs.items(): + statuses[d] = fut.get() + print(f">>> dataset ready: {statuses[d]['dataset_id']} " + f"n={statuses[d]['utterances']} sha={statuses[d]['manifest_sha256'][:12]}") + + print(f">>> launching {len(cells)} {gpu} containers in parallel...") + n = None if n_utts < 0 else n_utts + engine_sha = _local_engine_sha() + futs = [(c, runner.spawn(c["repo"], c["file"], c["dataset"], n, + c["bs"], sort_by_length, build_dir, c["timestamps"], + c.get("language", ""), stream_chunk_ms, stream_att_right, + statuses[c["dataset"]], engine_sha, + c.get("publication_profile", ""), c.get("backend", ""))) + for c in cells] + + rows, failures = [], [] + for c, fut in futs: + bs = c["bs"] + slug = c["file"].replace(".gguf", "") + (f" b{bs}" if bs != 1 else "") + try: + res = fut.get() + # b1 stays untagged (matches the published-run filenames); b>1 is + # tagged .b{bs} so batched hyps score independently for comparison. + path = write_hyp(repo_root, res["hyp_jsonl"], c["file"], c["dataset"], + batch_size=(bs if bs != 1 else None), + timestamps=c["timestamps"], + stream_chunk_ms=stream_chunk_ms, + stream_att_right=stream_att_right, + n_utts=(n_utts if n_utts >= 0 else None)) + s = res["summary"] + rows.append((slug, c["dataset"], s["n_utts"], s["audio_s"], + s["wall_s"], s["rtf_wall"], str(path))) + tag = "CACHED" if res.get("cached") else "OK" + print(f" [{tag}] {slug} {c['dataset']}: {s['wall_s']:.1f}s, " + f"RTF {s['rtf_wall']:.1f}x -> {path}") + except Exception as e: # noqa: BLE001 - one cell's failure must not hide the rest + failures.append((slug, c["dataset"], repr(e))) + print(f" [FAIL] {slug} {c['dataset']}: {e} (check Modal dashboard for stderr)") + + for ds in datasets: + ds_rows = [r for r in rows if r[1] == ds] + if not ds_rows: + continue + summary_path = repo_root / "reports" / "wer" / f"remote_sweep.{dataset_id(ds)}.summary.tsv" + with open(summary_path, "w") as f: + f.write("slug\tdataset\tn_utts\taudio_s\twall_s\trtf\tpath\n") + for r in ds_rows: + f.write("\t".join(str(x) for x in r) + "\n") + + print("\n========== sweep summary ==========") + print(f"{'slug':<48} {'dataset':<24} {'n':>5} {'audio':>9} {'wall':>8} {'rtf':>6}") + for slug, ds, n_, audio, wall, rtf, _ in rows: + print(f"{slug:<48} {ds:<24} {n_:>5} {audio:>9.1f} {wall:>8.1f} {rtf:>6.1f}") + if failures: + print("\nfailures:") + for slug, ds, err in failures: + print(f" {slug} {ds}: {err}") + ids = sorted({dataset_id(d) for d in datasets}) + print("\nscore locally: for f in reports/wer/*.{" + ",".join(ids) + "}.jsonl; " + "do uv run scripts/wer/score.py \"$f\"; done") + return rows, failures + + @app.local_entrypoint() def sweep( models: str, @@ -830,13 +962,15 @@ def sweep( language: str = "", stream_chunk_ms: int = 0, stream_att_right: int = -1, + publication_profile: str = "", + backend: str = "", ) -> None: - """Fan WER across one or more models on one GPU class. + """Fan WER across one or more models on one dataset and GPU class. - --models Comma-separated. Each entry is either an hf_card slug - (e.g. "moonshine-base" → scripts/hf_cards/moonshine-base.yaml) + --models Comma-separated. Each entry is either a catalog variant + (e.g. "moonshine-base" -> catalog/moonshine-base.json) or a HF repo path (e.g. "handy-computer/foo-gguf"). - Slugs pin the quant set; repo paths discover via HF API. + Variants pin the quant set; repo paths discover via HF API. --dataset "librispeech:test-clean" (default), "librispeech:", or "fleurs:". --quants Optional substring filter, e.g. "Q8_0,F16". @@ -862,10 +996,8 @@ def sweep( repo_root = pathlib.Path(REPO) resolved = [(s, *resolve_model(repo_root, s)) for s in specs] - needs_listing = sorted({repo for _, repo, fns in resolved if fns is None}) listings = dict(list_ggufs.remote(needs_listing)) if needs_listing else {} - sizes = [int(x) for x in batch_sizes.split(",") if x.strip()] or [1] cells: list[dict] = [] @@ -883,8 +1015,10 @@ def sweep( continue for f in fns: for bs in sizes: - cells.append({"repo": repo, "file": f, "dataset": dataset, "bs": bs}) - + cells.append({"repo": repo, "file": f, "dataset": dataset, "bs": bs, + "language": language, "timestamps": timestamps, + "publication_profile": publication_profile, + "backend": backend}) if skipped: print(">>> skipped (no cells generated):") for s, r in skipped: @@ -892,75 +1026,74 @@ def sweep( print(f">>> {len(cells)} cells to run") if not cells: raise SystemExit("nothing to do") + _dispatch(cells, gpu, clean=clean, n_utts=n_utts, sort_by_length=sort_by_length, + stream_chunk_ms=stream_chunk_ms, stream_att_right=stream_att_right) - runner = _GPU_FNS.get(gpu) - if runner is None: - raise SystemExit( - f"--gpu {gpu!r} not registered; choose one of: {sorted(_GPU_FNS)}" - ) - arch = GPU_TO_ARCH[gpu] - build_dir = _build_dir(gpu) - print(f">>> build (arch sm_{arch} -> {build_dir})") - build.remote(arch=arch, build_dir=build_dir, clean=clean) - print(f">>> prefetch dataset ({dataset})") - dataset_status = prefetch_dataset.remote(dataset) - print(f">>> dataset ready: {dataset_status['dataset_id']} " - f"n={dataset_status['utterances']} " - f"sha={dataset_status['manifest_sha256'][:12]}") - - print(f">>> launching {len(cells)} {gpu} containers in parallel...") - n = None if n_utts < 0 else n_utts - futs = [(c, runner.spawn(c["repo"], c["file"], c["dataset"], n, - c["bs"], sort_by_length, build_dir, timestamps, - language, stream_chunk_ms, stream_att_right, - dataset_status)) - for c in cells] - - rows, failures = [], [] - for c, fut in futs: - bs = c["bs"] - slug = c["file"].replace(".gguf", "") + (f" b{bs}" if bs != 1 else "") - try: - res = fut.get() - # b1 stays untagged (matches the published-run filenames); b>1 is - # tagged .b{bs} so batched hyps score independently for comparison. - p = write_hyp(repo_root, res["hyp_jsonl"], c["file"], c["dataset"], - batch_size=(bs if bs != 1 else None), - timestamps=timestamps, - stream_chunk_ms=stream_chunk_ms, - stream_att_right=stream_att_right) - s = res["summary"] - rows.append((slug, c["dataset"], s["n_utts"], s["audio_s"], - s["wall_s"], s["rtf_wall"], str(p))) - tag = "CACHED" if res.get("cached") else "OK" - print(f" [{tag}] {slug}: {s['wall_s']:.1f}s, RTF {s['rtf_wall']:.1f}x -> {p}") - except Exception as e: - failures.append((slug, repr(e))) - print(f" [FAIL] {slug}: {e} (check Modal dashboard for stderr)") - - if rows: - ds_id = dataset_id(dataset) - summary_path = pathlib.Path(REPO) / "reports" / "wer" / \ - f"remote_sweep.{ds_id}.summary.tsv" - with open(summary_path, "w") as f: - f.write("slug\tdataset\tn_utts\taudio_s\twall_s\trtf\tpath\n") - for r in rows: - f.write("\t".join(str(x) for x in r) + "\n") - print(f"\nsummary: {summary_path}") +@app.local_entrypoint() +def publication_sweep( + models: str, + profile: str = "", + missing_only: bool = True, + clean: bool = False, + plan_only: bool = False, +) -> None: + """Run every accuracy cell a catalog publication profile still needs. - print("\n========== sweep summary ==========") - print(f"{'slug':<48} {'dataset':<24} {'n':>5} {'audio':>9} {'wall':>8} {'rtf':>6}") - for slug, ds, n_, audio, wall, rtf, _ in rows: - print(f"{slug:<48} {ds:<24} {n_:>5} {audio:>9.1f} {wall:>8.1f} {rtf:>6.1f}") - if failures: - print("\nfailures:") - for slug, err in failures: - print(f" {slug}: {err}") - if skipped: - print(f"\nskipped: {len(skipped)} entries (listed at config time above)") - print(f"\nscore locally: for f in reports/wer/*.{dataset_id(dataset)}.jsonl; " - f"do uv run scripts/wer/score.py \"$f\"; done") + Datasets, quants, batch size, timestamps, language prompts, and GPU come + from the checked-in profile. Every cell across every dataset is planned + up front and dispatched at once, one container each. + """ + profile_id, profile_data = benchmark_profiles.load_profile(profile or None) + records = catalog_common.load_records() + selected = [item.strip() for item in models.split(",") if item.strip()] + unknown = [item for item in selected if item not in records] + if unknown: + raise SystemExit(f"no catalog record for: {', '.join(unknown)}") + if not selected: + raise SystemExit("--models is required (comma-separated catalog variants)") + + by_gpu: dict[str, list[dict]] = {} + for variant in selected: + record = records[variant] + repo = record.get("published_repo") + if not repo: + raise SystemExit(f"{variant}: catalog has no published_repo") + files = {d["quant"]: d["filename"] for d in record.get("downloads", [])} + expected = benchmark_profiles.apply_exceptions( + record, "accuracy", + benchmark_profiles.expected_accuracy(record, profile_data)) + valid = {benchmark_profiles.profile_key(row) + for row in record.get("accuracy_benchmarks", []) if row.get("engine_sha")} + legacy = {benchmark_profiles.accuracy_core_key(row) + for row in record.get("accuracy_benchmarks", []) + if row.get("measurement_provenance") == "legacy-published"} + for cell in expected: + if missing_only and (benchmark_profiles.profile_key(cell) in valid + or benchmark_profiles.accuracy_core_key(cell) in legacy): + continue + by_gpu.setdefault(cell["gpu"], []).append({ + "repo": repo, "file": files[cell["quant"]], + "dataset": benchmark_profiles.dataset_spec(cell), + "bs": cell["batch_size"], "language": cell["runtime_language"], + "timestamps": cell["timestamps"], "publication_profile": profile_id, + "backend": cell["backend"], "_variant": variant, + "_sort": cell["sort_by_length"], + }) + + total = sum(len(cells) for cells in by_gpu.values()) + print(f"publication profile {profile_id}: {total} cell(s) across " + f"{len({c['dataset'] for cells in by_gpu.values() for c in cells})} dataset(s), " + f"{len(by_gpu)} GPU class(es)") + for gpu, cells in sorted(by_gpu.items()): + for c in cells: + print(f" {gpu}: {c['_variant']:40s} {c['dataset']:22s} {c['file']} " + f"lang={c['language']} bs={c['bs']} ts={c['timestamps']}") + if plan_only or not total: + return + for gpu, cells in sorted(by_gpu.items()): + sorts = {c["_sort"] for c in cells} + _dispatch(cells, gpu, clean=clean, sort_by_length=all(sorts)) @app.local_entrypoint() @@ -1039,7 +1172,8 @@ def batch_sweep( # Launch all batch sizes in parallel (each its own container). Pass the # locally-computed build_dir so the runner reads exactly what build() wrote. futs = [(bs, runner.spawn(repo, model_file, dataset, n, bs, sort_by_length, - build_dir, "none", language, 0, -1, dataset_status)) + build_dir, "none", language, 0, -1, dataset_status, + _local_engine_sha(), "", "")) for bs in sizes] rows: list[tuple] = [] diff --git a/scripts/wer/remote/model_specs.py b/scripts/wer/remote/model_specs.py index ced3bcee..13f093c5 100644 --- a/scripts/wer/remote/model_specs.py +++ b/scripts/wer/remote/model_specs.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json from pathlib import Path @@ -9,31 +10,29 @@ def resolve_model(root: Path, spec: str) -> tuple[str, list[str] | None]: Rules: - Spec contains '/': treat as a HF repo path. Filenames are None (caller will discover via the HF API at dispatch time). - - Otherwise: treat as an hf_card slug. Reads - scripts/hf_cards/.yaml, returns its target_repo and the - pinned `quants[].filename` list. + - Otherwise: treat as a variant slug and read catalog/.json for + `published_repo` and the `downloads[].filename` list. + + The catalog is the source here because that is where the published repo + and quant set now live. scripts/hf_cards/.yaml used to carry + `target_repo` and `quants[].filename`; both are derived from the catalog + now, so a card no longer states them and parsing it finds nothing. """ if "/" in spec: return spec, None - card_path = root / "scripts" / "hf_cards" / f"{spec}.yaml" - if not card_path.exists(): + + record_path = root / "catalog" / f"{spec}.json" + if not record_path.exists(): raise SystemExit( - f"no hf_card at {card_path}; pass a HF repo path " - f"(e.g. handy-computer/{spec}-gguf) if the card doesn't exist yet" + f"no catalog record at {record_path}; pass a HF repo path " + f"(e.g. handy-computer/{spec}-gguf) if the variant isn't in the " + f"catalog yet" ) - # Tiny manual parser so the local entrypoint has no non-stdlib deps. The card - # schema has target_repo at top level and filename: only under quants[]. - target_repo: str | None = None - filenames: list[str] = [] - for raw in open(card_path): - line = raw.split("#", 1)[0].rstrip() - stripped = line.strip() - if line.startswith("target_repo:"): - target_repo = line.split(":", 1)[1].strip() - elif stripped.startswith("filename:"): - filenames.append(stripped.split(":", 1)[1].strip()) - if not target_repo: - raise SystemExit(f"hf_card {spec!r}: missing target_repo") + record = json.loads(record_path.read_text()) + repo = record.get("published_repo") + if not repo: + raise SystemExit(f"catalog record {spec!r}: no published_repo") + filenames = [d["filename"] for d in record.get("downloads", []) if d.get("filename")] if not filenames: - raise SystemExit(f"hf_card {spec!r} has no quants[].filename entries") - return target_repo, filenames + raise SystemExit(f"catalog record {spec!r} has no downloads[].filename entries") + return repo, filenames diff --git a/scripts/wer/remote/output_paths.py b/scripts/wer/remote/output_paths.py index 7f89062f..19d6724e 100644 --- a/scripts/wer/remote/output_paths.py +++ b/scripts/wer/remote/output_paths.py @@ -31,6 +31,7 @@ def write_hyp( timestamps: str = "none", stream_chunk_ms: int = 0, stream_att_right: int = -1, + n_utts: int | None = None, ) -> Path: out_dir = root / "reports" / "wer" out_dir.mkdir(parents=True, exist_ok=True) @@ -40,6 +41,7 @@ def write_hyp( ts_tag = "" if timestamps == "none" else f".ts-{timestamps}" stream_tag = "" if stream_chunk_ms <= 0 else f".stream{stream_chunk_ms}ms" r_tag = "" if stream_att_right < 0 else f".r{stream_att_right}" - out_path = out_dir / f"{slug}.{ds}{bs_tag}{ts_tag}{stream_tag}{r_tag}.jsonl" + subset_tag = "" if n_utts is None else f".n{n_utts}" + out_path = out_dir / f"{slug}.{ds}{bs_tag}{ts_tag}{stream_tag}{r_tag}{subset_tag}.jsonl" out_path.write_text(hyp_jsonl) return out_path diff --git a/scripts/wer/run.py b/scripts/wer/run.py index 9bc3a169..d39399d2 100644 --- a/scripts/wer/run.py +++ b/scripts/wer/run.py @@ -81,11 +81,13 @@ sys.path.insert(0, str(REMOTE_HELPERS)) from dataset_specs import ( # noqa: E402 + default_language_for, ingest_args_for, local_manifest_path_for, parse_dataset_spec, ) +from languages import LANGUAGE_ALIASES # noqa: E402 def read_stderr_tail(path: str, max_lines: int = 50, max_bytes: int = 65536) -> str: @@ -173,6 +175,27 @@ def resolve_dataset(repo: Path, spec: str) -> tuple[Path, str | None]: return manifest, default_lang +def engine_sha() -> str | None: + """Short SHA of the checkout that built transcribe-cli, if it is a repo. + + TRANSCRIBE_ENGINE_SHA overrides the lookup, for a runner that has the + binary but not the checkout: a Modal container is handed a source tree + with no .git, so `git rev-parse` there finds nothing and every remote + sweep would land an unattributable row. The dispatcher passes the sha of + the tree it built from instead. + """ + override = os.environ.get("TRANSCRIBE_ENGINE_SHA", "").strip() + if override: + return override + try: + out = subprocess.run(["git", "rev-parse", "--short", "HEAD"], + capture_output=True, text=True, timeout=5, + cwd=Path(__file__).resolve().parents[2]) + return out.stdout.strip() or None if out.returncode == 0 else None + except (OSError, subprocess.SubprocessError): + return None + + def main() -> int: repo = find_repo_root(Path(__file__).parent) @@ -207,7 +230,7 @@ def main() -> int: "batched encoder pads to the group max). Output is " "keyed by file, so id mapping is preserved.") p.add_argument("--backend", - choices=("auto", "cpu", "cpu_accel", "metal", "vulkan"), + choices=("auto", "cpu", "cpu_accel", "metal", "vulkan", "cuda"), default=None, help="Compute backend (default: transcribe-cli default)") p.add_argument("--kv-type", @@ -222,6 +245,9 @@ def main() -> int: p.add_argument("--diarize", action="store_true", help="Request diarization and retain timed speaker " "intervals for scripts/wer/der.py") + p.add_argument("--publication-profile", default="", + help="publication profile that selected this run; normally " + "set by the profile-aware local or Modal dispatcher") p.add_argument("--itn", choices=("off", "on", "default"), default="off", help="Inverse text normalization for ITN-aware families " "(sensevoice, funasr_nano). Pinned to 'off' — the " @@ -293,7 +319,10 @@ def main() -> int: # models like nemotron-3.5-asr-streaming-0.6b require because their # caps.languages list carries only the BCP-47 long forms. def _primary(tag: str) -> str: - return tag.split("-", 1)[0].lower() if tag else tag + if not tag: + return tag + primary = tag.split("-", 1)[0].lower() + return LANGUAGE_ALIASES.get(tag.lower(), LANGUAGE_ALIASES.get(primary, primary)) manifest_langs = { e["language"] for e in manifest if e.get("language") } @@ -459,6 +488,11 @@ def _dur(e: dict) -> float: "backend": args.backend or "default", "kv_type": args.kv_type or "default", "decode": "greedy+default-fallback", + # The build that produced the hypotheses. A score with no + # engine behind it cannot be reproduced or superseded, and + # the catalog refuses to publish one. + "engine_sha": engine_sha(), + "publication_profile": args.publication_profile or None, } fout.write(json.dumps(result) + "\n") fout.flush() diff --git a/scripts/wer/run_reference_cohere_transformers.py b/scripts/wer/run_reference_cohere_transformers.py new file mode 100644 index 00000000..a45bbd33 --- /dev/null +++ b/scripts/wer/run_reference_cohere_transformers.py @@ -0,0 +1,238 @@ +#!/usr/bin/env python3 +""" +run_reference_cohere_transformers.py — Cohere ASR WER baseline. + +Loads a Cohere Transcribe variant via native Hugging Face Transformers +(`CohereAsrForConditionalGeneration`, `trust_remote_code=False` — the +remote-code path is known-broken, see the family doc and HF discussion +#28 on cohere-transcribe-03-2026) and runs greedy decode over a WER +manifest. Writes run.py-compatible JSONL so scripts/wer/score.py can +score the output the same way it scores the C++ port's report. + +Language comes from each manifest entry's `language` field (ingest.py +writes it), overridable globally with --language. The processor builds +the 10-token decoder prompt from the language, so no prompt text is +constructed here. + +Usage (from repo root): + + uv run --project scripts/envs/cohere \\ + scripts/wer/run_reference_cohere_transformers.py \\ + --model CohereLabs/cohere-transcribe-arabic-07-2026 \\ + --manifest samples/wer/fleurs-ar.manifest.jsonl \\ + --out reports/wer/cohere-transcribe-arabic-07-2026-REF.fleurs-ar.jsonl +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path + + +def main() -> int: + p = argparse.ArgumentParser( + description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + p.add_argument("--manifest", type=Path, required=True, + help="Input manifest JSONL (id/audio/ref_text/language).") + p.add_argument("--out", type=Path, required=True, + help="Output JSONL path (run.py-compatible).") + p.add_argument("--model", required=True, + help="HF repo id (CohereLabs/cohere-transcribe-03-2026) " + "or local directory.") + p.add_argument("--revision", default=None, + help="HF revision to pin (ignored for local paths).") + p.add_argument("--language", default=None, + help="Force one language for every utterance. Default: " + "each entry's manifest `language` field, else 'en'.") + p.add_argument("--no-punctuation", action="store_true", + help="Use the <|nopnc|> prompt slot instead of <|pnc|>.") + p.add_argument("--device", default="cpu", + help="torch device (default: cpu; 'cuda' on Modal).") + p.add_argument("--torch-threads", type=int, default=0, + help="torch.set_num_threads (0 = unchanged).") + p.add_argument("--max-new-tokens", type=int, default=256) + p.add_argument("--dtype", default="bf16", + choices=["bf16", "f16", "f32"], + help="Model dtype (default bf16 = checkpoint dtype).") + p.add_argument("--limit", type=int, default=0, + help="Process only the first N utterances (0 = all).") + p.add_argument("--batch-size", type=int, default=1, + help="Group N utterances per batched generate() call. " + ">1 batches audio through the processor; on any batch " + "error it falls back to per-utterance so a bad sample " + "can't drop the group. Uniform across reference " + "runners so the Modal reference_sweep drives every " + "family the same way.") + args = p.parse_args() + + if not args.manifest.exists(): + print(f"error: manifest not found: {args.manifest}", file=sys.stderr) + return 2 + args.out.parent.mkdir(parents=True, exist_ok=True) + + import torch + if args.torch_threads > 0: + torch.set_num_threads(args.torch_threads) + torch.set_num_interop_threads(1) + + import soundfile as sf + import transformers + from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor + + local_only = Path(args.model).is_dir() + revision = args.revision if not local_only else None + + print( + f"loading: {args.model} (transformers {transformers.__version__}, " + f"device={args.device}, dtype={args.dtype})" + ) + t0 = time.monotonic() + processor = AutoProcessor.from_pretrained( + args.model, revision=revision, + trust_remote_code=False, local_files_only=local_only, + ) + dtype = {"bf16": torch.bfloat16, + "f16": torch.float16, + "f32": torch.float32}[args.dtype] + model = AutoModelForSpeechSeq2Seq.from_pretrained( + args.model, revision=revision, + trust_remote_code=False, local_files_only=local_only, + dtype=dtype, + ).eval().to(args.device) + load_ms = (time.monotonic() - t0) * 1000 + + conv_dtype = model.model.encoder.subsampling.layers[0].weight.dtype + + with open(args.manifest) as f: + manifest = [json.loads(line) for line in f if line.strip()] + if args.limit > 0: + manifest = manifest[:args.limit] + total = len(manifest) + print(f"manifest: {args.manifest} ({total} utterances)") + print(f"output: {args.out}") + + def entry_language(entry: dict) -> str: + return args.language or entry.get("language") or "en" + + def load_pcm(audio_path: str): + pcm, sr = sf.read(audio_path, dtype="float32") + if pcm.ndim > 1: + pcm = pcm[:, 0] + if sr != 16000: + raise RuntimeError(f"cohere_asr expects 16kHz; got {sr}Hz") + return pcm + + # Shared generate path: the processor emits input_features plus the + # language-conditioned decoder_input_ids prompt (same 10-token length + # for every row, so one slice drops the prompt for the whole batch). + def infer_group(entries: list) -> list: + language = entry_language(entries[0]) + if any(entry_language(e_) != language for e_ in entries): + raise RuntimeError("mixed languages in one batch group") + pcms = [load_pcm(e_["audio"]) for e_ in entries] + inputs = processor( + audio=pcms, + language=language, + punctuation=not args.no_punctuation, + sampling_rate=16000, + return_tensors="pt", + ) + inputs = {k: (v.to(args.device) if hasattr(v, "to") else v) + for k, v in inputs.items()} + if inputs["input_features"].dtype != conv_dtype: + inputs["input_features"] = inputs["input_features"].to(conv_dtype) + prompt_len = int(inputs["decoder_input_ids"].shape[1]) + with torch.inference_mode(): + gen = model.generate(**inputs, max_new_tokens=args.max_new_tokens, + do_sample=False, num_beams=1) + seqs = gen.sequences if hasattr(gen, "sequences") else gen + eos_id = processor.tokenizer.eos_token_id + out = [] + for row in seqs: + ids = row.detach().cpu().tolist()[prompt_len:] + if eos_id is not None and eos_id in ids: + ids = ids[:ids.index(eos_id)] + out.append(processor.tokenizer.decode( + ids, skip_special_tokens=True).strip()) + return out + + n_done = 0 + n_errors = 0 + t_loop = time.monotonic() + + with open(args.out, "w") as fout: + fout.write(json.dumps({ + "type": "batch_header", + "load_ms": round(load_ms, 1), + "framework": "transformers", + "model": args.model, + "language": args.language, + "dtype": args.dtype, + }) + "\n") + fout.flush() + + bs = max(1, args.batch_size) + for start in range(0, total, bs): + group = manifest[start:start + bs] + k = len(group) + t_start = time.monotonic() + hyps = [""] * k + errs = [""] * k + try: + hyps = infer_group(group) + except Exception: + if k == 1: + e = sys.exc_info()[1] + errs[0] = f"{type(e).__name__}: {e}" + n_errors += 1 + else: + # Fall back to per-utterance for this group. + for i, e_ in enumerate(group): + try: + hyps[i] = infer_group([e_])[0] + except Exception as e2: + errs[i] = f"{type(e2).__name__}: {e2}" + n_errors += 1 + per_ms = round((time.monotonic() - t_start) * 1000 / k, 1) + + for i, entry in enumerate(group): + rec = { + "id": entry["id"], + "ref_text": entry.get("ref_text", ""), + "hyp_text": (hyps[i] or "").strip(), + "mel_ms": 0, + "encode_ms": 0, + "decode_ms": per_ms, + "latency_ms": per_ms, + "error": errs[i], + } + fout.write(json.dumps(rec, ensure_ascii=False) + "\n") + fout.flush() + n_done += 1 + + if start // bs % 10 == 0 or n_done == total: + wall = time.monotonic() - t_loop + rate = n_done / wall if wall > 0 else 0 + eta = (total - n_done) / rate if rate > 0 else 0 + print( + f" [{n_done}/{total}] {rate:.2f} utt/s, " + f"ETA {eta/60:.1f} min, errors={n_errors}", + flush=True, + ) + + wall = time.monotonic() - t_loop + print( + f"\ndone. {n_done} utterances in {wall:.1f}s " + f"({n_done / wall:.2f} utt/s), {n_errors} errors" + ) + print(f"report: {args.out}") + return 0 if n_errors == 0 else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/wer/score.py b/scripts/wer/score.py index 7061c3aa..60d0db37 100644 --- a/scripts/wer/score.py +++ b/scripts/wer/score.py @@ -11,7 +11,7 @@ Routes the metric and text normalizer by language: en → WER + EnglishTextNormalizer - zh/yue/ja/ko/th → CER + BasicTextNormalizer + zh/yue/ja/ko/th/km/lo/my → CER + BasicTextNormalizer other → WER + BasicTextNormalizer Region suffixes (zh-tw, pt-br, ...) are stripped for routing, so @@ -47,10 +47,7 @@ from whisper_normalizer.basic import BasicTextNormalizer from whisper_normalizer.english import EnglishTextNormalizer - -# Languages where CER is the canonical metric. Region suffix is stripped -# before lookup so zh / zh-cn / zh-tw all resolve to the same set entry. -CER_LANGUAGES = {"zh", "yue", "ja", "ko", "th"} +from languages import CER_LANGUAGES # Optional diarization metadata spans: timestamps `[6.98]`, speaker tags @@ -154,6 +151,7 @@ def main() -> int: # per-utterance results. entries: list[dict] = [] header_language: str | None = None + recipe: dict = {} with open(args.report) as f: for line in f: if not line.strip(): @@ -161,6 +159,10 @@ def main() -> int: rec = json.loads(line) if rec.get("type") == "batch_header": header_language = rec.get("language") + # The decode recipe travels with the score, not just with the + # hypotheses: a WER is only comparable to another WER measured + # the same way, and the catalog keys its rows on it. + recipe = rec.get("recipe") or {} continue entries.append(rec) @@ -283,6 +285,10 @@ def main() -> int: "latency_p50_ms": round(lat_p50, 1), "latency_p99_ms": round(lat_p99, 1), "report_file": str(args.report), + "recipe": recipe, + "timestamps": recipe.get("timestamps"), + "batch_size": recipe.get("batch_size"), + "engine_sha": recipe.get("engine_sha"), "per_utterance": per_utt, } # Backward-compat aliases for WER reports. porting-7-wer SKILL.md