source of truth catalog file - #160
Merged
Merged
Conversation
Fold the 4750U publication sweep into the catalog: 245 profile-stamped cells, 137 of them replacing legacy rows that carried a rounded xRT and no stage timings. Ryzen legacy rows drop from 176 to 39. Rows last measured in April and May move 50-90% faster, which is the optimization work since landing rather than a measurement change, so they go in with --force. The moonshine language fine-tunes waive their dots cell. Fed 35 s of out-of-language audio they degenerate into a repetition loop, never emit end-of-stream, and decode hits the 194-token position cap, so the cell times a loop and the bench binary exits non-zero. Their published dots rows were hand-written round numbers; drop them with the requirement, since a waived cell with a row present is an extra. scripts/bench/run.py built its candidate matrix from the profile's top-level samples only, so a family that overrides them (gigaam benches a Russian clip) matched nothing and every variant was silently skipped with exit 0. Union the override samples in before the per-variant filter. Document scripts/catalog/new_record.py as Step 0 of porting-6, the first stage that writes to the catalog, and correct the stale instruction there to transcribe bench numbers into the doc and the card by hand.
All 16 cells, replacing round hand-written placeholders with measured figures. Q8_0 on CPU read 9.00 for all four variants and now reads 12.77 to 14.38; e2e-rnnt on Vulkan read 22.00 and now reads 15.7. Ryzen legacy rows drop from 39 to 2.
English jfk and dots decode out of distribution on a single-language fine-tune: the next-token distribution is nearly flat, so an argmax flip starts a repetition loop that never emits end-of-stream and runs to the 194-token position cap. Whether that happens depends on the backend, since GPU logits are not bit-identical to CPU. moonshine-tiny-ja terminates on CPU and loops on Metal from the same file; moonshine-base-ar did the reverse on Vulkan. A cell whose success is decided by whether nonsense happens to stop is not measuring anything. Point those six variants at their own language clip, the way gigaam already points at a Russian one. Output is then a real transcript and identical across backends. The dots waiver goes with them, and the English rows they held are dropped, so 24 cells read as missing until both rigs bench them. ar, uk and vi keep the waiver; no fixture exists in those languages yet. Underneath the profile, check.py's integrity pass now refuses a record with no speed rows at all. The profile decides which cells are required; this is the floor that stops a model shipping a page and a card with no performance on them. It fires on those six and nothing else.
Both sides fixed the same bug: a bench report names its variant by the models/ directory, which mirrors the upstream repo casing while the record is kebab-case. Keep the resolver that also falls back to the upstream repo slug when exactly one record claims it, so models/SenseVoiceSmall resolves to sensevoice-small, and the uniqueness guard keeps models/GigaAM-v3 from resolving to an arbitrary one of the four gigaam records. scripts/bench/run.py takes both changes: catalog_key for the profile lookup, and the candidate matrix unioning in samples named only by a family or model override.
The twelve moonshine language fine-tunes and the four gigaam variants now bench on a pair of in-language FLEURS clips at the same two lengths as jfk/dots, instead of English audio they decode out of distribution or a 2.2 s clip whose figure was load and encode dominated. vi-long started at the longest Vietnamese utterance, 38.1 s, which exceeds moonshine's 194-token decode cap: the transcript tracked the reference word for word and then stopped mid-word about 95% through. Replaced with a 25.1 s clip, verified to transcribe to completion on both sizes and both backends. Every speed row in the catalog now names the build that measured it, 996 of 996, against 506 of 964 before this sweep. Both rigs are fully profile measured; the only rows outside a profile are the sortformer diarizer's, which no ASR profile claims.
cjpais
changed the title
NairoDorian
added a commit
to NairoDorian/transcribe.cpp
that referenced
this pull request
Sep 18, 2026
…mputer#160, conformer singleton stride fix) into fork
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Basically, a lot of this work was done as a result of building giant sets of word error rate testing across multiple languages for all of the models to get an idea of how well they perform, at least on some set of data, so people can offer better onboarding experiences for local models. Especially since the number of local models is incredibly overwhelming.
This also lays the foundation so that every new model that comes in (officially on the handy-computer hf org) does have more things that need to be tested that we have word error rate data for every language they support, and performance data as well.