Repository navigation
fix(asr/vocab): no context bias against a word TDT heard confidently - #1100
Merged
Merged
Conversation
…967) A listed term could replace a common word that was actually said on the context-biasing weight alone: analytics -> Qualtrics, sort -> Sora, going -> GOG. When TDT decoded every word of the span at >= 0.99 token confidence and the span isn't a close spelling of the term (>= 0.80 similarity, possessive aside), the term now gets no bias and must beat the span on CTC evidence alone. Word timings now carry their lowest token confidence. Adds a real FLEURS fixture (sort-of-shell) that main corrupts and two model-free tests.
…nes (#967) The gate is calibrated on TDT token confidences; the Unified engines feed emission probabilities and weren't measured, so their itnDefaultConfig keeps today's behaviour.
Alex-Wengg
added a commit
that referenced
this pull request
Nov 1, 2026
…asured trade-off (#1101) Closes #899. #900 fixed #899's bug. The issue stayed open for one question: should #702's similarity floors (0.30 / 0.50) become the sliding-window streaming default? Measured on main b3eda84, streaming `SlidingWindowAsrManager` with the CLI's default `--streaming` config: | | Floors off (default) | Floors on | |---|---|---| | Earnings-22 keyword clips (772): TP/FP/FN | 1220/4/54 | 1082/2/192 | | Earnings-22: WER | 26.10% | 19.68% | | FLEURS en_us (200) with an unrelated term list: false term insertions | 332 | 40 | Floors on lose 138 real brand and personal names (recall 95.8% → 84.9%) and gain none, but they stop the rescue pass firing on unrelated speech. Batch shows the same trade-off. The default stays off. This PR documents the floors in `CustomVocabulary.md` as an opt-in, with these numbers and the `VocabularyRescorer.Config` to enable them. Since #1100, floors-off false insertions on the FLEURS set are 220. Docs only. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Speaker Diarization Benchmark ResultsSpeaker Diarization PerformanceEvaluating "who spoke when" detection accuracy
Diarization Pipeline Timing BreakdownTime spent in each stage of speaker diarization
Speaker Diarization Research ComparisonResearch baselines typically achieve 18-30% DER on standard datasets
Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:
🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 67.7s diarization time • Test runtime: 4m 53s • 11/01/2026, 05:14 PM EST |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #967.
Why is this change needed?
On main, the default
VocabularyBoostingSessionstill replaces common words that were actually said with a listed term. The term wins only because of the context-biasing weight (cbw 4.5–6): CTC prefers the original word by 2–5 points, and the boost overturns that. Examples:The fixes it does make are all on words TDT was unsure of. On the 13 #967 clips, every correct fix has a minimum token confidence of 0.70 or less ("Sarab" → "Saurabh", "Envious Whisper" → "EnviousWispr"). The common-word swaps sit at 0.89–1.00.
What changes?
When TDT decoded every word of the span at ≥ 0.99 token confidence, the term gets no bias against it and must beat the span on CTC evidence alone. This reuses the
min(0, boost)gate that already guards adjacent duplicates. Close spellings keep their bias (≥ 0.80 similarity, possessive aside, e.g. "Quiltor's" / "Quilter", "Rashid" / "Rasyid"), so the listed spelling still wins there.WordTimingnow carries its lowest token confidence.Measured on main b3eda84
Earnings-22 keyword set (772 clips) plus the #967 13 clips and their 12 visible terms. The false-positive set is 200 FLEURS en_us clips with the #967 terms, and again with 50 terms (those 12 plus 38 from the Earnings-22 dictionaries).
The 4 Earnings-22 TPs lost were wrong swaps on main that scored only because the term occurs elsewhere in the clip: "line of" → "Nico", "releases" → "Witcher", "carbon" → "Jon", "black Friday" → "Black Fin". Each replaced word is in the reference.
Not fixed
Tests
sort-of-shell: a real FLEURS fixture (CC BY 4.0, provenance and SHA-256 in the fixture README). Main turns "sort" into "Sora", and the test fails without the gate.swift format lintis clean.Scope: the gate is on in
VocabularyRescorer.Config.default, which the TDT batch and sliding-window paths use, and where it was measured.VocabularyBoostingSession.itnDefaultConfig, which the Unified engines use, turns it off (confidentWordBiasGate: false): those engines feed emission probabilities as token confidence, a scale that wasn't measured. Their behaviour is unchanged.🤖 Generated with Claude Code