Skip to content

docs(asr/vocab): document the rescue floors as an opt-in, with the measured trade-off - #1101

Merged
Alex-Wengg merged 1 commit into
mainfrom
docs/899-rescue-floors
Nov 1, 2026
Merged

Alex-Wengg merged 1 commit into
mainfrom
docs/899-rescue-floors

Conversation

@Alex-Wengg

Copy link
Copy Markdown
Member

Closes #899.

#900 fixed #899's bug. The issue stayed open for one question: should #702's similarity floors (0.30 / 0.50) become the sliding-window streaming default? Measured on main b3eda84, streaming SlidingWindowAsrManager with the CLI's default --streaming config:

Floors off (default) Floors on
Earnings-22 keyword clips (772): TP/FP/FN 1220/4/54 1082/2/192
Earnings-22: WER 26.10% 19.68%
FLEURS en_us (200) with an unrelated term list: false term insertions 332 40

Floors on lose 138 real brand and personal names (recall 95.8% → 84.9%) and gain none, but they stop the rescue pass firing on unrelated speech. Batch shows the same trade-off. The default stays off. This PR documents the floors in CustomVocabulary.md as an opt-in, with these numbers and the VocabularyRescorer.Config to enable them. Since #1100, floors-off false insertions on the FLEURS set are 220.

Docs only.

🤖 Generated with Claude Code

…asured trade-off (#899)

Streaming and batch keep the floors off by default: on Earnings-22 they cost 138 brand and personal names, while they cut false insertions on unrelated speech from 332 to 40.
@Alex-Wengg
Alex-Wengg merged commit 608ec6a into main Nov 1, 2026
1 of 15 checks passed
@Alex-Wengg
Alex-Wengg deleted the docs/899-rescue-floors branch November 1, 2026 20:42
@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown

Speaker Diarization Benchmark Results

Speaker Diarization Performance

Evaluating "who spoke when" detection accuracy

Metric Value Target Status Description
DER 15.1% <30% ✅ Diarization Error Rate (lower is better)
JER 24.9% <25% ✅ Jaccard Error Rate
RTFx 17.16x >1.0x ✅ Real-Time Factor (higher is faster)

Diarization Pipeline Timing Breakdown

Time spent in each stage of speaker diarization

Stage Time (s) % Description
Model Download 12.590 20.6 Fetching diarization models
Model Compile 5.396 8.8 CoreML compilation
Audio Load 0.060 0.1 Loading audio file
Segmentation 18.331 30.0 Detecting speech regions
Embedding 30.552 50.0 Extracting speaker voices
Clustering 12.221 20.0 Grouping same speakers
Total 61.139 100 Full pipeline

Speaker Diarization Research Comparison

Research baselines typically achieve 18-30% DER on standard datasets

Method DER Notes
FluidAudio 15.1% On-device CoreML
Research baseline 18-30% Standard dataset performance

Note: RTFx shown above is from GitHub Actions runner. On Apple Silicon with ANE:

  • M2 MacBook Air (2022): Runs at 150 RTFx real-time
  • Performance scales with Apple Neural Engine capabilities

🎯 Speaker Diarization Test • AMI Corpus ES2004a • 1049.0s meeting audio • 61.1s diarization time • Test runtime: 4m 22s • 11/01/2026, 05:14 PM EST

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SlidingWindow streaming: default rescue floors replace an unrelated opening word with a vocabulary term ("Hey" → "Codex")

1 participant