Whisper: start each clip_timestamps clip at its own start time - #1452
Open
shw1606 wants to merge 1 commit into
Open
Whisper: start each clip_timestamps clip at its own start time#1452shw1606 wants to merge 1 commit into
shw1606 wants to merge 1 commit into
Conversation
The clip loop only checked each clip's end, so `seek` entered a clip wherever the previous one left it and the gap in between was decoded as if it had been requested. openai/whisper clamps `seek` to the clip start on every iteration; the nested-loop port here dropped that clamp. Besides returning text the caller asked to skip, decoding the gaps makes each clip boundary truncate the window, so many short near-silent windows get decoded instead of a few full ones. On a 141s file with five utterances this took 27.4s and returned 30 segments (26 of them outside the requested clips); it now takes 4.2s and returns 5. Likely the cause of ml-explore#1285. Add a regression test that checks which windows the loop visits. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The clip loop checks only where a clip ends, never where it starts:
seekcarries over from the previous clip, so the loop decodes the gap betweentwo clips as if you had asked for it. Pass three clips and you get back one
span running from the first clip's start to the last clip's end.
This file is a port of
openai/whisper, which clampsseekon everyiteration:
The comment above that loop reads "A later commit should turn this into a
simpler nested loop", and it describes the nested form this file uses. The port
took the shape and left the clamp behind.
Fix
Clamp
seekto the clip start on entering each clip, the nested-loopequivalent of what upstream does. The line does nothing unless you pass
clip_timestamps.Test
test_transcribe_clip_timestampsrequestsclip_timestamps="0,1,3,4"on thebundled
ls_test.flacand checks which windows the loop decodes. Settingno_speech_threshold=Nonekeeps every visited window in the output, so thetest can assert that each segment's
seeklands inside a requested clip andthat the loop reaches both clips.
Window starts, in seconds:
[0.0, 1.0], and 1.0s sits inside the excluded 1s-3s gap[0.0, 3.0], the two clip startsOn
mainthe second window decodes the gap and returns "Good soul openlysorted the boat and sh...", text the caller asked to skip. The test fails on
main, passes with this change, and runs in 1.2s on the existingtinyfixture.
End-to-end effect
141s of Korean speech: five utterances, 21s of speech, 30s gaps of digital
silence and inaudible noise between them.
mlx-community/whisper-turboon anM1 Pro (16GB), macOS 26.5, mlx 0.32.2, clips from
silero-vad. Median ofthree trials with warm weights:
mainDrop
clip_timestampsand the two builds agree: 4.26s vs 4.30s median and thesame output, as a no-op path should behave.
The slowdown needs an explanation, since skipping audio ought to save time.
segment_sizeismin(N_FRAMES, content_frames - seek, seek_clip_end - seek),so once the loop walks into a gap, the next clip boundary truncates the window
to a sliver. Whisper decodes a long tail of short near-silent windows instead
of a few full ones, and most of those fall through to temperature fallback and
loop on repeated text. #1285 reports the same shape behind a
silero-vadfront end: a 40s file taking "~3-4 seconds" without
clip_timestampsand"~27-28 seconds" with it. That points at this bug as the cause.
Reproduction
The test audio is five
say -v Yunautterances separated by 30s blocks ofdigital silence and -50 dBFS noise. I take clips from
silero_vad.get_speech_timestamps, pad them by 0.3s, merge any pair less than1s apart, and hand the result to
clip_timestamps. The "outside the requestedclips" row counts segments whose
[start, end]no requested clip contains. Ican attach the benchmark scripts.