Skip to content

Whisper: start each clip_timestamps clip at its own start time - #1452

Open
shw1606 wants to merge 1 commit into
ml-explore:mainfrom
shw1606:fix/clip-timestamps-seek-clamp
Open

Whisper: start each clip_timestamps clip at its own start time#1452
shw1606 wants to merge 1 commit into
ml-explore:mainfrom
shw1606:fix/clip-timestamps-seek-clamp

Conversation

@shw1606

@shw1606 shw1606 commented Sep 9, 2026

Copy link
Copy Markdown

Problem

The clip loop checks only where a clip ends, never where it starts:

for seek_clip_start, seek_clip_end in seek_clips:
    while seek < seek_clip_end:

seek carries over from the previous clip, so the loop decodes the gap between
two clips as if you had asked for it. Pass three clips and you get back one
span running from the first clip's start to the last clip's end.

This file is a port of openai/whisper, which clamps seek on every
iteration:

while clip_idx < len(seek_clips):
    seek_clip_start, seek_clip_end = seek_clips[clip_idx]
    if seek < seek_clip_start:
        seek = seek_clip_start

The comment above that loop reads "A later commit should turn this into a
simpler nested loop", and it describes the nested form this file uses. The port
took the shape and left the clamp behind.

Fix

Clamp seek to the clip start on entering each clip, the nested-loop
equivalent of what upstream does. The line does nothing unless you pass
clip_timestamps.

Test

test_transcribe_clip_timestamps requests clip_timestamps="0,1,3,4" on the
bundled ls_test.flac and checks which windows the loop decodes. Setting
no_speech_threshold=None keeps every visited window in the output, so the
test can assert that each segment's seek lands inside a requested clip and
that the loop reaches both clips.

Window starts, in seconds:

visited
before [0.0, 1.0], and 1.0s sits inside the excluded 1s-3s gap
after [0.0, 3.0], the two clip starts

On main the second window decodes the gap and returns "Good soul openly
sorted the boat and sh...", text the caller asked to skip. The test fails on
main, passes with this change, and runs in 1.2s on the existing tiny
fixture.

End-to-end effect

141s of Korean speech: five utterances, 21s of speech, 30s gaps of digital
silence and inaudible noise between them. mlx-community/whisper-turbo on an
M1 Pro (16GB), macOS 26.5, mlx 0.32.2, clips from silero-vad. Median of
three trials with warm weights:

main this PR
wall clock 27.4s 4.2s
segments returned (5 expected) 30 5
segments outside the requested clips 26 0
consecutive duplicate segments 26 0
segment-start error vs ground truth 8.46s 0.23s

Drop clip_timestamps and the two builds agree: 4.26s vs 4.30s median and the
same output, as a no-op path should behave.

The slowdown needs an explanation, since skipping audio ought to save time.
segment_size is min(N_FRAMES, content_frames - seek, seek_clip_end - seek),
so once the loop walks into a gap, the next clip boundary truncates the window
to a sliver. Whisper decodes a long tail of short near-silent windows instead
of a few full ones, and most of those fall through to temperature fallback and
loop on repeated text. #1285 reports the same shape behind a silero-vad
front end: a 40s file taking "~3-4 seconds" without clip_timestamps and
"~27-28 seconds" with it. That points at this bug as the cause.

Reproduction

The test audio is five say -v Yuna utterances separated by 30s blocks of
digital silence and -50 dBFS noise. I take clips from
silero_vad.get_speech_timestamps, pad them by 0.3s, merge any pair less than
1s apart, and hand the result to clip_timestamps. The "outside the requested
clips" row counts segments whose [start, end] no requested clip contains. I
can attach the benchmark scripts.

The clip loop only checked each clip's end, so `seek` entered a clip
wherever the previous one left it and the gap in between was decoded as
if it had been requested. openai/whisper clamps `seek` to the clip start
on every iteration; the nested-loop port here dropped that clamp.

Besides returning text the caller asked to skip, decoding the gaps makes
each clip boundary truncate the window, so many short near-silent windows
get decoded instead of a few full ones. On a 141s file with five
utterances this took 27.4s and returned 30 segments (26 of them outside
the requested clips); it now takes 4.2s and returns 5. Likely the cause
of ml-explore#1285.

Add a regression test that checks which windows the loop visits.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant