Conversation
The qwen3_asr model supports over an hour of input audio. Remove the hard-coded 256 output token limit, which practically constrained input audio to around 1-2 minutes before running into output truncated error. Also remove duplicate code in several places.
So that run() and run_batch() are more parallel.
|
Have you considered #74? |
|
#74 looks nice! After that lands, then the scope of this PR would be reduced to providing convenient defaults even if |
|
Okay, I've decided that I'm going to close number 74 and this because fundamentally, certainly, we do need to resolve this, but there are other model families that this happens for, so we can't just do it for this model family. And I think the methodology probably needs to be changed in terms of how we handle this compared to how this PR was written What I mean is, I think our fundamental implementation is broken, and that we shouldn't have had this hard-coded constant to begin with, and we need a solution that crosses model families in a unified way so that other model families that come in in the future use a nice implementation rather than hard coding another constant |
2 participants
Summary
Problem: qwen3_asr model would truncate output on medium-length audio clips (anything longer than a minute, or 256 output tokens).
The original code had a hard-coded limit of 256 output tokens:
Even with the (very large) default context_size of 65536, the maximum number of generated output tokens was still limited to 256 (representing about a minute of spoken transcription text) even while the input could support thousands of minutes of input audio. Because of the fixed amount, the context allocation was effectively (99.7% audio, 0.3% output). This PR changes the context allocation strategy to a variable amount based on
--n-ctxor the model default (75% audio, 25% output tokens).Scope
The only file changed is
arch/qwen3_asr/model.cpp, and the behavior is functionally the same, except that now it won't truncate generated output on long audio inputs. When --n-ctx is not specified, memory usage will increase for short audio inputs, as the output token allocation will be large.Future work may consider a command line parameter to explicit set a gen_reserve override instead of automatically allocating 25% of the available context. This could be useful in edge cases of very memory-constrained environments that need precise control over audio/generated token allocation, or long recordings of someone speaking extremely quickly.
AI Assistance
No AI assistance was used to write, test, or comment the code.
Validation
There should not be any change to WER or numerical validation as a result of these changes.