Expose max_new_tokens run option - #74
vladkalinichencko wants to merge 2 commits into
Conversation
|
Thanks I will take a look tomorrow |
1d7e660 to
b14d32e
Compare
|
Sorry, it's taking me a bit longer to review this since it's a header change. I will want to take a closer look, which will just take some time |
|
No worries, thanks. I have rebased this onto the current main / 0.2.0 API, regenerated every binding and the Swift ABI hash, and rerun the native C API smoke test plus the Rust, TypeScript, Python, and Swift checks. The companion Handy PR cjpais/Handy#1633 is also rebased and conflict-free. |
# Conflicts: # bindings/python/src/transcribe_cpp/__init__.py # bindings/python/src/transcribe_cpp/_generated.py # bindings/rust/sys/src/transcribe_sys.rs # bindings/swift/Sources/TranscribeCpp/ABIHash.swift # bindings/typescript/src/_generated.ts # include/transcribe.abihash
|
ill be reviewing this and pulling in soon, thank you! |
|
I just want to say I appreciate you submitting this. I have definitely thought quite a bit about it and I think the bigger issue here is we shouldn't have done this in our implementation to begin with, and it's a bug in the fundamental implementation itself. And I don't think exposing this knob clarifies things or makes the API better. So I think I'm going to close this because fundamentally this is just a bug in the code itself. And I think more or less all of this can fall under the |
This exposes a per-run
max_new_tokensoption through the C API and all generated language bindings.Some autoregressive ASR families can accept the audio input but still hit the fixed generation budget before emitting EOS. That returns
OUTPUT_TRUNCATEDwith a partial transcript, but callers previously had no way to request a larger output budget.The default behavior stays unchanged: non-positive values use the existing family defaults. Qwen3-ASR and Canary honor a positive value for single and batch runs, bounded by the decoder context.
Rebased onto the current
main/ 0.2.0 API, preserving the newer diarization fields and generated ABI.Validated with:
cargo fmt --all --checkcargo check -p transcribe-cpptranscribe_api_smoketest