reduce memory usage across models

docs/input-limits.md

@@ -71,7 +71,7 @@ need and do not have a length gate.
 
 | Families | Limit source | Behavior |
 | --- | --- | --- |
-| qwen3_asr, canary_qwen, funasr_nano, granite, granite_nar, voxtral, cohere, canary | decoder context window (`dec_max_position_embeddings` / `dec_max_seq`), or the encoder positional table (`enc_pos_emb_max_len`, for cohere/canary) — all from GGUF | KV cache grows to fit, clamped to the model's true max. Over-length input is **rejected before the decode** (or before the encoder, where the encoder table is the binding limit) with `TRANSCRIBE_ERR_INPUT_TOO_LONG`. |
+| qwen3_asr, canary_qwen, funasr_nano, granite, granite_nar, voxtral, cohere, canary, moonshine_streaming | decoder context window (`dec_max_position_embeddings` / `dec_max_seq`), or a learned encoder/adapter positional table (`enc_pos_emb_max_len`, and `adapter.pos_emb` for moonshine_streaming) — all from GGUF | KV cache grows to fit, clamped to the model's true max. Over-length input is **rejected before the decode** (or before the encoder, where a positional table is the binding limit) with `TRANSCRIBE_ERR_INPUT_TOO_LONG`. |
 
 These families wrap an LLM-style decoder whose context window
 (`audio_tokens + prompt + generation`) is the binding constraint. The number of
@@ -81,6 +81,12 @@ computes the prefill size *before* running the encoder and rejects an
 over-length clip immediately — the caller never pays for a compute pass that
 cannot fit. The rejection goes through the log callback, not raw stderr.
 
+Moonshine Streaming is encoder-decoder rather than audio-LLM, but has the same
+hard-gate behavior: its 4096-row learned adapter position table receives one
+row per 20 ms encoder frame, imposing an exact **81.92 s** audio limit. This is
+also the limit reported by `max_audio_ms`; one-shot, batch, and streaming calls
+reject audio past it before an out-of-range embedding lookup can occur.
+
 The one case that cannot be predicted up front is the transcript itself running
 long enough to exhaust the remaining budget mid-decode (rare — the output would
 have to be very large for the audio length). There, the run returns the hard
@@ -95,7 +101,7 @@ status (the whole-batch call still returns `TRANSCRIBE_OK`).
 
 | Families | Window | Behavior |
 | --- | --- | --- |
-| gigaam (~25 s), sensevoice (~30 s), medasr (~400 s), moonshine (output-bound, ~48 s), moonshine_streaming (output-bound, ~17 min) | training / positional window | Any length is accepted; past the window the library emits a `WARN` (degraded accuracy is possible) and proceeds. `max_audio_ms` reports the window as advisory. |
+| gigaam (~25 s), sensevoice (~30 s), medasr (~400 s), moonshine (output-bound, ~48 s) | training / positional window | Any length is accepted; past the window the library emits a `WARN` (degraded accuracy is possible) and proceeds. `max_audio_ms` reports the window as advisory. |
 
 These families have no hard architectural wall but were trained on a bounded
 window; beyond it, accuracy degrades rather than failing. The library does not
@@ -109,11 +115,6 @@ Moonshine is the honest edge case in this bucket: its cap is on *output*
 audio length — a dense short clip can hit it too. It is reported via
 `transcribe_was_truncated()` and a `WARN` (and, offline, the hard
 `TRANSCRIBE_ERR_OUTPUT_TRUNCATED` status) when the cap is reached.
-`moonshine_streaming` has the same output-bound shape with a much larger window
-(`dec_max_position_embeddings = 4096`, ≈ 17 min); because it also streams, its
-truncation follows the streaming rule below — `stream_finalize` still returns
-`TRANSCRIBE_OK` and the truncation surfaces only through
-`transcribe_was_truncated()`.
 
 ## Context sizing and the `n_ctx` knob
 
@@ -138,9 +139,10 @@ reports the model's default-context ceiling (`n_ctx == 0`); it is not re-derived
 for a session that narrows `n_ctx`. A session that lowers `n_ctx` may therefore
 reject audio shorter than the advertised `max_audio_ms`.
 
-Encoder-bound families are different. For cohere and canary, the input-audio
-limit is the encoder positional table, while `n_ctx` only bounds the decoder
-self-KV / output budget. In those families `transcribe_session_get_limits()`
+Encoder-bound families are different. For cohere, canary, and
+moonshine_streaming, the input-audio limit is an encoder or adapter positional
+table, while `n_ctx` only bounds the decoder self-KV / output budget. In those
+families `transcribe_session_get_limits()`
 reports a smaller `effective_n_ctx` and `max_kv_bytes` when `n_ctx` is lowered,
 but `effective_max_audio_ms` stays pinned to the encoder input bound.
 
@@ -182,9 +184,8 @@ and has its own terminal-state machine (`transcribe_stream_*`,
 IDLE/ACTIVE/FINISHED/FAILED), and `stream_feed` / `stream_finalize` return the
 status of *that step*, not a verdict on the whole transcript. So when a
 streaming decode reaches its context cap (e.g. `voxtral_realtime` at its
-absolute position limit — hours of continuous audio, or `moonshine_streaming`
-at its output window), the stream does **not** fail and `stream_finalize`
-returns `TRANSCRIBE_OK`; the truncation is surfaced through
+absolute position limit — hours of continuous audio), the stream does **not**
+fail and `stream_finalize` returns `TRANSCRIBE_OK`; the truncation is surfaced through
 `transcribe_was_truncated(session)` and a `WARN`. This is deliberate:
 forcing a stream into a failed terminal state on truncation would discard the
 committed text the caller has been consuming. A streaming caller that needs to

docs/models/moonshine-streaming-tiny.md

@@ -14,6 +14,11 @@ multilingual capability, and does not emit timestamps.
 See Useful Sensors' [model card](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny)
 for training data, intended use, and upstream evaluation methodology.
 
+The learned adapter position table has 4096 rows at one row per 20 ms encoder
+frame, giving an exact **81.92-second** input limit. Longer one-shot, batch, or
+streaming input returns `TRANSCRIBE_ERR_INPUT_TOO_LONG`; split longer recordings
+into utterances before transcription.
+
 Licensed MIT. Ported from upstream commit
 [`f8e9dfd`](https://huggingface.co/UsefulSensors/moonshine-streaming-tiny/commit/f8e9dfd8c562c257c151a907b7b7f2fe8ff8511a),
 pinned 2026-05-06.

docs/models/voxtral-realtime.md

@@ -26,7 +26,9 @@ Real-time and offline speech-to-text from a 16 kHz mono WAV.
 - **Configurable streaming delay** — `--stream-voxtral-delay <N>` (default 6 =
   480 ms; range 80 ms–2.4 s) sets the transcription delay.
 - **Accuracy-first offline default** — one-shot and batch inference use the
-  publisher's most accurate evaluated delay, 30 tokens (2.4 s).
+  publisher's most accurate evaluated delay, 30 tokens (2.4 s). Long one-shot
+  inputs are internally advanced through the bounded incremental encoder and
+  decoder instead of constructing a full-clip attention graph.
 - Auto language detection (the streaming processor is auto-detect only).
 
 ## Input limits
@@ -82,11 +84,10 @@ CLI flags:
 - `--stream-chunk-ms <N>` — incremental streaming at N-ms chunk granularity.
 - `--stream-voxtral-delay <N>` — transcription delay in audio slots (default
   6 = 480 ms).
-- `--spec-k-drafts <N>` — offline-path 1-gram-lookup speculative decoding
-  draft length. `-1` (default) uses the family default (`2`). `0` disables
-  spec (plain autoregression). `1..8` selects an explicit K. Speculation
-  applies to `transcribe_run` / `transcribe-cli` only — the streaming path
-  is unaffected.
+- `--spec-k-drafts <N>` — short-clip offline-path 1-gram-lookup speculative
+  decoding draft length. `-1` uses the family default (`1`), `0` disables
+  speculation, and `1..8` selects an explicit K. Long one-shot inputs switch
+  to the bounded incremental path, which currently uses plain greedy decoding.
 
 ## Performance
 
@@ -132,14 +133,15 @@ uv run scripts/bench/run.py \
 
 ## Speculative decoding
 
-The offline decoder runs 1-gram-lookup speculative decoding by default. Each
-verify pass processes K+1 positions in parallel: position 0 is the model's
-true next-token decision; positions 1..K verify K draft tokens read from the
-1-gram suffix lookup over the already-decoded prefix. Drafts are accepted as
-long as the model's argmax matches the drafted token; the first mismatch ends
-the accepted prefix. Because ~60–70% of audio slots emit `STREAMING_PAD` (id
-32), the 1-gram lookup hits high acceptance during silence and during repeated
-phrases.
+The short-clip offline decoder runs 1-gram-lookup speculative decoding by
+default. Each verify pass processes K+1 positions in parallel: position 0 is
+the model's true next-token decision; positions 1..K verify K draft tokens read
+from the 1-gram suffix lookup over the already-decoded prefix. Drafts are
+accepted as long as the model's argmax matches the drafted token; the first
+mismatch ends the accepted prefix. Because ~60–70% of audio slots emit
+`STREAMING_PAD` (id 32), the 1-gram lookup hits high acceptance during silence
+and during repeated phrases. Long one-shot inputs use the bounded incremental
+scheduler and plain greedy decoding so encoder and decoder memory stay bounded.
 
 The transcript is byte-identical to the K=0 (no-spec) path; only wall-clock
 time changes.

docs/porting/families/moonshine_streaming.md

@@ -266,8 +266,9 @@ Highlights:
 7. `tie_word_embeddings=false` — converter must NOT tie, GGUF must carry
    an explicit `lm_head` tensor.
 8. `pad_token_id=0` (vs moonshine's 2). Tokenizer `vocab_sha256` differs.
-9. `max_position_embeddings=4096` (vs moonshine's 194). Decoder KV cache
-   sizing must accommodate the longer max length.
+9. `max_position_embeddings=4096` (vs moonshine's 194) sizes both the decoder
+   positions and the learned adapter position table. At one adapter row per
+   20 ms encoder frame, the table imposes a hard 81.92-second audio limit.
 
 ## Capability Validation
 
@@ -332,8 +333,7 @@ projection would.
    in encoder-frame units (with `frontend_pad = 4` enc frames of
    conv-stack history beyond the L_total mask context), encode, then
    on the emit slice `[T_emitted, stable_T)`:
-   - apply the adapter with absolute pos_ids → append to
-     `stream_adapter_committed`;
+   - apply the adapter with absolute pos_ids;
    - run the cross-KV projection graph → append per-layer K and V to
      `stream_cross_k_committed[il]` / `stream_cross_v_committed[il]`.
 4. `T_emitted = stable_T`. Bump `audio_committed_ms` to match.

docs/porting/families/voxtral_realtime.md

@@ -53,7 +53,9 @@ Pattern: **audio-llm**, streaming, ADDITIVE audio fusion.
   sliding-KV(8192) re-run incrementally; downsample_factor=4 enc frames per
   decode step (12.5 Hz); output length clamped to
   `ceil(mel_frames / audio_length_per_tok=8)`; configurable `num_delay_tokens`
-  (default 6 = 480 ms).
+  (default 6 = 480 ms). Long one-shot inference uses this same bounded
+  scheduler with the offline delay (30), while short clips retain the faster
+  whole-graph path.
 
 ## Family-specific requirements (do not flow through convert/validate)
 

include/transcribe/voxtral_realtime.h

@@ -47,9 +47,9 @@ extern "C" {
  *   min_decode_interval_ms
  *
  *     Minimum audio-time interval between tentative partial decodes while a
- *     stream is ACTIVE. Voxtral Realtime's partial decode reprocesses the
- *     accumulated buffer, so this knob bounds partial-decode compute at the
- *     cost of less frequent tentative transcripts. stream_finalize always
+ *     stream is ACTIVE. Voxtral Realtime advances its incremental encoder and
+ *     decoder when this interval elapses, so the knob trades partial-result
+ *     latency for larger compute batches. stream_finalize always
  *     performs the final decode regardless of this throttle, and that final
  *     decode is byte-identical to offline inference when both use delay 30.
  *

scripts/hf_cards/moonshine-streaming-tiny.yaml

@@ -34,8 +34,9 @@ summary: |
   English speech-to-text in both one-shot and streaming modes. A 34M-parameter
   encoder-decoder ASR model designed for streaming use (ergodic encoder +
   sliding-window attention, 50 Hz time-domain frontend). Takes a 16 kHz mono
-  WAV and produces a transcript. No translation, no multilingual capability,
-  no timestamps.
+  WAV and produces a transcript. The 4096-row learned adapter position table
+  limits each utterance to 81.92 seconds. No translation, no multilingual
+  capability, no timestamps.
 
 default_quant_index: 2  # Q8_0
 

src/arch/canary/canary.h

@@ -110,10 +110,6 @@ struct CanaryModel final : public transcribe_model {
     ggml_context *        bn_fused_ctx    = nullptr;
     ggml_backend_buffer_t bn_fused_buffer = nullptr;
 
-    // CPU-only F16 -> F32 promotion buffer for conformer 1x1 pointwise convs.
-    ggml_context *        conv_pw_f32_ctx    = nullptr;
-    ggml_backend_buffer_t conv_pw_f32_buffer = nullptr;
-
     std::optional<transcribe::MelFrontend> mel;
 
     CanaryModel() = default;

src/arch/canary/encoder.cpp

@@ -102,10 +102,12 @@ EncoderBuild build_encoder_graph(ggml_context *        ctx,
                                  bool                  use_flash,
                                  const char *          backend_name) {
     conf::ConvPolicy policy{};
-    policy.direct_pw               = conf::detect_direct_pw(backend_name);
-    const bool direct_dw           = detect_direct_dw_in_block(backend_name);
-    policy.direct_dw_in_block      = direct_dw;
-    policy.direct_dw_in_pre_encode = false;  // parakeet-style: im2col here
+    policy.direct_pw                = conf::detect_direct_pw(backend_name);
+    policy.promote_pw_in_graph      = backend_name != nullptr && std::strstr(backend_name, "CPU") != nullptr;
+    const bool direct_dw            = detect_direct_dw_in_block(backend_name);
+    policy.direct_dw_in_block       = direct_dw;
+    policy.direct_dw_in_pre_encode  = false;  // parakeet-style: im2col here
+    policy.pre_encode_dw_time_chunk = 256;
 
     EncoderBuild eb{};
 

src/arch/canary/model.cpp

@@ -182,14 +182,6 @@ CanaryModel::~CanaryModel() {
         safe_buffer_free(bn_fused_buffer);
         bn_fused_buffer = nullptr;
     }
-    if (conv_pw_f32_ctx != nullptr) {
-        ggml_free(conv_pw_f32_ctx);
-        conv_pw_f32_ctx = nullptr;
-    }
-    if (conv_pw_f32_buffer != nullptr) {
-        safe_buffer_free(conv_pw_f32_buffer);
-        conv_pw_f32_buffer = nullptr;
-    }
     if (ctx_meta != nullptr) {
         ggml_free(ctx_meta);
         ctx_meta = nullptr;
@@ -322,23 +314,6 @@ transcribe_status fuse_batch_norm(CanaryModel & m) {
     return TRANSCRIBE_OK;
 }
 
-// On CPU primary backend, dequantize 1x1 conformer pointwise convs
-// from F16 to F32. Same rationale as parakeet/cohere.
-transcribe_status promote_conv_pw_to_f32_on_cpu(CanaryModel & m) {
-    std::vector<load_common::ConvPwF32Slot> slots;
-    slots.reserve(m.weights.blocks.size() * 2);
-    for (auto & b : m.weights.blocks) {
-        if (b.conv_pw1_w != nullptr && b.conv_pw1_w->type == GGML_TYPE_F16) {
-            slots.push_back({ &b.conv_pw1_w, b.conv_pw1_w });
-        }
-        if (b.conv_pw2_w != nullptr && b.conv_pw2_w->type == GGML_TYPE_F16) {
-            slots.push_back({ &b.conv_pw2_w, b.conv_pw2_w });
-        }
-    }
-    return load_common::promote_conv_pw_f16_to_f32_on_cpu(m.plan, slots, "canary", &m.conv_pw_f32_ctx,
-                                                          &m.conv_pw_f32_buffer);
-}
-
 constexpr const char k_default_variant[] = "canary";
 
 extern transcribe_status load(Loader &, const transcribe_model_load_params *, transcribe_model **);
@@ -505,10 +480,6 @@ transcribe_status load(Loader & loader, const transcribe_model_load_params * par
     if (const transcribe_status st = fuse_batch_norm(*m); st != TRANSCRIBE_OK) {
         return st;
     }
-    if (const transcribe_status st = promote_conv_pw_to_f32_on_cpu(*m); st != TRANSCRIBE_OK) {
-        return st;
-    }
-
     m->t_load_us = ggml_time_us() - t_load_start;
     *out_model   = m.release();
     return TRANSCRIBE_OK;

src/arch/canary_qwen/encoder.cpp

@@ -108,9 +108,11 @@ EncoderBuild build_encoder_graph(ggml_context *            ctx,
                                  bool                      use_flash,
                                  const char *              backend_name) {
     conf::ConvPolicy policy{};
-    policy.direct_pw               = conf::detect_direct_pw(backend_name);
-    policy.direct_dw_in_block      = detect_direct_dw_in_block(backend_name);
-    policy.direct_dw_in_pre_encode = false;
+    policy.direct_pw                = conf::detect_direct_pw(backend_name);
+    policy.direct_dw_in_block       = detect_direct_dw_in_block(backend_name);
+    policy.direct_dw_in_pre_encode  = false;
+    policy.inplace_pre_encode       = true;
+    policy.pre_encode_dw_time_chunk = 256;
 
     EncoderBuild eb{};
 

src/arch/canary_qwen/model.cpp

@@ -713,24 +713,6 @@ transcribe_status init_context(transcribe_model *                model,
     cc->decoder_use_flash = true;
     transcribe::flash::apply_env_overrides(cc->encoder_use_flash, cc->decoder_use_flash);
 
-    auto * cm = static_cast<CanaryQwenModel *>(model);
-    {
-        ggml_type kv_type = GGML_TYPE_F16;
-        if (cc->kv_type == TRANSCRIBE_KV_TYPE_F32) {
-            kv_type = GGML_TYPE_F32;
-        }
-        if (!transcribe::causal_lm::kv_init(cc->kv_cache, cm->plan.primary,
-                                            /*n_ctx=*/2048, cm->hparams.dec_n_kv_heads, cm->hparams.dec_head_dim,
-                                            cm->hparams.dec_n_layers, kv_type)) {
-            transcribe::log_msg(TRANSCRIBE_LOG_LEVEL_ERROR,
-                                "canary_qwen init_context: KV cache allocation failed "
-                                "(n_ctx=2048, %d kv-heads x %d head-dim x %d layers) — "
-                                "out of memory.",
-                                cm->hparams.dec_n_kv_heads, cm->hparams.dec_head_dim, cm->hparams.dec_n_layers);
-            return TRANSCRIBE_ERR_OOM;
-        }
-    }
-
     *out_ctx = cc.release();
     return TRANSCRIBE_OK;
 }
@@ -924,7 +906,7 @@ transcribe_status run(transcribe_session *          context,
     // hold prompt + generation budget, rounded up to a power of two (the step
     // graph's flash-attn path wants pow2 attention width). A pre-allocated
     // smaller cache is freed and re-allocated.
-    int want_n_ctx = 1024;
+    int want_n_ctx = 256;
     while (want_n_ctx < T_prompt + k_max_new) {
         want_n_ctx *= 2;
     }

src/arch/cohere/cohere.h

@@ -138,12 +138,6 @@ struct CohereModel final : public transcribe_model {
     ggml_context *        bn_fused_ctx    = nullptr;
     ggml_backend_buffer_t bn_fused_buffer = nullptr;
 
-    // On CPU primary backend, the conformer 1×1 pointwise conv weights
-    // are dequantized F16->F32 at load time (Zen 2 has no native F16
-    // compute). Tensors live here; CohereBlock slots point at them.
-    ggml_context *        conv_pw_f32_ctx    = nullptr;
-    ggml_backend_buffer_t conv_pw_f32_buffer = nullptr;
-
     std::optional<transcribe::MelFrontend> mel;
 
     CohereModel() = default;

src/arch/cohere/encoder.cpp

@@ -112,10 +112,13 @@ EncoderBuild build_encoder_graph(ggml_context *        ctx,
                                  bool                  use_flash,
                                  const char *          backend_name) {
     conf::ConvPolicy policy{};
-    policy.direct_pw               = conf::detect_direct_pw(backend_name);
-    const bool direct_dw           = detect_direct_dw(backend_name);
-    policy.direct_dw_in_block      = direct_dw;
-    policy.direct_dw_in_pre_encode = direct_dw;
+    policy.direct_pw                = conf::detect_direct_pw(backend_name);
+    policy.promote_pw_in_graph      = backend_name != nullptr && std::strstr(backend_name, "CPU") != nullptr;
+    const bool direct_dw            = detect_direct_dw(backend_name);
+    policy.direct_dw_in_block       = direct_dw;
+    policy.direct_dw_in_pre_encode  = direct_dw;
+    policy.inplace_pre_encode       = !direct_dw;
+    policy.pre_encode_dw_time_chunk = 256;
 
     EncoderBuild eb{};