Session start blocks the loop in AudioStream.from_track, and xai/stt-1 init fails with Streaming ASR unavailable (2006)

[bml1g12]

Bug Description

I have observed a warning and error when using the below setup, it does not seem to obviously cause anything to break, but I as far as I can see it is coming from the framework rather than my own code, so likely is a bug.

Two failures show up in the first second after a remote microphone track is subscribed. They share that window. They are not the same failure.

1. livekit.agents logs event loop blocked for about 106–119 ms. The sampled stack is inside the SDK, not application code. It ends in a synchronous FFI call:

room.py _listen_task
  -> room.py _on_room_event
  -> room_io/_input.py _on_track_available
  -> _create_stream
  -> rtc.AudioStream.from_track
  -> audio_stream.py _create_owned_stream
  -> livekit/rtc/_ffi_client.py:295 request
  -> livekit_ffi_request

The measured stall had threshold 0.1, gc_time about 0, and cpu_time about equal to duration. The loop thread was busy in native code, not descheduled. This only crossed 100 ms when noise_cancellation.BVC() was passed to AudioInputOptions. The same mic subscribe without BVC stayed under the threshold and did not warn. There is no app-level way to move that request() off the job loop. The warning text says to move the work to a thread or an async client, but the call is inside livekit.rtc.

2. livekit.agents.inference.stt raises APIStatusError from recv_task, then logs the same error again as failed to recognize speech and retries. This happened with and without BVC, including a session that did not print the loop warning. A later session with the same agent hit the loop warning and did not hit the STT error, so the provider reject is intermittent but fairly reproducible.

livekit.agents._exceptions.APIStatusError: message='LiveKit Inference STT returned error: failed to send initialization message: STT provider rejected the session before it was ready: Streaming ASR unavailable: code: \'The operation was cancelled\', message: "operation was canceled", source: tonic::transport::Error(Transport, hyper::Error(Canceled, "connection closed"))', status_code=2006, retryable=True

The body is type=error, code=2006. The nested source is tonic::transport::Error(Transport, hyper::Error(Canceled, "connection closed")).

Expected Behavior

Subscribing to a microphone track, including with noise_cancellation.BVC(), should not run a synchronous FFI request on the job event loop long enough to trip the 100 ms loop monitor. If that FFI call is inherently synchronous, the monitor should not tell the application to move it to a thread.

inference.STT(model="xai/stt-1") should finish session init, or retry inside the gateway before the job logs an ERROR for a retryable=True reject that happens before the session is ready.

Reproduction Steps

This is the voice quickstart, not an application-specific worker. Python 3.13. `livekit` 1.1.18, `livekit-agents` 1.8.2. LiveKit Cloud, region Germany 2 on the repro runs.

`agent.py`:


from dotenv import load_dotenv
from livekit import agents
from livekit.agents import Agent, AgentServer, AgentSession, inference, room_io
from livekit.plugins import noise_cancellation

load_dotenv(".env.local")

class Assistant(Agent):
    def __init__(self) -> None:
        super().__init__(instructions="You are a helpful voice AI assistant.")

server = AgentServer()

@server.rtc_session(agent_name="lk-repro")
async def my_agent(ctx: agents.JobContext) -> None:
    session = AgentSession(
        stt=inference.STT(model="xai/stt-1", language="ja"),
        llm=inference.LLM(model="google/gemma-4-31b-it"),
        tts=inference.TTS(
            model="fishaudio/s2.1-pro",
            voice="fa4c9eb3dccc4806b382b40d61c6b10a",
        ),
    )
    await session.start(
        room=ctx.room,
        agent=Assistant(),
        room_options=room_io.RoomOptions(
            audio_input=room_io.AudioInputOptions(
                noise_cancellation=noise_cancellation.BVC(),
            ),
        ),
    )

if __name__ == "__main__":
    agents.cli.run_app(server)


Then:

1. Set `LIVEKIT_URL`, `LIVEKIT_API_KEY`, and `LIVEKIT_API_SECRET`.
2. Start `python agent.py dev --no-reload`. Wait for `registered worker`.
3. From another process, join that room as a participant and publish a `SOURCE_MICROPHONE` track of silence. Then explicitly dispatch `lk-repro` into the room. Leave the track publishing for a few seconds.
4. Read the worker log.

Operating System

Linux x86_64

Models Used

LiveKit Inference: xai/stt-1 (language=ja), google/gemma-4-31b-it, fishaudio/s2.1-pro. Room input uses livekit.plugins.noise_cancellation.BVC().

Package Versions

python==3.13
livekit==1.1.18
livekit-agents==1.8.2

Session/Room/Call IDs

Repro session that logged both, 2026-09-24:

room: lk-repro-1790250552
roomID: RM_tijrHmP3Ljd6
jobID: AJ_8gUqi34pNAqD
dispatchID: AD_mJwyWSjkcxXu
workerID: AW_52zRU2hsqLB9
STT session_id: 1dbf9861-6685-42b5-b716-79903d20f7fd

Loop stall on that run: 119 ms (duration 0.1187, cpu_time 0.1377).

A later run hit only the loop warning (106 ms) and no STT error:

room: lk-repro-1790251181
roomID: RM_pLecTJTxbAGz
jobID: AJ_KHQr7hjggQ8o

Proposed Solution

Make `AudioStream.from_track` / `FfiClient.request` non-blocking on the job loop when it has to initialize BVC, or run that request off the loop. A 100 ms stall in framework code is what the new loop monitor is for, but the log currently blames the application.

For STT, treat "provider rejected the session before it was ready" / `connection closed` during init as a gateway retry, not an `ERROR` in `recv_task`, when `retryable` is already true. The xAI streaming session is being canceled before the init message is accepted.

Additional Context

The loop warning comes from telemetry.loop_monitor (agents PR 7128, merged 2026-09-09). Warn threshold is 100 ms. These stalls are just over that line, so they are easy to miss on a warm process.

The STT ERROR and the following failed to recognize speech, retrying in 0.1s warning are the same exception logged twice. stt.py recv_task raises APIStatusError. SpeechStream._main_task catches it because retryable=True and logs the retry.

Screenshots and Recordings

No response