Skip to content

fix(gladia): detect a silently dropped STT socket instead of hanging - #7404

Open
pstepanovum wants to merge 3 commits into
livekit:mainfrom
pstepanovum:fix/gladia-stt-websocket-heartbeat
Open

pstepanovum wants to merge 3 commits into
livekit:mainfrom
pstepanovum:fix/gladia-stt-websocket-heartbeat

Conversation

@pstepanovum

Copy link
Copy Markdown

Same bug as #7206 (deepgram), #7357 (soniox), #7359 (telnyx), #7364 (xai) and #7369 (simplismart),
still present in the Gladia plugin.

On a half-open connection, no FIN and no RST, the Gladia SpeechStream sits on a dead socket
forever. The session stays open, no transcripts arrive, and nothing is logged.

Two causes here:

  • Neither ws_connect call passes heartbeat=. aiohttp defaults it to None, so
    async for msg in self._ws never returns and the retry in _main_task, which only runs when
    something raises, never gets a turn. On the one-shot _recognize_impl path the only backstop is
    ws_receive, set to five times the connect timeout.
  • WSMsgType.ERROR falls into the "Unexpected message type from Gladia" branch and the loop
    continues. That is where a heartbeat timeout arrives, because aiohttp closes the socket itself
    when a ping goes unanswered rather than sending a close frame, and ws.exception() is the only
    place the reason survives.

Gladia has no keepalive task, so the third cause from the soniox and deepgram PRs does not apply,
and _send_audio_task already reconnects on a write failure.

The fix

WSMsgType.CLOSED handling is unchanged. With a real socket aiohttp's async iterator stops on
close, so that branch does not run.

Tests

New tests/test_plugin_gladia_stt.py, same shape as the tests added in #7206 and #7359. It runs a
real SpeechStream and its real _run loop against fake sockets handed out by a fake aiohttp
session, so the connect kwargs and the reconnect behaviour are assertable without a network.

  • test_socket_is_opened_with_a_heartbeat asserts heartbeat=30.0 reaches ws_connect.
  • test_heartbeat_timeout_reconnects_without_spinning feeds WSMsgType.ERROR frames and asserts
    the stream opens a second socket after exactly one of them, rather than logging three and
    recovering by accident when the iterator happens to stop.

Both fail on main and pass with this change. On main the second one times out after five
seconds having logged "Unexpected message type from Gladia: 258" three times, which is the bug.

I run multilingual STT in production for telehealth sessions. A dead STT socket that logs nothing
mid-session is the failure mode this fixes.

Copilot AI lite review requested due to automatic review settings September 23, 2026 00:29
@pstepanovum
pstepanovum requested a review from a team as a code owner September 23, 2026 00:29
@CLAassistant

CLAassistant commented Sep 23, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ pstepanovum
❌ Pavel Stepanov


Pavel Stepanov seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Critical issues remain in the one-shot error path and regression-test setup.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity

Open (2)
What changed in this PR

This PR adds heartbeat monitoring and reconnect handling for dropped Gladia STT WebSocket connections.

Changes:

  • Adds 30-second heartbeats to streaming and one-shot connections.
  • Handles socket errors as retryable failures.
  • Adds regression tests for heartbeat and reconnection behavior.
File Review findings
tests/​test_plugin_gladia_stt.py Critical (1 vote): The fake response lacks status, causing initialization to fail before testing WebSocket behavior. Moderate (1 vote): The fake iterator yields ERROR, unlike aiohttp’s real iterator; use receive() returning ERROR then CLOSED.
livekit-plugins/​livekit-plugins-gladia/​livekit/​plugins/​gladia/​stt.py Critical (1 vote): The one-shot path can consume heartbeat errors through the async iterator and return an empty transcript instead of raising a retryable/timeout error; it needs an explicit receive() loop. This also applies at line 1027.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +369 to +373
# Without this a silently dropped socket (half-open TCP, no FIN/RST) is
# only noticed once ws_receive expires, which is five times the connect
# timeout. The heartbeat closes the socket as soon as a ping goes
# unanswered, and that surfaces as WSMsgType.ERROR below.
heartbeat=30.0,
Comment thread tests/test_plugin_gladia_stt.py Outdated
pstepanovum and others added 2 commits September 22, 2026 20:33
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
… socket

The heartbeat closes the socket from aiohttp's side, so `async for` ends on
WSMsgType.CLOSED and the ERROR branch in `_recognize_impl` never runs. The
one-shot path then returned whatever it had collected, which on a dead socket
is nothing, where the ws_receive timeout used to raise.

Track whether post_final_transcript arrived and raise a retryable
APIConnectionError with `ws.exception()` attached when it did not.
@pstepanovum

Copy link
Copy Markdown
Author

Thanks for the review. Both findings looked at, one was real.

The one-shot path: fixed in 45dd74a. This one is right and I had missed it. _handle_ping_pong_exception calls _set_closed(), so the next receive() returns WS_CLOSED_MESSAGE and __anext__ raises StopAsyncIteration. The async for in _recognize_impl ends before the ERROR branch runs, and the method returned whatever it had collected, which on a dead socket is nothing. Adding the heartbeat therefore turned a ws_receive timeout into a silent empty transcript on that path.

_recognize_impl now tracks whether post_final_transcript arrived and raises a retryable APIConnectionError with ws.exception() attached when it did not. test_recognize_raises_when_the_socket_dies_before_the_final_transcript covers it: without the guard it reports DID NOT RAISE, which is the empty-transcript case.

The test finding was a false positive. There are two _init_live_session methods in this file. The one on STT reads res.status, the one on SpeechStream calls res.raise_for_status(), and the two stream tests only ever reach the second. They passed before that change and pass now. The fake does need status for the new one-shot test, so the attribute is there either way.

All three tests pass and make check is green.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants