fix(gladia): detect a silently dropped STT socket instead of hanging - #7404
pstepanovum wants to merge 3 commits into
Conversation
|
Pavel Stepanov seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Critical issues remain in the one-shot error path and regression-test setup.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (2)
What changed in this PR
This PR adds heartbeat monitoring and reconnect handling for dropped Gladia STT WebSocket connections.
Changes:
- Adds 30-second heartbeats to streaming and one-shot connections.
- Handles socket errors as retryable failures.
- Adds regression tests for heartbeat and reconnection behavior.
| File | Review findings |
|---|---|
tests/test_plugin_gladia_stt.py |
Critical (1 vote): The fake response lacks status, causing initialization to fail before testing WebSocket behavior. Moderate (1 vote): The fake iterator yields ERROR, unlike aiohttp’s real iterator; use receive() returning ERROR then CLOSED. |
livekit-plugins/livekit-plugins-gladia/livekit/plugins/gladia/stt.py |
Critical (1 vote): The one-shot path can consume heartbeat errors through the async iterator and return an empty transcript instead of raising a retryable/timeout error; it needs an explicit receive() loop. This also applies at line 1027. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| # Without this a silently dropped socket (half-open TCP, no FIN/RST) is | ||
| # only noticed once ws_receive expires, which is five times the connect | ||
| # timeout. The heartbeat closes the socket as soon as a ping goes | ||
| # unanswered, and that surfaces as WSMsgType.ERROR below. | ||
| heartbeat=30.0, |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
… socket The heartbeat closes the socket from aiohttp's side, so `async for` ends on WSMsgType.CLOSED and the ERROR branch in `_recognize_impl` never runs. The one-shot path then returned whatever it had collected, which on a dead socket is nothing, where the ws_receive timeout used to raise. Track whether post_final_transcript arrived and raise a retryable APIConnectionError with `ws.exception()` attached when it did not.
|
Thanks for the review. Both findings looked at, one was real. The one-shot path: fixed in 45dd74a. This one is right and I had missed it.
The test finding was a false positive. There are two All three tests pass and |

Same bug as #7206 (deepgram), #7357 (soniox), #7359 (telnyx), #7364 (xai) and #7369 (simplismart),
still present in the Gladia plugin.
On a half-open connection, no FIN and no RST, the Gladia
SpeechStreamsits on a dead socketforever. The session stays open, no transcripts arrive, and nothing is logged.
Two causes here:
ws_connectcall passesheartbeat=. aiohttp defaults it toNone, soasync for msg in self._wsnever returns and the retry in_main_task, which only runs whensomething raises, never gets a turn. On the one-shot
_recognize_implpath the only backstop isws_receive, set to five times the connect timeout.WSMsgType.ERRORfalls into the "Unexpected message type from Gladia" branch and the loopcontinues. That is where a heartbeat timeout arrives, because aiohttp closes the socket itself
when a ping goes unanswered rather than sending a close frame, and
ws.exception()is the onlyplace the reason survives.
Gladia has no keepalive task, so the third cause from the soniox and deepgram PRs does not apply,
and
_send_audio_taskalready reconnects on a write failure.The fix
heartbeat=30.0on bothws_connectcalls, matching the plugins above._recv_messages_taskends onWSMsgType.ERRORand raises a retryableAPIConnectionErrorwithws.exception()attached, so_main_taskreconnects and the reason reaches the log.closing_wsmoved onto the stream so the recv loop can tell a shutdown this side asked for froma socket that went away underneath it. Same guard as (deepgram stt): detect a silently dropped socket instead of hanging #7206 and fix(telnyx): detect a silently dropped STT socket instead of hanging #7359.
WSMsgType.CLOSEDhandling is unchanged. With a real socket aiohttp's async iterator stops onclose, so that branch does not run.
Tests
New
tests/test_plugin_gladia_stt.py, same shape as the tests added in #7206 and #7359. It runs areal
SpeechStreamand its real_runloop against fake sockets handed out by a fake aiohttpsession, so the connect kwargs and the reconnect behaviour are assertable without a network.
test_socket_is_opened_with_a_heartbeatassertsheartbeat=30.0reachesws_connect.test_heartbeat_timeout_reconnects_without_spinningfeedsWSMsgType.ERRORframes and assertsthe stream opens a second socket after exactly one of them, rather than logging three and
recovering by accident when the iterator happens to stop.
Both fail on
mainand pass with this change. Onmainthe second one times out after fiveseconds having logged "Unexpected message type from Gladia: 258" three times, which is the bug.
I run multilingual STT in production for telehealth sessions. A dead STT socket that logs nothing
mid-session is the failure mode this fixes.