feat: add automatic recovery for streaming ASR/TTS - #189
Conversation
Add resilient streaming wrappers that automatically reconnect on transient gRPC failures (UNAVAILABLE, DEADLINE_EXCEEDED, INTERNAL, etc.). - riva/client/retry.py: shared retry utilities with exponential backoff - riva/client/asr.py: ResilientStreamingASR with audio lookback buffer and final-transcript deduplication - riva/client/tts.py: ResilientStreamingTTS with segment-level retry - riva/client/auth.py: default gRPC keepalive for faster dead-connection detection - scripts/asr/transcribe_file.py: --auto-recover, --max-retries, --lookback-seconds - scripts/tts/talk.py: --auto-recover, --max-retries - tests/unit/test_retry.py: unit tests for retry logic
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
| uri: str = "localhost:50051", | ||
| metadata: Optional[List[Tuple[str, str]]] = None, | ||
| options: Optional[List[Tuple[str, str]]] = [], | ||
| options: Optional[List[Tuple[str, Union[str, int]]]] = None, |
There was a problem hiding this comment.
didn't get why this change is needed?
There was a problem hiding this comment.
You’re right — this change was unrelated to streaming recovery. I have reverted the auth.py change.
| "--auto-recover", | ||
| action="store_true", | ||
| help="Retry retryable streaming gRPC failures using a bounded audio lookback.", | ||
| ) |
There was a problem hiding this comment.
instead of adding new arguments to the script, can we utilize generic custom_configuration argument to pass these as key value pairs?
There was a problem hiding this comment.
Agreed. I removed the dedicated CLI arguments and now use reserved custom_configuration key-value pairs: client_auto_recover, client_max_retries, and client_lookback_seconds. These client-only keys are stripped before the request is sent to Riva.
|
|
||
|
|
||
|
|
||
| class ResilientStreamingASR: |
There was a problem hiding this comment.
instead of new wrapper classes, is it possible to add this logic into existing recognize/synthesize functions? default disabled and only take effect when custom_configuration arguments are sent
There was a problem hiding this comment.
removed the wrapper classes and moved the recovery logic into the existing streaming ASR and TTS methods. It remains disabled by default and is enabled only when the corresponding custom_configuration values are provided.
2 participants
Summary
Adds optional automatic recovery for streaming ASR and TTS when a transient gRPC failure occurs (
UNAVAILABLE,DEADLINE_EXCEEDED,INTERNAL,RESOURCE_EXHAUSTED, orABORTED).Changes
File | Change -- | -- riva/client/retry.py | Shared retryable-status detection and exponential backoff with jitter. riva/client/asr.py | Adds ResilientStreamingASR, which reconnects and replays a bounded recent PCM-audio lookback buffer. Recovery is best effort; callers that require exactly-once transcript output should handle duplicate transcripts after reconnect. riva/client/tts.py | Adds ResilientStreamingTTS, which retries a failed text segment and buffers that segment’s responses before delivery so retrying does not duplicate already delivered audio. Segment size is the latency/recovery trade-off. scripts/asr/transcribe_file.py | Adds --auto-recover, --max-retries, and --lookback-seconds. scripts/tts/talk.py | Adds --auto-recover and --max-retries. tests/unit/test_retry.py | Adds unit coverage for retryable status detection, backoff behaviour, and TTS partial-stream failure recovery.Usage
Validation