diff --git a/docs/deployments/container/accessing-images.mdx b/docs/deployments/container/accessing-images.mdx index fda15cae..6246d102 100644 --- a/docs/deployments/container/accessing-images.mdx +++ b/docs/deployments/container/accessing-images.mdx @@ -131,7 +131,7 @@ This image contains both models. To load only one of them, see [Running only one ### Melia 1 model -Melia 1 is a multilingual, GPU-only model, currently available in Batch. For an overview, see [Models](/speech-to-text/models#melia-1). +Melia 1 is a multilingual, GPU-only model, generally available in Batch, with Real-time (streaming) available as an early preview. For an overview, see [Models](/speech-to-text/models#melia-1). {`# pulling the Melia 1 Batch transcriber with the ${smVariables.latestMelia1ContainerVersion} tag: diff --git a/docs/deployments/container/gpu-speech-to-text-melia-1.mdx b/docs/deployments/container/gpu-speech-to-text-melia-1.mdx index 66741298..d3d125e1 100644 --- a/docs/deployments/container/gpu-speech-to-text-melia-1.mdx +++ b/docs/deployments/container/gpu-speech-to-text-melia-1.mdx @@ -10,8 +10,7 @@ import melia1GpuSttDockerCompose from './assets/gpu-stt-melia-1-docker-compose.y # GPU Speech to text container (Melia 1) -Melia 1 is a multilingual, GPU-only model, currently available in Batch. For an overview of the model, see [Models](/speech-to-text/models#melia-1). - +Melia 1 is a multilingual, GPU-only model, currently available in Batch. For an overview, see [Models](/speech-to-text/models#melia-1). ## Prerequisites - [A license file or a license token](/deployments/container/licensing) @@ -87,14 +86,15 @@ I0705 08:12:55.598672 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0: The Melia 1 inference server currently runs in _batch_ mode only, processing whole files and returning the transcript at the end. -Once the inference server is running, run the Melia 1 transcriber with the `SM_INFERENCE_ENDPOINT` environment variable set to the gRPC endpoint of the inference server. The transcriber offloads inference to the server, so it does not require a GPU: +Once the inference server is running, run the Melia 1 transcriber with the `SM_INFERENCE_ENDPOINT` environment variable set to the gRPC endpoint of the inference server. The transcriber offloads inference to the server, so it does not require a GPU. The transcriber is a single unified image for both batch and real-time, so the `SM_ASR_MODE` environment variable must also be set, to `batch` or `rt`: ```bash docker run --rm -it \ -e SM_INFERENCE_ENDPOINT=: \ + -e SM_ASR_MODE=batch \ -v $PWD/license.json:/license.json \ -v $PWD/example.wav:/input.audio \ - + speechmaticspublic.azurecr.io/sm-asr-transcriber-melia-1:${smVariables.latestMelia1ContainerVersion} ``` To accept multiple jobs over an HTTP API without restarting the container between jobs, run the transcriber as a [batch persistent worker](/deployments/container/batch-persistent-worker). diff --git a/docs/private/assets/gpu-stt-melia-1-rt-docker-compose.yml b/docs/private/assets/gpu-stt-melia-1-rt-docker-compose.yml new file mode 100644 index 00000000..dfc38f61 --- /dev/null +++ b/docs/private/assets/gpu-stt-melia-1-rt-docker-compose.yml @@ -0,0 +1,36 @@ +version: "3.8" + +networks: + transcriber: + driver: bridge + +services: + triton: + image: speechmaticspublic.azurecr.io/sm-asr-inference-server-melia-1:{{ CONTAINER_VERSION }} + deploy: + resources: + reservations: + devices: + - driver: nvidia + ### Limit to N GPUs + # count: 1 + ### Pick specific GPUs by device ID + # device_ids: + # - 0 + # - 3 + capabilities: + - gpu + container_name: triton + networks: + - transcriber + expose: + - 8000/tcp + - 8001/tcp + - 8002/tcp + environment: + - NVIDIA_DRIVER_CAPABILITIES=all + - NVIDIA_VISIBLE_DEVICES=all + - CUDA_VISIBLE_DEVICES=0 + - SM_STREAMING=true + volumes: + - $PWD/license.json:/license.json:ro diff --git a/docs/private/melia-1-realtime.mdx b/docs/private/melia-1-realtime.mdx index c4cbe21b..6e4b5939 100644 --- a/docs/private/melia-1-realtime.mdx +++ b/docs/private/melia-1-realtime.mdx @@ -7,6 +7,7 @@ import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; import meliaRealtimePythonExample from "./assets/melia-1-rt-example.py?raw"; import meliaRealtimeJsExample from "./assets/melia-1-rt-example.js?raw"; +import melia1GpuSttRtDockerCompose from './assets/gpu-stt-melia-1-rt-docker-compose.yml?raw'; # Melia 1 Realtime Preview - Multilingual transcription @@ -35,6 +36,82 @@ The preview runs on a dedicated endpoint, separate from the production Realtime The preview is deployed in France (EU) and Oregon (USA), and is available on SaaS only. +### Real-time inference on prem + +:::warning +Real-time (streaming) inference on prem for Melia 1 is an early preview for evaluation only, and is not intended for production use. +::: + +The Melia 1 inference server can run in two modes: _batch_ or _real-time_. The default mode is batch. To configure the server for real-time, set the `SM_STREAMING` environment variable to `true` when starting the inference server: + + + {`docker run --rm -it \\ + -v $PWD/license.json:/license.json \\ + --gpus '"device=0"' \\ + -e CUDA_VISIBLE_DEVICES \\ + -e SM_STREAMING=true \\ + -p 8001:8001 \\ + speechmaticspublic.azurecr.io/sm-asr-inference-server-melia-1:1.7.0 +`} + + +When the Container starts you should see output similar to this, +indicating that the server has started and is ready to serve requests. + +``` +I0911 11:04:02.321449 1 server.cc:709] ++--------------------+---------+--------+ +| Model | Version | Status | ++--------------------+---------+--------+ +| aed | 1 | READY | +| body | 1 | READY | +| preprocessor | 1 | READY | +| streaming | 1 | READY | +| streaming_ensemble | 1 | READY | ++--------------------+---------+--------+ +... +I0911 11:04:02.449550 1 grpc_server.cc:2579] "Started GRPCInferenceService at 0.0.0.0:8001" +I0911 11:04:02.449744 1 http_server.cc:4961] "Started HTTPService at 0.0.0.0:8000" +I0911 11:04:02.491582 1 http_server.cc:400] "Started Metrics Service at 0.0.0.0:8002" +``` + +The server can only support one of these modes at once. + +Once the inference server is running in real-time mode, run the Melia 1 transcriber as normal, with the `SM_INFERENCE_ENDPOINT` environment variable set to the gRPC endpoint of the inference server. Unlike the [Standard and Enhanced GPU container](/deployments/container/gpu-speech-to-text#batch-and-realtime-inference), Melia 1 uses the same transcriber image for both batch and real-time — there are no separate `rt-` and `batch-` prefixed images. Set `SM_ASR_MODE=rt` to run the transcriber in real-time mode: + +```bash +docker run --rm -it \ + -e SM_INFERENCE_ENDPOINT=: \ + -e SM_ASR_MODE=rt \ + -v $PWD/license.json:/license.json \ + -p 9000:9000 \ + speechmaticspublic.azurecr.io/sm-asr-transcriber-melia-1:1.7.0 +``` + +Connect over WebSocket as described in [Realtime transcription](/deployments/container/cpu-speech-to-text#realtime-transcription). + +### Multi-session containers + +By default the real-time container will accept only one websocket connection at a time. To enable multiple connections, set the environment variable `SM_MAX_CONCURRENT_CONNECTIONS` to the maximum number of sessions to allow. When this is set, the `/ready` [health check endpoint](/deployments/container/cpu-speech-to-text#health-service) will return true if there is a free connection available. + +CPU usage scales with the number of active sessions, whereas most memory usage is shared between connections. + +### Monitoring the server + +The inference server is based on [Nvidia's Triton architecture](https://developer.nvidia.com/nvidia-triton-inference-server) and as such +can be monitored using Triton's inbuilt Prometheus metrics, or the GRPC/HTTP APIs. To expose these, configure an external mapping for port +8002(Prometheus) or 8000(HTTP). + +## Docker compose example + +This Docker Compose file will create a Speechmatics Melia 1 GPU Inference Server: + +(assumes your `license.json` file is in the current working directory) + + +{melia1GpuSttRtDockerCompose.replace('{{ CONTAINER_VERSION }}', '1.7.0')} + + ### Minimal example Example minimal `StartRecognition` message: @@ -127,8 +204,8 @@ Message format: | Word timings | ✅ | | | Language labeling | ✅ Word-level | | | Alphanumeric smart formatting | ✅ Improved accuracy, e.g. recognition of email/web addresses transcription | | -| Language hints (guide predicted languages) | ⚠️ Not yet | Targeting August 2026 | -| End of Utterance / Force end of Utterance | ⚠️ Not yet | Targeting August 2026 | +| Language hints (guide predicted languages) | ⚠️ Not yet | Targeting September 2026 | +| End of Utterance / Force end of Utterance | ⚠️ Not yet | Targeting September 2026 | | Speaker diarization | ⚠️ Not yet | Planned Q4 2026 | | Speaker identification | ⚠️ Not yet | Prioritized according to customer need | | `max_delay` / `max_delay_mode` latency control | ⚠️ Not yet. Finals latency averages around 4 seconds and varies | Lower latency and control planned Q3/Q4 2026 |