Skip to content

feat(serve): serve speech-to-text models through Lemonade (ROCMAI-672) - #724

Draft
rominf wants to merge 7 commits into
feat/service-api-docsfrom
feat/serve-speech-to-text
Draft

rominf wants to merge 7 commits into
feat/service-api-docsfrom
feat/serve-speech-to-text

Conversation

@rominf

@rominf rominf commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Why

ROCm CLI can only serve chat models, so a user who wants speech-to-text on their AMD GPU has no rocm serve path for it, even though Lemonade, which ROCm CLI already runs, ships whisper.cpp with ROCm and Vulkan builds. Serving a Whisper model as if it were a chat model fails in a less obvious way: every chat-shaped path (readiness probing with a chat completion, rocm chat, the Console chat pane, rocm bench load, the API docs) treats the speech server as something it is not.

What changes

  • A model task. Recipes and service records carry what a model does: chat (the default, and what every existing record reads as) or transcription. A task value written by a newer ROCm CLI reads as unknown rather than as chat. Such a server is never probed or offered as chat, and restarting, recovering or serving it is refused before anything is stopped, naming the newer version as what can serve it. The engine protocol carries the task in resolve/launch requests and in each engine's advertised tasks, omitted for chat so older engines and CLIs see unchanged messages.
  • Serving. rocm serve whisper (Whisper Large-v3-Turbo; whisper-tiny is the smallest) runs through Lemonade's router with the whispercpp recipe. The backend is named explicitly: ROCm, or Vulkan when Lemonade reports ROCm unsupported. Lemonade's CPU build is never selected, even when installed, and rocm serve's GPU gate still applies. Before reporting ready, the engine checks that Lemonade loaded the model on that backend and that a short silent recording transcribes with HTTP 200. vLLM refuses speech models, and rocm serve whisper --engine vllm is refused before the GPU gate with the Lemonade command that works.
  • Readiness is judged per task. A speech-to-text server is listed only once Lemonade's /v1/health shows it loaded on a GPU backend (its /v1/models lists every downloaded model). It is then probed with a transcription that must answer 200, never with a chat request. Another engine advertising speech-to-text is judged by its /v1/models, as its chat models are.
  • Output. The deployment summary names the task in place of the chat timing rows. The summary, the plain and attached launch output, and rocm services list print a speech-to-text endpoint line (from one shared helper) beside the unchanged base URL. Restarts and the helper's recovery relaunch the service with its task.
  • Chat paths stay off speech servers.
    • rocm chat --model whisper and rocm bench load refuse with the reason.
    • The chat provider no longer lists speech models.
    • The Console chat pane skips speech services, including when its well-known-port fallback lands on one.
    • The assistant's services tool names each server's task and, for speech, its transcription route.
  • API docs (from the base PR) describe a speech-to-text server by its task: POST /v1/audio/transcriptions with a file upload and the model already filled in, and no chat routes. Route lists per task live in rocm-engine-protocol (supported_routes), shared by Lemonade and the docs.

Verification

  • e2e, on every lane (mock servers, or none):
    • serve-26: refusal on vLLM with the working command.
    • serve-28: listed with task and endpoint, and that endpoint reaches the transcription route.
    • chat-10: chat refuses and says why.
    • chat-11: the interactive chat pane ignores a newer speech service (Linux lanes).
    • bench-07: bench refuses.
    • service-docs-05: docs list only the transcription route, and an upload through them is transcribed.
  • The transcription mock only answers a real multipart upload with a non-empty file and its own model id; service-docs-05 fails if the forwarder drops Content-Type. The mock runs on its own thread, and the steps send their transcription requests from a blocking thread, so harness workers blocked on CLI subprocesses cannot starve either side under full-lane concurrency.
  • serve-27 serves Whisper-Tiny on a real GPU, transcribes through the printed endpoint, checks the service list, and opens the printed API docs link (Lemonade GPU lanes, @requires-gpu @requires-engine:lemonade).
  • Unit tests cover:
    • task parsing and the protocol wire format;
    • backend selection, where a CPU-only listing is refused;
    • loaded-model health parsing, where a CPU-backend Whisper is not ready;
    • the transcription probe's 200 rule;
    • refusal of unknown tasks before stop/recovery, with the record left untouched;
    • restart argv carrying the task;
    • the provider, bench and dash refusals;
    • the services-tool text.
  • Manual: in headless Chromium, choosing a WAV under Swagger's "Try it out" for a transcription server sends a multipart upload with the model pre-filled through the docs page and shows the transcript.
  • Not verified here: a real Whisper serve (no GPU in the development environment; covered by serve-27 on the Lemonade GPU lanes).
  • Known gap: Lemonade reports a model's device from the backend name, so it is not trusted. That makes the loaded backend checkable but not where a Vulkan build actually ran. With no usable Vulkan device, whisper.cpp's Vulkan build may run on the CPU unreported, the same gap the chat path's Vulkan llama.cpp has. docs/engine-plugins.md states it; the ROCm backend is not affected.
  • Not covered by a test: the task-aware readiness wait after rocm services restart; only the restarted argv is unit-tested.

Scope

  • Kokoro and Moonshine, Lemonade's other speech models, are not offered: they run only on the CPU.

  • No rocm transcribe command; a client sends the request to the printed endpoint (README has a curl example).

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (Not a bug fix.)

  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration and dispatch wiring staying in main.rs/lib.rs is expected, not a violation). (engines/lemonade/src/speech.rs, crates/rocm-core/src/model_task.rs.)

  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.

@rominf

rominf commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Depends on #723: this branch is stacked on it, so the diff here shows only the speech-to-text change. It stays a draft until #723 merges, then it gets rebased onto main.

Of the servers rocm-cli runs, only vLLM generates API docs; Lemonade and
the llama.cpp server behind it serve none, so a user of `rocm serve` gets
an endpoint URL and no reference for what it accepts.

The background helper `rocm serve` already starts now serves docs for
every running model server on 127.0.0.1 (port 11430, `api_docs` in
config.json): an OpenAPI description per server built with utoipa from
the routes each engine reports, rendered by Swagger UI (vendored, works
offline). "Try it out" goes through a same-origin forwarder to the
server, so it works on every engine whatever its cross-origin settings.
The forwarder bypasses proxies, passes on only Content-Type, Accept and
Authorization, and refuses non-loopback Host and cross-site Origin
headers.

`rocm serve` and `rocm services list` print a `docs` link, but only
once the docs answer with that server's description, so a link is never
printed for docs that are off, a taken port, or an older helper.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Every served model was assumed to be a chat model: the protocol had no
notion of what a model does, readiness was proven with a chat request, and
the serve summary always printed /v1/chat/completions.

Add a model task (chat or transcription) to catalog recipes, service
records and the engine protocol. Every new field defaults to chat, so
records, state files and external engine plugins written before this
change keep their behaviour, and a plugin that does not advertise tasks
is treated as chat-only.

- rocm serve refuses a task the selected engine does not advertise
  before the GPU gate, naming an engine that can serve it.
- The Lemonade adapter serves transcription models through its router
  with an explicit whispercpp backend (ROCm, else Vulkan, never CPU) and
  proves a test transcription before reporting ready. Lemonade's
  per-model device label is not trusted: 11.5.1 reports ROCm Whisper as
  cpu, so the loaded backend is verified instead, as for chat.
- Readiness probes send a transcription for a speech-to-text service
  rather than a chat request, whose 404 would read as serving.
- The task survives restarts from both rocm and rocmd.
- vLLM advertises chat only and refuses other tasks itself.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Add Whisper-Large-v3-Turbo (alias `whisper`, `whisper-turbo`) and
Whisper-Tiny as speech-to-text recipes served by Lemonade.

A speech-to-text server cannot answer chat, so the commands that need a
chat model now recognise one and say why instead of failing obscurely:

- rocm chat --model <speech model> refuses before sending any request,
  and the built-in assistant selection never picks a speech service.
- The Console chat pane skips speech-to-text services, and its
  well-known-port fallback will not adopt a port one of them holds.
- rocm bench load refuses a model the catalog marks as speech-to-text.
- rocm services list shows the task and the transcription route for a
  speech-to-text server, and the assistant's services tool reports the
  task.

Scenarios cover the chat-only engine refusal, the services list entry
and its working endpoint, the chat and bench refusals, and (on the
Lemonade GPU lanes) a real Whisper serve answering on the printed
endpoint.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Describe `rocm serve whisper` and its transcription endpoint in the
README and `rocm serve --help`, the engine `tasks` capability and the
Lemonade whisper.cpp backend choice in the engine-plugin guide, a
manual GPU check for speech-to-text, the focused tests, and the two new
modules in the architecture map.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Review of the speech-to-text change found paths that still treated every
service as chat:

- rocmd recovery rebuilt the record as chat and relaunched it that way;
  the task is now carried over from the registry like the other fields.
- The CLI's own readiness wait after a launch or restart required a
  llama.cpp backend and sent a chat probe, so a Whisper service ended its
  launch as `starting`. It now checks for the model loaded through
  whispercpp on a GPU backend and probes with a transcription.
- An unrecognised task value failed the whole capabilities message,
  record or recipe index. It now degrades only its own entry; a recipe
  with such a task fails when it is served.
- A failed whisper.cpp backend install advised a command that only
  reinstalls the llama.cpp backend.

Also: `rocm services list` keeps `endpoint` as the base URL and adds a
speech-to-text endpoint line; the attached serve prints that line too;
the chat refusal says to drop --model; wording no longer promises more
than is checked (the CPU build is never selected). Adds the missing
regression tests (record, argv and recovery threading, readiness) and a
scenario for the interactive chat pane skipping a newer speech service.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A speech-to-text server answers the OpenAI transcription route, not
the chat routes, so its API docs listed routes it does not serve and
offered no way to try the one it does.

The docs now pick the routes from the server's task, from the same
per-task list the engines report as supported routes: a speech-to-text
server's page lists POST /v1/audio/transcriptions with an audio file
upload and the model already filled in, and no chat routes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
- A task recorded by a newer rocm-cli is refused before anything is
  stopped on restart, recovery and Lemonade resolve/launch, and engine
  state with an unparsable task reads as unknown rather than chat.
- A speech-to-text readiness probe needs a 200 transcript; a 4xx no
  longer counts as serving.
- Another engine advertising speech-to-text is judged by its model list
  instead of never becoming ready.
- The chat provider no longer lists speech-to-text models, and the
  assistant's services tool names their task and route.
- The task endpoint line has one shared helper.
- The e2e transcription mock runs on its own thread, and the steps send
  their transcription requests from a blocking thread, so harness workers
  blocked on CLI subprocesses cannot starve either side; the mock rejects
  anything but a real upload for its
  model, the GPU serve steps check the docs' routes per task again, and
  the Vulkan whisper.cpp GPU claim is stated as the known gap it is.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant