Skip to content

feat(serve): serve text-to-speech models through Lemonade (ROCMAI-673) - #725

Draft
rominf wants to merge 1 commit into
feat/serve-speech-to-textfrom
feat/text-to-speech-serving
Draft

rominf wants to merge 1 commit into
feat/serve-speech-to-textfrom
feat/text-to-speech-serving

Conversation

@rominf

@rominf rominf commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Why

With speech-to-text served (base PR), the other half of a local voice loop is still missing: there is no way to have ROCm CLI serve a text-to-speech model on an AMD GPU. Lemonade ships OpenMOSS-TTS with ROCm and Vulkan builds, so it fits the same serving path. A text-to-speech probe also behaves differently from the others: synthesizing one word takes far longer than a one-token chat answer, which the shared readiness timing did not allow for.

What changes

  • Serving. A speech model task (POST /v1/audio/speech) and an OpenMOSS-TTS recipe (aliases tts, moss-tts; a 12.5 GB download, about 14 GiB of GPU memory), served by Lemonade's openmoss recipe on its ROCm or Vulkan build; it has no CPU build.
    • Both speech tasks now go through one table of recipe, load flag, health key and weight format, so readiness and backend checks are shared.
    • Before reporting ready, the engine checks that the model is loaded on the chosen backend, and that one spoken word comes back with HTTP 200. It then records the server as verified, so the CLI does not repeat that probe.
    • vLLM refuses the task, and rocm serve tts --engine vllm is refused with the Lemonade command that works.
  • Probe timing. A text-to-speech probe gets 30 s instead of a chat probe's 8 s.
    • Each task's budget is explicit (inference_probe_timeout), replacing a floor hidden inside the request.
    • Failed probes are spaced by at least twice their budget, so a slow probe is never due again when it returns.
    • rocm serve's readiness wait caps a probe at its own deadline.
    • The chat provider no longer probes non-chat servers at all, so rocm chat never waits on a speech server.
  • API docs for a text-to-speech server list POST /v1/audio/speech, with an example naming the served model and an audio answer.
    • The forwarder gives audio/* answers an attachment disposition, and Swagger UI then offers the WAV as a download.
    • Without it, Swagger renders a player that re-requests the route with GET, which plays nothing.
  • Docs: --help, README, docs/engine-plugins.md and docs/architecture.md cover the new task.

Verification

  • e2e:
    • serve-30 (every lane): a chat-only engine refuses a text-to-speech model and names the Lemonade command that serves it.
    • serve-29 (nightly, Lemonade GPU lanes; the model is a 12.5 GB download) runs that command, checks the printed endpoint returns audio and the service list names its task, and opens the printed API docs link.
  • Unit tests cover:
    • Speech in the probe's 200-or-not-serving rule;
    • a failed text-to-speech probe not being retried 20 s later, where a chat probe is. This is checked at both throttles (the CLI's record and the engine's state file), and each test fails when its call site goes back to the fixed 15 s interval;
    • chat never contacting a text-to-speech server while still refusing it by name (fails with the skip removed);
    • the vLLM refusal naming rocm serve tts --engine lemonade, which passes the same capability check;
    • each speech task reading its own recipe's rows from lemonade backends;
    • resolving a text-to-speech model to the openmoss loader;
    • the docs' speech route and audio answer type;
    • audio forwarded byte for byte, with the download disposition.
  • Manual: in headless Chromium, Swagger's Execute on a stand-in text-to-speech server shows a "Download file" link that saves a valid WAV.
  • Not verified here: a real OpenMOSS serve (no GPU in the development environment), first-synthesis time on a cold Vulkan pipeline against the 30 s budget, and try-it through the docs against a real text-to-speech engine. serve-29 covers the serve on the nightly lane.
  • Known gap: as for whisper.cpp, a Vulkan OpenMOSS build with no usable Vulkan device may run on the CPU unreported; docs/engine-plugins.md states it.

Scope

  • Lemonade's Kokoro text-to-speech model is not offered: it runs only on the CPU.

  • Chat commands judge non-chat servers by their recorded status alone, without contacting them. A speech server whose engine died before its record caught up can still be named in a refusal as if it were running; that costs a misleading name in an error, where probing it would have cost every chat command up to 30 s.

  • The engine's post-load smoke test is a single attempt within the 30 s budget. A first synthesis slower than that fails the serve rather than being retried.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (Not a bug fix.)

  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md (the clap declaration and dispatch wiring staying in main.rs/lib.rs is expected, not a violation). (Extends engines/lemonade/src/speech.rs.)

  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. Covers claims of an outcome, remediation advice naming a command, and promises that something will not happen.

@rominf

rominf commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Depends on #724 (and through it #723): this branch is stacked on #724, so the diff here shows only the text-to-speech change. It stays a draft until both merge, then it gets rebased onto main.

Add a `speech` model task (`POST /v1/audio/speech`) and an OpenMOSS-TTS
recipe (aliases `tts`, `moss-tts`), served by Lemonade's `openmoss`
recipe on its ROCm or Vulkan build. Lemonade's Kokoro model is not
offered because it runs only on the CPU.

The Lemonade speech path serves both speech tasks from one table of
recipe, load flag, backend option and weight format, and proves a
text-to-speech model with one spoken word, answered with 200, before
reporting ready; that smoke test also records the server as verified,
so the CLI does not repeat it.

A text-to-speech probe has a 30 s budget instead of a chat probe's 8 s.
Each task's probe budget is now explicit, failed probes are spaced by
at least twice their budget so a slow probe is never due again when it
returns, the serve wait caps a probe at its own deadline, and chat
commands no longer probe servers that are not chat servers.

The API docs of a text-to-speech server list its speech route with an
example naming the served model. Audio answers are forwarded with an
attachment disposition, so the docs page offers them as a download:
without it Swagger UI renders a player that re-requests the route with
GET, which plays nothing.

serve-30 checks that a chat-only engine refuses a text-to-speech model
and names the Lemonade command; a nightly GPU scenario (serve-29)
serves the 12.5 GB model, checks the printed endpoint returns audio and
the service list names its task, and opens the printed API docs link.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant