Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 21 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -801,8 +801,9 @@ prints a deployment summary: a progress indicator while it starts, then a table
with the status, the full inference endpoint, a link to the server's API docs
(see API docs below), the API-qualified model name, and a
quick smoke test (time to first token and approximate tokens per second; a
speech-to-text model is checked with a test transcription before it reports
ready, and its summary names the task instead). Control
speech-to-text or text-to-speech model is checked with a test transcription or
one spoken word before it reports ready, and its summary names the task
instead). Control
returns to your shell with the server still running; manage it later with `rocm
services` (below).

Expand All @@ -828,7 +829,7 @@ The vLLM engine (Instinct) serves
A safetensors-only id has no GGUF build, so serving it through Lemonade fails
instead of silently substituting a different model.

#### Speech-to-text models
#### Speech models

`rocm serve whisper` serves Whisper Large-v3-Turbo through Lemonade
(`rocm serve whisper-tiny` serves the smallest Whisper model). A
Expand All @@ -842,13 +843,21 @@ curl http://127.0.0.1:11435/v1/audio/transcriptions \
-F model=Whisper-Large-v3-Turbo -F file=@recording.wav
```

Lemonade runs Whisper on its ROCm whisper.cpp build, or its Vulkan build when it
reports ROCm unsupported for the GPU; its CPU build is never selected, and
`rocm serve tts` serves OpenMOSS-TTS, a text-to-speech model, the same way: it
answers the OpenAI-compatible `POST /v1/audio/speech` route with audio (ask for
`"response_format": "wav"` to get a WAV file).
It is a 12.5 GB download and needs a GPU with about 14 GiB free. Lemonade's
smaller Kokoro text-to-speech model is not offered, because it runs only on the
CPU.

Lemonade runs Whisper and OpenMOSS-TTS on its ROCm build, or its Vulkan build
when it reports ROCm unsupported for the GPU; a CPU build is never selected, and
`rocm serve` refuses to start without a usable AMD GPU. vLLM serves chat
models only, so `rocm serve whisper --engine vllm` is refused with the command
that works. `rocm chat`, the Console chat pane, and `rocm bench load` need a
chat model: they do not use a speech-to-text server. `rocm chat` and `rocm
bench load` say so when you name one; the Console chat pane skips it.
models only, so `rocm serve whisper --engine vllm` (or `tts`) is refused with
the command that works. `rocm chat`, the Console chat pane, and `rocm bench
load` need a chat model: they do not use a speech-to-text or text-to-speech
server. `rocm chat` and `rocm bench load` say so when you name one; the Console
chat pane skips it.

Some models (such as Llama) are gated and require Hugging Face authentication.
Log in with `huggingface-cli login` or set `HF_TOKEN` in your environment
Expand Down Expand Up @@ -923,8 +932,9 @@ alone, without it):
```

The page lists the routes the server answers (the model list, the engine's
health check, and chat and text completions, or for a speech-to-text server the
transcription route, whose "Try it out" takes an audio file upload), with
health check, and chat and text completions; for a speech-to-text server the
transcription route, whose "Try it out" takes an audio file upload; for a
text-to-speech server the speech route, whose audio the page offers as a download), with
request examples already filled in with the served model. "Try it out" sends the request through the
docs page to the server, so it works for every engine without enabling
cross-origin access on the server; for a server started with an endpoint key,
Expand Down
3 changes: 2 additions & 1 deletion apps/rocm/src/dash.rs
Original file line number Diff line number Diff line change
Expand Up @@ -396,7 +396,8 @@ impl ManagedRegistryProbe for LiveManagedRegistryProbe {
/// `managed_service_endpoint_readiness`, which probes the endpoint — a model
/// listing bounded by `SERVICE_LIVENESS_CHECK_TIMEOUT`, and for a record
/// that lists but has not latched a verification, a real inference request
/// bounded by `INFERENCE_PROBE_TIMEOUT` (8s). Those are per record and would
/// bounded by `inference_probe_timeout` (8s, or 30s for text-to-speech).
/// Those are per record and would
/// be paid serially on the plain synchronous thread that `resolved_args`
/// runs on, delaying the dashboard's first paint by seconds on exactly the
/// hosts that serve models. It also *writes* records, which launching a
Expand Down
87 changes: 66 additions & 21 deletions apps/rocm/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -480,13 +480,15 @@ rocm model --verbose"
/// server running, or Ctrl-C to stop it. Inspect or stop servers later with
/// `rocm services`.
///
/// Speech-to-text models such as `whisper` are served through Lemonade and answer
/// on `/v1/audio/transcriptions` instead of chat; the summary prints that endpoint.
/// Speech-to-text models such as `whisper` and text-to-speech models such as `tts`
/// are served through Lemonade and answer on `/v1/audio/transcriptions` or
/// `/v1/audio/speech` instead of chat; the summary prints that endpoint.
#[command(after_help = "EXAMPLES:\n \
rocm serve qwen\n \
rocm serve qwen --engine vllm --port 8000\n \
rocm serve qwen --verbose --device gpu_required\n \
rocm serve whisper")]
rocm serve whisper\n \
rocm serve tts")]
Serve {
/// Model name, alias, or local model file path.
model: String,
Expand Down Expand Up @@ -4645,7 +4647,8 @@ fn apply_service_prune_plan(
/// age floor can — the window is not short and is not bounded, because
/// `spawn_managed_engine_child`'s idempotency guard runs `load_managed_services`
/// first, and that refreshes every `ready`/`running` record through a 750ms
/// listing plus an up-to-8s inference probe ([`rocm_core::INFERENCE_PROBE_TIMEOUT`]),
/// listing plus an inference probe of up to 8s, or 30s for text-to-speech
/// ([`rocm_core::inference_probe_timeout`]),
/// sequentially, over a record count the user controls.
///
/// **Scope: the whole operation, not just the scan.** Holding it across
Expand Down Expand Up @@ -6687,8 +6690,8 @@ pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result<String
if !record.task.is_chat() {
// `endpoint` stays the base URL for every service, so tools that
// take one keep working. Chat clients append `/chat/completions` to
// it, which a speech-to-text server does not have, so name the
// route it does serve.
// it, which a speech-to-text or text-to-speech server does not
// have, so name the route it does serve.
let _ = writeln!(output, " task: {}", record.task.display_name());
if let Some(line) = service_hints::task_endpoint_line(record.task, &record.endpoint_url)
{
Expand Down Expand Up @@ -8309,7 +8312,7 @@ fn refresh_managed_service_runtime_liveness(
record,
endpoint_api_key.as_deref(),
SERVICE_LIVENESS_CHECK_TIMEOUT,
rocm_core::INFERENCE_PROBE_TIMEOUT,
rocm_core::inference_probe_timeout(record.task),
)
} else {
EndpointReadinessOutcome {
Expand All @@ -8319,8 +8322,9 @@ fn refresh_managed_service_runtime_liveness(
};
let readiness = outcome.readiness;
if readiness == EndpointReadiness::Serving {
// Mirror `providers.rs::ready_local_services()`: a live, probe-passing
// service should be persisted as "ready", not left stuck at "running".
// As `providers.rs::ready_local_services()` does for chat services: a
// live, probe-passing service should be persisted as "ready", not left
// stuck at "running".
if record.status == "running" {
record.status = "ready".to_owned();
return true;
Expand Down Expand Up @@ -11464,12 +11468,19 @@ fn wait_for_service_http_ready_with_progress(
});
if listed {
best = EndpointReadiness::Listing;
// The task's probe budget, cut to what is left of this wait (at
// least a second, so a probe is never sent with no time at all): a
// text-to-speech probe would otherwise overrun the deadline by up
// to its whole 30 s.
let probe_timeout = rocm_core::inference_probe_timeout(task)
.min(timeout.saturating_sub(start.elapsed()))
.max(Duration::from_secs(1));
if rocm_core::openai_inference_probe(
task,
&endpoint,
canonical_model_id,
endpoint_api_key,
rocm_core::INFERENCE_PROBE_TIMEOUT,
probe_timeout,
)
.unwrap_or(false)
{
Expand Down Expand Up @@ -11589,12 +11600,13 @@ fn service_http_readiness_response_ready(

/// Whether one readiness response shows the model loaded, judged for its task.
///
/// On Lemonade a speech-to-text model is listed only by `/v1/health`, loaded
/// through `whispercpp` on a GPU backend: Lemonade's `/v1/models` lists every
/// downloaded model, loaded or not, so it says nothing about a Whisper model.
/// Any other engine that advertises speech-to-text is judged by its
/// `/v1/models`, as its chat models are; either way the transcription probe
/// that follows decides whether it actually serves.
/// On Lemonade a speech model (speech-to-text or text-to-speech) is listed only
/// by `/v1/health`, loaded through its speech recipe (`whispercpp`,
/// `openmoss`) on a GPU backend: Lemonade's `/v1/models` lists every downloaded
/// model, loaded or not, so it says nothing about one. Any other engine that
/// advertises a speech task is judged by its `/v1/models`, as its chat models
/// are; either way the probe for the task that follows decides whether it
/// actually serves.
fn service_http_readiness_response_ready_for_task(
task: rocm_core::ModelTask,
engine: &str,
Expand All @@ -11606,9 +11618,11 @@ fn service_http_readiness_response_ready_for_task(
if status != 200 {
return false;
}
if task == rocm_core::ModelTask::Transcription && engine == "lemonade" {
if engine == "lemonade"
&& let Some((recipe, backend_option)) = rocm_engine_lemonade::speech_model_health_keys(task)
{
return path == "/v1/health"
&& lemonade_health_has_transcription_model(body, canonical_model_id);
&& lemonade_health_has_speech_model(body, canonical_model_id, recipe, backend_option);
}
match (engine, path) {
("lemonade", "/v1/health") => lemonade_health_ready_for_model(body, canonical_model_id),
Expand Down Expand Up @@ -11636,7 +11650,12 @@ fn lemonade_health_ready_for_model(body: &str, canonical_model_id: &str) -> bool
})
}

fn lemonade_health_has_transcription_model(body: &str, canonical_model_id: &str) -> bool {
fn lemonade_health_has_speech_model(
body: &str,
canonical_model_id: &str,
recipe: &str,
backend_option: &str,
) -> bool {
let Ok(value) = serde_json::from_str::<serde_json::Value>(body.trim()) else {
return false;
};
Expand All @@ -11648,10 +11667,10 @@ fn lemonade_health_has_transcription_model(body: &str, canonical_model_id: &str)
let field = |key: &str| model.get(key).and_then(serde_json::Value::as_str);
field("model_name")
.is_some_and(|loaded| service_model_names_match(loaded, canonical_model_id))
&& field("recipe") == Some("whispercpp")
&& field("recipe") == Some(recipe)
&& model
.get("recipe_options")
.and_then(|options| options.get("whispercpp_backend"))
.and_then(|options| options.get(backend_option))
.and_then(serde_json::Value::as_str)
.is_some_and(|backend| matches!(backend, "rocm" | "vulkan"))
})
Expand Down Expand Up @@ -13643,6 +13662,32 @@ pub(crate) mod tests {
Ok(())
}

#[test]
fn a_text_to_speech_model_is_listed_only_when_loaded_on_a_gpu_openmoss_backend() {
let health = |recipe: &str, option: &str, backend: &str| {
format!(
r#"{{"all_models_loaded":[{{"model_name":"OpenMOSS-TTS","recipe":"{recipe}","recipe_options":{{"{option}":"{backend}"}}}}]}}"#
)
};
let ready = |body: &str| {
service_http_readiness_response_ready_for_task(
rocm_core::ModelTask::Speech,
"lemonade",
"/v1/health",
200,
body,
"OpenMOSS-TTS",
)
};
assert!(ready(&health("openmoss", "openmoss_backend", "rocm")));
assert!(ready(&health("openmoss", "openmoss_backend", "vulkan")));
assert!(!ready(&health("openmoss", "openmoss_backend", "cuda")));
assert!(
!ready(&health("whispercpp", "whispercpp_backend", "rocm")),
"a loaded speech-to-text model is not a text-to-speech one"
);
}

#[test]
fn a_whisper_model_on_lemonades_cpu_build_is_not_listed_as_ready() {
let health = |backend: &str| {
Expand Down
54 changes: 54 additions & 0 deletions apps/rocm/src/providers.rs
Original file line number Diff line number Diff line change
Expand Up @@ -541,6 +541,16 @@ fn ready_local_services(paths: &AppPaths) -> Result<Vec<ManagedServiceRecord>> {
if record.refresh_from_engine_state().unwrap_or(false) {
let _ = record.write();
}
// Chat never uses another kind of server, so it is never probed here:
// a speech probe can take far longer than a chat one, and every chat
// command would wait on it. It is still returned by its recorded
// status, so naming it as the chat model is refused with the reason.
if !record.task.is_chat() {
if matches!(record.status.as_str(), "ready" | "running") {
services.push(record);
}
continue;
}
// Probe with the service's key so a protected public service is not
// filtered out here (an anonymous /v1/models would 401) before
// `LocalProvider` can send the bearer token.
Expand Down Expand Up @@ -2265,6 +2275,50 @@ mod tests {
Ok(())
}

#[test]
fn chat_never_contacts_a_text_to_speech_service_yet_refuses_it_by_name() -> Result<()> {
let (root, paths) = temp_app_paths("local-provider-tts-service");
paths.ensure()?;
let server = spawn_models_only_server("OpenMOSS-TTS")?;

let mut tts = ManagedServiceRecord::new(
&paths,
"svc-tts",
"lemonade",
"tts",
"OpenMOSS-TTS",
"127.0.0.1",
server.port(),
"managed",
123,
None,
None,
None,
);
tts.task = rocm_core::ModelTask::Speech;
tts.status = "ready".to_owned();
tts.write()?;

let error = select_local_chat_service(&paths, Some("OpenMOSS-TTS"))
.expect_err("a text-to-speech service is not a chat target")
.to_string();
let status = LocalProvider { paths: &paths }.status()?;
let contacted = server.stop()?;
let _ = std::fs::remove_dir_all(root);

assert!(
error.contains("is served by `svc-tts` as a text-to-speech model, not a chat model"),
"{error}"
);
assert!(status.models.is_empty(), "{:?}", status.models);
// A text-to-speech probe can take 30 s; chat must not pay it.
assert_eq!(
contacted, 0,
"chat sent requests to the text-to-speech server"
);
Ok(())
}

#[test]
fn local_provider_refuses_a_speech_to_text_service_named_as_the_chat_model() -> Result<()> {
let (root, paths) = temp_app_paths("local-provider-speech-service");
Expand Down
44 changes: 44 additions & 0 deletions apps/rocm/src/serve_cmd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1100,6 +1100,50 @@ mod tests {
let _ = std::fs::remove_dir_all(root);
}

#[test]
fn a_text_to_speech_model_is_refused_by_vllm_and_its_named_remedy_passes() {
let recipe = resolve_builtin_model_recipe("tts").expect("tts recipe");
assert_eq!(
recipe.model_task().expect("valid task"),
rocm_core::ModelTask::Speech
);
let root = std::env::temp_dir().join(format!(
"rocm-serve-tts-task-check-{}",
rocm_core::unix_time_millis()
));
let paths = AppPaths {
config_dir: root.join("config"),
data_dir: root.join("data"),
cache_dir: root.join("cache"),
};

let error = ensure_engine_serves_task(
&paths,
"vllm",
rocm_core::ModelTask::Speech,
"tts",
Some(&recipe),
)
.expect_err("vLLM advertises chat only")
.to_string();
assert_eq!(
error,
"`tts` is a text-to-speech model, which the vllm engine cannot serve; \
serve it with `rocm serve tts --engine lemonade`"
);
// The named remedy passes this same check; the nightly GPU scenario
// `@id:serve-text-to-speech-model-speaks-on-the-printed-endpoint` runs it.
ensure_engine_serves_task(
&paths,
"lemonade",
rocm_core::ModelTask::Speech,
"tts",
Some(&recipe),
)
.expect("Lemonade advertises text-to-speech");
let _ = std::fs::remove_dir_all(root);
}

#[test]
fn unsupported_task_message_offers_no_engine_when_the_recipe_names_none_other() {
assert_eq!(
Expand Down
2 changes: 1 addition & 1 deletion apps/rocm/src/serve_summary.rs
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String {
]
};

// Token timings describe text generation. A speech-to-text server has none
// Token timings describe text generation. A speech server has none
// to report, and two permanent `n/a` rows would read as a failed
// measurement; it names its task instead, since its endpoint is not the one
// chat clients expect.
Expand Down
Loading
Loading