diff --git a/README.md b/README.md index 7183ebe1d1..8ea2815c13 100644 --- a/README.md +++ b/README.md @@ -801,8 +801,9 @@ prints a deployment summary: a progress indicator while it starts, then a table with the status, the full inference endpoint, a link to the server's API docs (see API docs below), the API-qualified model name, and a quick smoke test (time to first token and approximate tokens per second; a -speech-to-text model is checked with a test transcription before it reports -ready, and its summary names the task instead). Control +speech-to-text or text-to-speech model is checked with a test transcription or +one spoken word before it reports ready, and its summary names the task +instead). Control returns to your shell with the server still running; manage it later with `rocm services` (below). @@ -828,7 +829,7 @@ The vLLM engine (Instinct) serves A safetensors-only id has no GGUF build, so serving it through Lemonade fails instead of silently substituting a different model. -#### Speech-to-text models +#### Speech models `rocm serve whisper` serves Whisper Large-v3-Turbo through Lemonade (`rocm serve whisper-tiny` serves the smallest Whisper model). A @@ -842,13 +843,21 @@ curl http://127.0.0.1:11435/v1/audio/transcriptions \ -F model=Whisper-Large-v3-Turbo -F file=@recording.wav ``` -Lemonade runs Whisper on its ROCm whisper.cpp build, or its Vulkan build when it -reports ROCm unsupported for the GPU; its CPU build is never selected, and +`rocm serve tts` serves OpenMOSS-TTS, a text-to-speech model, the same way: it +answers the OpenAI-compatible `POST /v1/audio/speech` route with audio (ask for +`"response_format": "wav"` to get a WAV file). +It is a 12.5 GB download and needs a GPU with about 14 GiB free. Lemonade's +smaller Kokoro text-to-speech model is not offered, because it runs only on the +CPU. + +Lemonade runs Whisper and OpenMOSS-TTS on its ROCm build, or its Vulkan build +when it reports ROCm unsupported for the GPU; a CPU build is never selected, and `rocm serve` refuses to start without a usable AMD GPU. vLLM serves chat -models only, so `rocm serve whisper --engine vllm` is refused with the command -that works. `rocm chat`, the Console chat pane, and `rocm bench load` need a -chat model: they do not use a speech-to-text server. `rocm chat` and `rocm -bench load` say so when you name one; the Console chat pane skips it. +models only, so `rocm serve whisper --engine vllm` (or `tts`) is refused with +the command that works. `rocm chat`, the Console chat pane, and `rocm bench +load` need a chat model: they do not use a speech-to-text or text-to-speech +server. `rocm chat` and `rocm bench load` say so when you name one; the Console +chat pane skips it. Some models (such as Llama) are gated and require Hugging Face authentication. Log in with `huggingface-cli login` or set `HF_TOKEN` in your environment @@ -923,8 +932,9 @@ alone, without it): ``` The page lists the routes the server answers (the model list, the engine's -health check, and chat and text completions, or for a speech-to-text server the -transcription route, whose "Try it out" takes an audio file upload), with +health check, and chat and text completions; for a speech-to-text server the +transcription route, whose "Try it out" takes an audio file upload; for a +text-to-speech server the speech route, whose audio the page offers as a download), with request examples already filled in with the served model. "Try it out" sends the request through the docs page to the server, so it works for every engine without enabling cross-origin access on the server; for a server started with an endpoint key, diff --git a/apps/rocm/src/dash.rs b/apps/rocm/src/dash.rs index 0b3019adff..cf008e9f64 100644 --- a/apps/rocm/src/dash.rs +++ b/apps/rocm/src/dash.rs @@ -396,7 +396,8 @@ impl ManagedRegistryProbe for LiveManagedRegistryProbe { /// `managed_service_endpoint_readiness`, which probes the endpoint — a model /// listing bounded by `SERVICE_LIVENESS_CHECK_TIMEOUT`, and for a record /// that lists but has not latched a verification, a real inference request -/// bounded by `INFERENCE_PROBE_TIMEOUT` (8s). Those are per record and would +/// bounded by `inference_probe_timeout` (8s, or 30s for text-to-speech). +/// Those are per record and would /// be paid serially on the plain synchronous thread that `resolved_args` /// runs on, delaying the dashboard's first paint by seconds on exactly the /// hosts that serve models. It also *writes* records, which launching a diff --git a/apps/rocm/src/main.rs b/apps/rocm/src/main.rs index 01b1509622..430493262c 100644 --- a/apps/rocm/src/main.rs +++ b/apps/rocm/src/main.rs @@ -480,13 +480,15 @@ rocm model --verbose" /// server running, or Ctrl-C to stop it. Inspect or stop servers later with /// `rocm services`. /// - /// Speech-to-text models such as `whisper` are served through Lemonade and answer - /// on `/v1/audio/transcriptions` instead of chat; the summary prints that endpoint. + /// Speech-to-text models such as `whisper` and text-to-speech models such as `tts` + /// are served through Lemonade and answer on `/v1/audio/transcriptions` or + /// `/v1/audio/speech` instead of chat; the summary prints that endpoint. #[command(after_help = "EXAMPLES:\n \ rocm serve qwen\n \ rocm serve qwen --engine vllm --port 8000\n \ rocm serve qwen --verbose --device gpu_required\n \ -rocm serve whisper")] +rocm serve whisper\n \ +rocm serve tts")] Serve { /// Model name, alias, or local model file path. model: String, @@ -4645,7 +4647,8 @@ fn apply_service_prune_plan( /// age floor can — the window is not short and is not bounded, because /// `spawn_managed_engine_child`'s idempotency guard runs `load_managed_services` /// first, and that refreshes every `ready`/`running` record through a 750ms -/// listing plus an up-to-8s inference probe ([`rocm_core::INFERENCE_PROBE_TIMEOUT`]), +/// listing plus an inference probe of up to 8s, or 30s for text-to-speech +/// ([`rocm_core::inference_probe_timeout`]), /// sequentially, over a record count the user controls. /// /// **Scope: the whole operation, not just the scan.** Holding it across @@ -6687,8 +6690,8 @@ pub(crate) fn render_services_text(paths: &AppPaths, all: bool) -> Result lemonade_health_ready_for_model(body, canonical_model_id), @@ -11636,7 +11650,12 @@ fn lemonade_health_ready_for_model(body: &str, canonical_model_id: &str) -> bool }) } -fn lemonade_health_has_transcription_model(body: &str, canonical_model_id: &str) -> bool { +fn lemonade_health_has_speech_model( + body: &str, + canonical_model_id: &str, + recipe: &str, + backend_option: &str, +) -> bool { let Ok(value) = serde_json::from_str::(body.trim()) else { return false; }; @@ -11648,10 +11667,10 @@ fn lemonade_health_has_transcription_model(body: &str, canonical_model_id: &str) let field = |key: &str| model.get(key).and_then(serde_json::Value::as_str); field("model_name") .is_some_and(|loaded| service_model_names_match(loaded, canonical_model_id)) - && field("recipe") == Some("whispercpp") + && field("recipe") == Some(recipe) && model .get("recipe_options") - .and_then(|options| options.get("whispercpp_backend")) + .and_then(|options| options.get(backend_option)) .and_then(serde_json::Value::as_str) .is_some_and(|backend| matches!(backend, "rocm" | "vulkan")) }) @@ -13643,6 +13662,32 @@ pub(crate) mod tests { Ok(()) } + #[test] + fn a_text_to_speech_model_is_listed_only_when_loaded_on_a_gpu_openmoss_backend() { + let health = |recipe: &str, option: &str, backend: &str| { + format!( + r#"{{"all_models_loaded":[{{"model_name":"OpenMOSS-TTS","recipe":"{recipe}","recipe_options":{{"{option}":"{backend}"}}}}]}}"# + ) + }; + let ready = |body: &str| { + service_http_readiness_response_ready_for_task( + rocm_core::ModelTask::Speech, + "lemonade", + "/v1/health", + 200, + body, + "OpenMOSS-TTS", + ) + }; + assert!(ready(&health("openmoss", "openmoss_backend", "rocm"))); + assert!(ready(&health("openmoss", "openmoss_backend", "vulkan"))); + assert!(!ready(&health("openmoss", "openmoss_backend", "cuda"))); + assert!( + !ready(&health("whispercpp", "whispercpp_backend", "rocm")), + "a loaded speech-to-text model is not a text-to-speech one" + ); + } + #[test] fn a_whisper_model_on_lemonades_cpu_build_is_not_listed_as_ready() { let health = |backend: &str| { diff --git a/apps/rocm/src/providers.rs b/apps/rocm/src/providers.rs index 1d695a5521..1289a851d8 100644 --- a/apps/rocm/src/providers.rs +++ b/apps/rocm/src/providers.rs @@ -541,6 +541,16 @@ fn ready_local_services(paths: &AppPaths) -> Result> { if record.refresh_from_engine_state().unwrap_or(false) { let _ = record.write(); } + // Chat never uses another kind of server, so it is never probed here: + // a speech probe can take far longer than a chat one, and every chat + // command would wait on it. It is still returned by its recorded + // status, so naming it as the chat model is refused with the reason. + if !record.task.is_chat() { + if matches!(record.status.as_str(), "ready" | "running") { + services.push(record); + } + continue; + } // Probe with the service's key so a protected public service is not // filtered out here (an anonymous /v1/models would 401) before // `LocalProvider` can send the bearer token. @@ -2265,6 +2275,50 @@ mod tests { Ok(()) } + #[test] + fn chat_never_contacts_a_text_to_speech_service_yet_refuses_it_by_name() -> Result<()> { + let (root, paths) = temp_app_paths("local-provider-tts-service"); + paths.ensure()?; + let server = spawn_models_only_server("OpenMOSS-TTS")?; + + let mut tts = ManagedServiceRecord::new( + &paths, + "svc-tts", + "lemonade", + "tts", + "OpenMOSS-TTS", + "127.0.0.1", + server.port(), + "managed", + 123, + None, + None, + None, + ); + tts.task = rocm_core::ModelTask::Speech; + tts.status = "ready".to_owned(); + tts.write()?; + + let error = select_local_chat_service(&paths, Some("OpenMOSS-TTS")) + .expect_err("a text-to-speech service is not a chat target") + .to_string(); + let status = LocalProvider { paths: &paths }.status()?; + let contacted = server.stop()?; + let _ = std::fs::remove_dir_all(root); + + assert!( + error.contains("is served by `svc-tts` as a text-to-speech model, not a chat model"), + "{error}" + ); + assert!(status.models.is_empty(), "{:?}", status.models); + // A text-to-speech probe can take 30 s; chat must not pay it. + assert_eq!( + contacted, 0, + "chat sent requests to the text-to-speech server" + ); + Ok(()) + } + #[test] fn local_provider_refuses_a_speech_to_text_service_named_as_the_chat_model() -> Result<()> { let (root, paths) = temp_app_paths("local-provider-speech-service"); diff --git a/apps/rocm/src/serve_cmd.rs b/apps/rocm/src/serve_cmd.rs index 962b7c42a9..d16119cec7 100644 --- a/apps/rocm/src/serve_cmd.rs +++ b/apps/rocm/src/serve_cmd.rs @@ -1100,6 +1100,50 @@ mod tests { let _ = std::fs::remove_dir_all(root); } + #[test] + fn a_text_to_speech_model_is_refused_by_vllm_and_its_named_remedy_passes() { + let recipe = resolve_builtin_model_recipe("tts").expect("tts recipe"); + assert_eq!( + recipe.model_task().expect("valid task"), + rocm_core::ModelTask::Speech + ); + let root = std::env::temp_dir().join(format!( + "rocm-serve-tts-task-check-{}", + rocm_core::unix_time_millis() + )); + let paths = AppPaths { + config_dir: root.join("config"), + data_dir: root.join("data"), + cache_dir: root.join("cache"), + }; + + let error = ensure_engine_serves_task( + &paths, + "vllm", + rocm_core::ModelTask::Speech, + "tts", + Some(&recipe), + ) + .expect_err("vLLM advertises chat only") + .to_string(); + assert_eq!( + error, + "`tts` is a text-to-speech model, which the vllm engine cannot serve; \ + serve it with `rocm serve tts --engine lemonade`" + ); + // The named remedy passes this same check; the nightly GPU scenario + // `@id:serve-text-to-speech-model-speaks-on-the-printed-endpoint` runs it. + ensure_engine_serves_task( + &paths, + "lemonade", + rocm_core::ModelTask::Speech, + "tts", + Some(&recipe), + ) + .expect("Lemonade advertises text-to-speech"); + let _ = std::fs::remove_dir_all(root); + } + #[test] fn unsupported_task_message_offers_no_engine_when_the_recipe_names_none_other() { assert_eq!( diff --git a/apps/rocm/src/serve_summary.rs b/apps/rocm/src/serve_summary.rs index 83ca6bf8eb..09b53cd44c 100644 --- a/apps/rocm/src/serve_summary.rs +++ b/apps/rocm/src/serve_summary.rs @@ -127,7 +127,7 @@ pub(crate) fn render_summary(summary: &DeploymentSummary) -> String { ] }; - // Token timings describe text generation. A speech-to-text server has none + // Token timings describe text generation. A speech server has none // to report, and two permanent `n/a` rows would read as a failed // measurement; it names its task instead, since its endpoint is not the one // chat clients expect. diff --git a/apps/rocmd/src/api_docs.rs b/apps/rocmd/src/api_docs.rs index 7757c39a48..721b57e724 100644 --- a/apps/rocmd/src/api_docs.rs +++ b/apps/rocmd/src/api_docs.rs @@ -365,6 +365,15 @@ async fn forward( let content_type = answer.headers().get(header::CONTENT_TYPE).cloned(); let mut response = Response::new(Body::from_stream(answer.bytes_stream())); *response.status_mut() = status; + if let Some(disposition) = content_type + .as_ref() + .and_then(|value| value.to_str().ok()) + .and_then(audio_attachment) + { + response + .headers_mut() + .insert(header::CONTENT_DISPOSITION, disposition); + } if let Some(content_type) = content_type { response .headers_mut() @@ -390,6 +399,24 @@ async fn forward( } } +/// A `Content-Disposition` that makes Swagger UI offer an audio answer as a +/// download. Without it Swagger renders an audio player pointed back at the +/// request URL, which re-sends it as a `GET` the speech route does not answer, +/// so the audio the server made could be neither played nor saved. +fn audio_attachment(content_type: &str) -> Option { + let media_type = content_type.split(';').next()?.trim().to_ascii_lowercase(); + let subtype = media_type.strip_prefix("audio/")?; + let extension = match subtype { + "wav" | "x-wav" | "wave" | "vnd.wave" => "wav", + "mpeg" | "mp3" => "mp3", + "x-flac" | "flac" => "flac", + "ogg" | "opus" | "aac" => subtype, + "l16" | "pcm" => "pcm", + _ => "audio", + }; + HeaderValue::from_str(&format!("attachment; filename=\"speech.{extension}\"")).ok() +} + /// The part of `/services//api/` after `api/`, still percent-encoded. fn raw_forwarded_path(request_path: &str) -> Option<&str> { let mut segments = request_path.splitn(5, '/'); @@ -422,6 +449,8 @@ struct RouteDoc { summary: &'static str, description: &'static str, body: RouteBody, + /// What the server answers with. + answer_type: &'static str, } /// What a route's request carries, as "Try it out" should offer it. @@ -449,6 +478,7 @@ fn routes_for(record: &ManagedServiceRecord) -> Vec { path: health_path, summary: "Check the server is up", description: "Answers 200 once the server accepts requests.", + answer_type: "application/json", body: RouteBody::None, }, RouteDoc { @@ -457,6 +487,7 @@ fn routes_for(record: &ManagedServiceRecord) -> Vec { summary: "List the served models", description: "OpenAI-compatible model list. The `id` of each entry is the \ `model` value requests use.", + answer_type: "application/json", body: RouteBody::None, }, ]; @@ -469,8 +500,18 @@ fn routes_for(record: &ManagedServiceRecord) -> Vec { description: "OpenAI-compatible speech-to-text. Upload an audio file as `file` \ (WAV is accepted by every backend); the answer's `text` is the \ transcript.", + answer_type: "application/json", body: RouteBody::AudioUpload, }), + ModelTask::Speech => routes.push(RouteDoc { + method: HttpMethod::Post, + path: "/v1/audio/speech", + summary: "Speak text", + description: "OpenAI-compatible text-to-speech. The answer is the spoken `input` \ + as a WAV file, which the page offers to download.", + answer_type: "audio/wav", + body: RouteBody::Json(speech_example), + }), // A task this build does not know: only the routes every server has. ModelTask::Unknown => {} } @@ -485,6 +526,7 @@ fn chat_routes() -> [RouteDoc; 2] { summary: "Chat completion", description: "OpenAI-compatible chat completion. Set `stream` to `true` to \ receive server-sent events as tokens are generated.", + answer_type: "application/json", body: RouteBody::Json(chat_completion_example), }, RouteDoc { @@ -492,6 +534,7 @@ fn chat_routes() -> [RouteDoc; 2] { path: "/v1/completions", summary: "Text completion", description: "OpenAI-compatible text completion of a raw prompt.", + answer_type: "application/json", body: RouteBody::Json(completion_example), }, ] @@ -518,6 +561,14 @@ fn audio_upload_schema(model: &str) -> ObjectBuilder { .required("model") } +fn speech_example(model: &str) -> Value { + json!({ + "model": model, + "input": "Hello from ROCm.", + "response_format": "wav", + }) +} + fn chat_completion_example(model: &str) -> Value { json!({ "model": model, @@ -564,7 +615,7 @@ fn service_openapi(record: &ManagedServiceRecord) -> OpenApi { "200", ResponseBuilder::new() .description("The server's answer") - .content("application/json", ContentBuilder::new().build()), + .content(route.answer_type, ContentBuilder::new().build()), ); let request_body = match route.body { RouteBody::None => None, @@ -742,6 +793,7 @@ mod tests { for task in [ ModelTask::Chat, ModelTask::Transcription, + ModelTask::Speech, ModelTask::Unknown, ] { let mut server = record(&paths, "svc-1", engine, 9); @@ -760,6 +812,28 @@ mod tests { fs::remove_dir_all(root).ok(); } + #[test] + fn audio_answers_are_offered_as_downloads() { + assert_eq!( + audio_attachment("audio/wav").unwrap(), + "attachment; filename=\"speech.wav\"" + ); + assert_eq!( + audio_attachment("audio/mpeg; charset=binary").unwrap(), + "attachment; filename=\"speech.mp3\"" + ); + assert_eq!( + audio_attachment("Audio/WAV").unwrap(), + "attachment; filename=\"speech.wav\"" + ); + assert_eq!( + audio_attachment("audio/L16; rate=24000").unwrap(), + "attachment; filename=\"speech.pcm\"" + ); + assert!(audio_attachment("application/json").is_none()); + assert!(audio_attachment("text/plain").is_none()); + } + #[test] fn forwarded_paths_keep_their_encoding() { assert_eq!( @@ -870,6 +944,30 @@ mod tests { fs::remove_dir_all(root).ok(); } + #[test] + fn text_to_speech_docs_offer_the_speech_route_answering_with_audio() { + let (root, paths) = temp_app_paths("api-docs-speech"); + let mut tts = record(&paths, "svc-tts", "lemonade", 9); + tts.task = ModelTask::Speech; + tts.canonical_model_id = "OpenMOSS-TTS".to_owned(); + let openapi = serde_json::to_value(service_openapi(&tts)).unwrap(); + let mut listed = operation_paths(&openapi); + listed.sort(); + assert_eq!(listed, ["/v1/audio/speech", "/v1/health", "/v1/models"]); + let speech = &openapi["paths"]["/v1/audio/speech"]["post"]; + assert_eq!( + speech["requestBody"]["content"]["application/json"]["example"]["model"], + "OpenMOSS-TTS" + ); + assert!( + speech["responses"]["200"]["content"] + .get("audio/wav") + .is_some(), + "the answer is documented as audio: {speech:#}" + ); + fs::remove_dir_all(root).ok(); + } + #[test] fn keyed_server_docs_ask_for_the_endpoint_key() { let (root, paths) = temp_app_paths("api-docs-keyed"); @@ -935,24 +1033,32 @@ mod tests { ); } + /// What the stand-in text-to-speech route answers: binary, not UTF-8. + const SPOKEN_WAV: &[u8] = b"RIFF\x24\x00\x00\x00WAVEfmt \xff\x00\x80"; + /// A stand-in model server that echoes what reached it, so the test can /// see exactly what the forwarder passed on. async fn start_echo_model_server() -> u16 { let listener = TcpListener::bind((DEFAULT_LOCAL_HOST, 0)).await.unwrap(); let port = listener.local_addr().unwrap().port(); - let app = Router::new().route( - "/v1/chat/completions", - post(|headers: HeaderMap, uri: Uri, body: Bytes| async move { - Json(json!({ - "query": uri.query(), - "authorization": headers - .get(header::AUTHORIZATION) - .and_then(|value| value.to_str().ok()), - "cookie": headers.get(header::COOKIE).and_then(|value| value.to_str().ok()), - "body": String::from_utf8_lossy(&body), - })) - }), - ); + let app = Router::new() + .route( + "/v1/chat/completions", + post(|headers: HeaderMap, uri: Uri, body: Bytes| async move { + Json(json!({ + "query": uri.query(), + "authorization": headers + .get(header::AUTHORIZATION) + .and_then(|value| value.to_str().ok()), + "cookie": headers.get(header::COOKIE).and_then(|value| value.to_str().ok()), + "body": String::from_utf8_lossy(&body), + })) + }), + ) + .route( + "/v1/audio/speech", + post(|| async { ([(header::CONTENT_TYPE, "audio/wav")], SPOKEN_WAV) }), + ); tokio::spawn(async move { axum::serve(listener, app).await.ok() }); port } @@ -1151,6 +1257,39 @@ mod tests { Ok(()) } + #[tokio::test(flavor = "multi_thread")] + async fn spoken_audio_comes_through_the_docs_intact_and_as_a_download() -> Result<()> { + let (root, paths) = temp_app_paths("api-docs-speech-forward"); + paths.ensure()?; + fs::create_dir_all(paths.services_dir())?; + let model_port = start_echo_model_server().await; + let mut tts = record(&paths, "svc-tts", "lemonade", model_port); + tts.task = ModelTask::Speech; + tts.write()?; + let server = start_api_docs_server(paths.clone(), 0)?; + + let answer = test_client() + .post(format!( + "{}/services/svc-tts/api/v1/audio/speech", + server.base_url + )) + .header(header::CONTENT_TYPE, "application/json") + .body(r#"{"model":"Qwen/Qwen3.5","input":"hi"}"#) + .send() + .await?; + assert_eq!(answer.status(), StatusCode::OK); + assert_eq!(answer.headers()[header::CONTENT_TYPE], "audio/wav"); + assert_eq!( + answer.headers()[header::CONTENT_DISPOSITION], + "attachment; filename=\"speech.wav\"" + ); + assert_eq!(answer.bytes().await?.as_ref(), SPOKEN_WAV); + + server.stop(); + fs::remove_dir_all(root).ok(); + Ok(()) + } + #[tokio::test(flavor = "multi_thread")] async fn forwarding_to_a_dead_server_reports_a_bad_gateway() -> Result<()> { let (root, paths) = temp_app_paths("api-docs-dead"); diff --git a/apps/rocmd/src/service.rs b/apps/rocmd/src/service.rs index eec7793f57..508c6b295a 100644 --- a/apps/rocmd/src/service.rs +++ b/apps/rocmd/src/service.rs @@ -918,7 +918,7 @@ fn engine_serve_http_args( )); args.extend(["--state-path".to_owned(), state_path.display().to_string()]); // Omitted for chat so a chat restart keeps the argv it always had; a - // speech-to-text service must come back as one. + // speech-to-text or text-to-speech service must come back as one. if !task.is_chat() { args.extend(["--task".to_owned(), task.as_str().to_owned()]); } diff --git a/crates/rocm-core/src/lib.rs b/crates/rocm-core/src/lib.rs index b89e632143..73722950ce 100644 --- a/crates/rocm-core/src/lib.rs +++ b/crates/rocm-core/src/lib.rs @@ -129,15 +129,16 @@ pub use net::{ Backoff, DEFAULT_LOCAL_HOST, DEFAULT_LOCAL_PORT, DOWNLOAD_MAX_ATTEMPTS, DownloadOutcome, DownloadRequest, EndpointReadiness, EndpointReadinessOutcome, HttpResponseParts, INFERENCE_PROBE_ATTEMPTED_STATE_KEY, INFERENCE_PROBE_RETRY_INTERVAL, INFERENCE_PROBE_TIMEOUT, - INFERENCE_VERIFIED_STATE_KEY, MultipartFile, connect_tcp_stream, download_file_streaming, - download_file_streaming_with_progress, download_file_to_path, + INFERENCE_VERIFIED_STATE_KEY, MultipartFile, SPEECH_PROBE_TIMEOUT, connect_tcp_stream, + download_file_streaming, download_file_streaming_with_progress, download_file_to_path, download_file_to_path_with_progress, engine_state_inference_verified, format_host_for_url, format_host_port, format_http_base_url, http_get_text, http_get_text_with_auth, - http_post_json_with_auth, http_post_multipart_with_auth, managed_service_endpoint_model_ready, + http_post_json_with_auth, http_post_multipart_with_auth, inference_probe_retry_interval, + inference_probe_timeout, managed_service_endpoint_model_ready, managed_service_endpoint_readiness, merge_json_state_file, openai_chat_completion_probe, openai_chat_completion_status, openai_inference_probe, openai_inference_status, - openai_models_endpoint_has_model, openai_transcription_status, parse_http_endpoint, - read_http_response_bounded, silent_wav_16khz_mono, write_all_tcp_stream, + openai_models_endpoint_has_model, openai_speech_status, openai_transcription_status, + parse_http_endpoint, read_http_response_bounded, silent_wav_16khz_mono, write_all_tcp_stream, }; pub use paths::{AppPaths, engine_plugin_dirs}; pub use proc_lifecycle::{ diff --git a/crates/rocm-core/src/model_catalog.json b/crates/rocm-core/src/model_catalog.json index 53bc23b8a5..5fad6a500c 100644 --- a/crates/rocm-core/src/model_catalog.json +++ b/crates/rocm-core/src/model_catalog.json @@ -561,6 +561,34 @@ "warnings": [ "speech-to-text; smallest Whisper model, for quick checks rather than accuracy" ] + }, + { + "canonical_model_id": "OpenMOSS-TTS", + "aliases": [ + "tts", + "moss-tts" + ], + "task": "speech", + "source": "recipe_index", + "revision": "main", + "loader": "openmoss", + "trust_remote_code": false, + "dtype": "gguf", + "device_policy": "gpu_required", + "min_gpu_mem_gb": 14, + "recommended_system_ram_gb": 24, + "quantization": "GGUF Q8_0; Lemonade OpenMOSS ROCm or Vulkan backend", + "artifact_hint": "Lemonade model id resolved and downloaded by the Lemonade engine", + "artifacts": [], + "engine_recipes": [], + "manual_alternatives": [], + "chat_template_mode": "none", + "preferred_engines": [ + "lemonade" + ], + "warnings": [ + "text-to-speech; serves POST /v1/audio/speech, not chat; a 12.5 GB download" + ] } ] } diff --git a/crates/rocm-core/src/model_task.rs b/crates/rocm-core/src/model_task.rs index a0f8d4022f..03c8350acf 100644 --- a/crates/rocm-core/src/model_task.rs +++ b/crates/rocm-core/src/model_task.rs @@ -2,12 +2,13 @@ // // SPDX-License-Identifier: MIT -//! What a served model does: answer chat requests or transcribe speech. +//! What a served model does: answer chat requests, transcribe speech, or speak +//! text aloud. //! //! The task decides which OpenAI-compatible route a server exposes, how its //! readiness is probed, and which commands may use it — `rocm chat` and the -//! dashboard chat pane need a chat model, and a speech-to-text server cannot -//! stand in for one. +//! dashboard chat pane need a chat model, and a speech-to-text or text-to-speech +//! server cannot stand in for one. //! //! Records written before the task existed carry none, and every model served //! then was a chat model, so a missing task deserializes as [`ModelTask::Chat`]. @@ -23,6 +24,8 @@ pub enum ModelTask { Chat, /// Speech-to-text: `POST /v1/audio/transcriptions`. Transcription, + /// Text-to-speech: `POST /v1/audio/speech`. + Speech, /// A task written by a newer rocm-cli or engine plugin that this build does /// not know. Kept so that one unfamiliar value degrades only the record or /// capability it sits in, instead of failing the whole message: it is not @@ -38,6 +41,7 @@ impl ModelTask { match self { Self::Chat => "chat", Self::Transcription => "transcription", + Self::Speech => "speech", Self::Unknown => "unknown", } } @@ -48,6 +52,7 @@ impl ModelTask { match self { Self::Chat => "chat", Self::Transcription => "speech-to-text", + Self::Speech => "text-to-speech", Self::Unknown => "unrecognised", } } @@ -59,6 +64,7 @@ impl ModelTask { match self { Self::Chat => "/chat/completions", Self::Transcription => "/audio/transcriptions", + Self::Speech => "/audio/speech", Self::Unknown => "", } } @@ -90,7 +96,10 @@ impl ModelTask { match value.trim().to_ascii_lowercase().as_str() { "chat" => Ok(Self::Chat), "transcription" => Ok(Self::Transcription), - other => bail!("unsupported model task `{other}`; expected `chat` or `transcription`"), + "speech" => Ok(Self::Speech), + other => bail!( + "unsupported model task `{other}`; expected `chat`, `transcription`, or `speech`" + ), } } } @@ -118,7 +127,7 @@ mod tests { #[test] fn task_round_trips_through_its_wire_name() { - for task in [ModelTask::Chat, ModelTask::Transcription] { + for task in [ModelTask::Chat, ModelTask::Transcription, ModelTask::Speech] { let json = serde_json::to_string(&task).expect("serialize"); assert_eq!(json, format!("\"{}\"", task.as_str())); assert_eq!(ModelTask::parse(task.as_str()).expect("parse"), task); @@ -154,5 +163,6 @@ mod tests { ModelTask::Transcription.api_route(), "/audio/transcriptions" ); + assert_eq!(ModelTask::Speech.api_route(), "/audio/speech"); } } diff --git a/crates/rocm-core/src/net.rs b/crates/rocm-core/src/net.rs index fce0b9069c..1831eb7448 100644 --- a/crates/rocm-core/src/net.rs +++ b/crates/rocm-core/src/net.rs @@ -677,11 +677,13 @@ pub const INFERENCE_PROBE_ATTEMPTED_STATE_KEY: &str = "inference_probe_attempted /// Minimum gap between inference probes against a service that is still loading. /// /// Only a *successful* probe latches, so without this a warming model would be -/// re-probed by every readiness poll — and each attempt costs up to -/// [`INFERENCE_PROBE_TIMEOUT`], which is the whole poll's latency. The +/// re-probed by every readiness poll — and each attempt costs up to its probe +/// budget ([`inference_probe_timeout`]), which is the whole poll's latency. The /// supervisor ticks every few seconds and `services list` sits in front of a /// user, so the unthrottled cost lands exactly where it is most visible. The -/// price of throttling is that readiness can be noticed up to this late. +/// price of throttling is that readiness can be noticed up to this late. This is +/// the floor; [`inference_probe_retry_interval`] lengthens it for a task whose +/// probe takes longer. pub const INFERENCE_PROBE_RETRY_INTERVAL: Duration = Duration::from_secs(15); /// Merge `patch`'s top-level keys into the JSON object stored at `path`. @@ -716,7 +718,7 @@ pub fn merge_json_state_file(path: &Path, patch: &serde_json::Value) -> Result<( /// Whether inference has been confirmed for a service, from its engine state /// file — probing at most once, and at most once per -/// [`INFERENCE_PROBE_RETRY_INTERVAL`] while it is still loading. +/// [`inference_probe_retry_interval`] while it is still loading. /// /// Shared by the engine adapters so the latch and backoff bookkeeping has one /// implementation: the engines differ in how they decide a model is *listed*, @@ -746,7 +748,8 @@ pub fn engine_state_inference_verified( } let now = unix_time_millis() as u64; if let Some(attempted_at) = state_u64(INFERENCE_PROBE_ATTEMPTED_STATE_KEY) - && now.saturating_sub(attempted_at) < INFERENCE_PROBE_RETRY_INTERVAL.as_millis() as u64 + && u128::from(now.saturating_sub(attempted_at)) + < inference_probe_retry_interval(task).as_millis() { return false; } @@ -759,7 +762,7 @@ pub fn engine_state_inference_verified( endpoint_url, model_ref, endpoint_api_key, - INFERENCE_PROBE_TIMEOUT, + inference_probe_timeout(task), ) .unwrap_or(false) { @@ -870,6 +873,58 @@ pub fn openai_transcription_status( Ok(response.status) } +/// The budget for a text-to-speech probe. +/// +/// Synthesizing even one word runs the whole model and its audio codec, which +/// on a large text-to-speech model takes far longer than the token a chat probe +/// asks for; a chat-sized budget would keep a working server from ever being +/// called ready. [`inference_probe_retry_interval`] spaces failed probes so +/// that this is not paid on every poll. +pub const SPEECH_PROBE_TIMEOUT: Duration = Duration::from_secs(30); + +/// The budget for one inference probe of a server for `task`. +#[must_use] +pub const fn inference_probe_timeout(task: crate::ModelTask) -> Duration { + match task { + crate::ModelTask::Speech => SPEECH_PROBE_TIMEOUT, + _ => INFERENCE_PROBE_TIMEOUT, + } +} + +/// The least time between two inference probes of a server for `task`. +/// +/// That is [`INFERENCE_PROBE_RETRY_INTERVAL`], or twice the probe's own budget +/// when that is longer, for a server still coming up. A probe that used its +/// whole budget must not be due again by the time it returns, or every +/// readiness poll would pay it. +#[must_use] +pub fn inference_probe_retry_interval(task: crate::ModelTask) -> Duration { + INFERENCE_PROBE_RETRY_INTERVAL.max(inference_probe_timeout(task) * 2) +} + +/// Ask a text-to-speech endpoint to say one word and return the HTTP status — +/// the text-to-speech counterpart of [`openai_chat_completion_status`]. +pub fn openai_speech_status( + endpoint_url: &str, + model_ref: &str, + endpoint_api_key: Option<&str>, + timeout: Duration, +) -> Result { + let body = serde_json::json!({ + "model": model_ref, + "input": "ok", + "response_format": "wav", + }); + let response = http_post_json_with_auth( + endpoint_url, + "/v1/audio/speech", + &body, + endpoint_api_key, + timeout, + )?; + Ok(response.status) +} + /// The smallest real request for a model's task, as an HTTP status. pub fn openai_inference_status( task: crate::ModelTask, @@ -885,6 +940,9 @@ pub fn openai_inference_status( crate::ModelTask::Transcription => { openai_transcription_status(endpoint_url, model_ref, endpoint_api_key, timeout) } + crate::ModelTask::Speech => { + openai_speech_status(endpoint_url, model_ref, endpoint_api_key, timeout) + } // Readiness is never claimed for a service this build cannot probe. crate::ModelTask::Unknown => bail!("no inference probe exists for an unrecognised task"), } @@ -893,9 +951,10 @@ pub fn openai_inference_status( /// Whether the endpoint answers the request that matches the model's task. /// /// A chat server counts as serving on the same "answered below 500" rule as -/// [`openai_chat_completion_probe`]. A speech-to-text server must answer 200: -/// the probe sends a valid recording, so a 4xx means the route or the model is -/// not there (a model not loaded, a decode failure), not that it is serving. +/// [`openai_chat_completion_probe`]. A speech-to-text or text-to-speech server +/// must answer 200: the probe sends a valid recording or sentence, so a 4xx +/// means the route or the model is not there (a model not loaded, a decode +/// failure), not that it is serving. /// Probing a speech-to-text server with a chat request would instead read its /// `404` as serving. pub fn openai_inference_probe( @@ -907,7 +966,7 @@ pub fn openai_inference_probe( ) -> Result { let status = openai_inference_status(task, endpoint_url, model_ref, endpoint_api_key, timeout)?; Ok(match task { - crate::ModelTask::Transcription => status == 200, + crate::ModelTask::Transcription | crate::ModelTask::Speech => status == 200, _ => status < 500, }) } @@ -1019,7 +1078,7 @@ pub struct EndpointReadinessOutcome { /// as before this check existed. /// /// A *failed* probe cannot latch, so those are throttled instead: a still-loading -/// service is re-probed at most once per [`INFERENCE_PROBE_RETRY_INTERVAL`], +/// service is re-probed at most once per [`inference_probe_retry_interval`], /// which keeps a warming model from costing every caller a full /// `probe_timeout`. /// @@ -1046,7 +1105,8 @@ pub fn managed_service_endpoint_readiness( } let now = unix_time_millis() as u64; if let Some(attempted_at) = record.inference_probe_attempted_at_unix_ms - && now.saturating_sub(attempted_at) < INFERENCE_PROBE_RETRY_INTERVAL.as_millis() as u64 + && u128::from(now.saturating_sub(attempted_at)) + < inference_probe_retry_interval(record.task).as_millis() { return outcome(EndpointReadiness::Listing, false); } @@ -2488,6 +2548,34 @@ mod tests { Ok(()) } #[test] + fn a_text_to_speech_service_is_probed_with_a_speech_request() -> Result<()> { + let (port, server) = spawn_canned_http_server(vec![ + Some(("HTTP/1.1 200 OK", MODELS_OK_BODY)), + Some(("HTTP/1.1 200 OK", "RIFF")), + ])?; + let mut record = probe_test_record(port); + record.task = crate::ModelTask::Speech; + + assert_eq!( + managed_service_endpoint_readiness( + &mut record, + None, + Duration::from_secs(2), + Duration::from_secs(2) + ) + .readiness, + EndpointReadiness::Serving + ); + let requests = server.join().expect("server thread should not panic")?; + let probe = requests.get(1).expect("an inference probe was sent"); + assert!( + probe.starts_with("POST /v1/audio/speech HTTP/1.1"), + "{probe}" + ); + assert!(probe.contains("\"input\":\"ok\""), "{probe}"); + Ok(()) + } + #[test] fn silent_wav_is_a_well_formed_16khz_mono_pcm_file() { let wav = silent_wav_16khz_mono(Duration::from_secs(1)); assert_eq!(wav.len(), 44 + 32_000); @@ -2593,30 +2681,137 @@ mod tests { Ok(()) } #[test] - fn a_speech_to_text_probe_needs_a_transcript_not_just_an_answer() -> Result<()> { - for (status_line, serving) in [ - ("HTTP/1.1 200 OK", true), - ("HTTP/1.1 404 Not Found", false), - ("HTTP/1.1 400 Bad Request", false), + fn a_speech_probe_needs_a_200_not_just_an_answer() -> Result<()> { + for task in [crate::ModelTask::Transcription, crate::ModelTask::Speech] { + for (status_line, serving) in [ + ("HTTP/1.1 200 OK", true), + ("HTTP/1.1 404 Not Found", false), + ("HTTP/1.1 400 Bad Request", false), + ] { + let (port, server) = + spawn_canned_http_server(vec![Some((status_line, r#"{"text":""}"#))])?; + assert_eq!( + openai_inference_probe( + task, + &format!("http://127.0.0.1:{port}/v1"), + "model", + None, + Duration::from_secs(2), + )?, + serving, + "{task} {status_line}" + ); + server.join().expect("server thread should not panic")?; + } + } + Ok(()) + } + #[test] + fn a_failed_speech_probe_is_retried_later_than_a_failed_chat_probe() -> Result<()> { + // 20 s after a failed attempt: past the chat retry interval, inside the + // text-to-speech one, which must be at least twice its 30 s probe. + let attempted = (unix_time_millis() as u64).saturating_sub(20_000); + for (task, reprobed) in [ + (crate::ModelTask::Chat, true), + (crate::ModelTask::Speech, false), + ] { + // A second answer is always on offer, so a probe that should not + // happen would be answered, and seen, rather than refused. + let (port, server) = spawn_canned_http_server(vec![ + Some(("HTTP/1.1 200 OK", MODELS_OK_BODY)), + Some(("HTTP/1.1 200 OK", CHAT_OK_BODY)), + ])?; + let mut record = probe_test_record(port); + record.task = task; + record.inference_probe_attempted_at_unix_ms = Some(attempted); + let readiness = managed_service_endpoint_readiness( + &mut record, + None, + Duration::from_secs(2), + Duration::from_secs(2), + ) + .readiness; + if !reprobed { + // Take the second answer so the stand-in server can finish. + if let Ok(mut sentinel) = std::net::TcpStream::connect(("127.0.0.1", port)) { + let _ = sentinel.write_all(b"GET /sentinel HTTP/1.1\r\n\r\n"); + // Read the answer before closing, so the stand-in server's + // write never meets a closed socket. + let _ = sentinel.set_read_timeout(Some(Duration::from_secs(2))); + let _ = sentinel.read_to_end(&mut Vec::new()); + } + } + let requests = server.join().expect("server thread should not panic")?; + if reprobed { + assert_eq!(readiness, EndpointReadiness::Serving, "{task}"); + } else { + assert_eq!(readiness, EndpointReadiness::Listing, "{task}"); + assert!( + requests.iter().all(|request| !request.starts_with("POST ")), + "a text-to-speech probe is not due again yet: {requests:?}" + ); + } + } + Ok(()) + } + #[test] + fn an_engine_does_not_reprobe_a_speech_server_inside_its_retry_interval() -> Result<()> { + let state_path = std::env::temp_dir().join(format!( + "rocm-speech-throttle-{}-{}.json", + std::process::id(), + unix_time_millis() + )); + let attempted = (unix_time_millis() as u64).saturating_sub(20_000); + let state = serde_json::json!({ INFERENCE_PROBE_ATTEMPTED_STATE_KEY: attempted }); + for (task, reprobed) in [ + (crate::ModelTask::Chat, true), + (crate::ModelTask::Speech, false), ] { let (port, server) = - spawn_canned_http_server(vec![Some((status_line, r#"{"text":""}"#))])?; + spawn_canned_http_server(vec![Some(("HTTP/1.1 200 OK", CHAT_OK_BODY))])?; + let verified = engine_state_inference_verified( + task, + &state_path, + Some(&state), + &format!("http://127.0.0.1:{port}"), + "model", + None, + ); + if !reprobed && let Ok(mut sentinel) = std::net::TcpStream::connect(("127.0.0.1", port)) + { + let _ = sentinel.write_all(b"GET /sentinel HTTP/1.1\r\n\r\n"); + // Read the answer before closing, so the stand-in server's + // write never meets a closed socket. + let _ = sentinel.set_read_timeout(Some(Duration::from_secs(2))); + let _ = sentinel.read_to_end(&mut Vec::new()); + } + let requests = server.join().expect("server thread should not panic")?; + assert_eq!(verified, reprobed, "{task}"); assert_eq!( - openai_inference_probe( - crate::ModelTask::Transcription, - &format!("http://127.0.0.1:{port}/v1"), - "Whisper-Tiny", - None, - Duration::from_secs(2), - )?, - serving, - "{status_line}" + requests.iter().any(|request| request.starts_with("POST ")), + reprobed, + "{task}: {requests:?}" ); - server.join().expect("server thread should not panic")?; } + let _ = fs::remove_file(&state_path); Ok(()) } #[test] + fn a_probe_that_used_its_whole_budget_is_not_due_again_when_it_returns() { + for task in [ + crate::ModelTask::Chat, + crate::ModelTask::Transcription, + crate::ModelTask::Speech, + ] { + assert!( + inference_probe_retry_interval(task) >= inference_probe_timeout(task) * 2, + "{task}" + ); + assert!(inference_probe_retry_interval(task) >= INFERENCE_PROBE_RETRY_INTERVAL); + } + assert!(inference_probe_timeout(crate::ModelTask::Speech) > INFERENCE_PROBE_TIMEOUT); + } + #[test] fn inference_probe_sends_the_service_key_to_a_protected_endpoint() -> Result<()> { let (port, server) = spawn_canned_http_server(vec![ Some(("HTTP/1.1 200 OK", MODELS_OK_BODY)), diff --git a/crates/rocm-core/src/service_record.rs b/crates/rocm-core/src/service_record.rs index 93f2966933..d2a2e6b19b 100644 --- a/crates/rocm-core/src/service_record.rs +++ b/crates/rocm-core/src/service_record.rs @@ -89,7 +89,7 @@ pub struct ManagedServiceRecord { pub stop_requested_unix_ms: Option, /// When the last inference probe was attempted. Throttles re-probing of a /// service that is listed but still loading — see - /// [`crate::INFERENCE_PROBE_RETRY_INTERVAL`]. Absent on records written before + /// [`crate::inference_probe_retry_interval`]. Absent on records written before /// readiness was gated on inference. #[serde(default)] pub inference_probe_attempted_at_unix_ms: Option, diff --git a/crates/rocm-engine-protocol/src/lib.rs b/crates/rocm-engine-protocol/src/lib.rs index f18b7e2e5c..bed27d9d47 100644 --- a/crates/rocm-engine-protocol/src/lib.rs +++ b/crates/rocm-engine-protocol/src/lib.rs @@ -640,6 +640,9 @@ pub const OPENAI_CHAT_ROUTES: [&str; 3] = ["/v1/models", "/v1/chat/completions", /// The OpenAI-compatible routes every speech-to-text server answers. pub const OPENAI_TRANSCRIPTION_ROUTES: [&str; 2] = ["/v1/models", "/v1/audio/transcriptions"]; +/// The OpenAI-compatible routes every text-to-speech server answers. +pub const OPENAI_SPEECH_ROUTES: [&str; 2] = ["/v1/models", "/v1/audio/speech"]; + /// The OpenAI-compatible routes a server for `task` answers, besides its /// health route. #[must_use] @@ -647,6 +650,7 @@ pub const fn task_routes(task: ModelTask) -> &'static [&'static str] { match task { ModelTask::Chat => &OPENAI_CHAT_ROUTES, ModelTask::Transcription => &OPENAI_TRANSCRIPTION_ROUTES, + ModelTask::Speech => &OPENAI_SPEECH_ROUTES, // A task this build does not know: only the route every server has. ModelTask::Unknown => &["/v1/models"], } @@ -1140,6 +1144,10 @@ mod tests { supported_routes("/v1/health", ModelTask::Transcription), ["/v1/health", "/v1/models", "/v1/audio/transcriptions"] ); + assert_eq!( + supported_routes("/v1/health", ModelTask::Speech), + ["/v1/health", "/v1/models", "/v1/audio/speech"] + ); // A task this build cannot name: only the routes every server has. assert_eq!( supported_routes("/v1/health", ModelTask::Unknown), diff --git a/docs/architecture.md b/docs/architecture.md index 84f931b220..9e41973cbf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -86,7 +86,7 @@ Already-extracted subsystem modules, following the same full-domain-extraction p - [`managed_runtime.rs`](../crates/rocm-core/src/managed_runtime.rs) — (ROCMAI-85, Phase 7a) the TheRock managed-runtime-environment surface — see the module's own doc comment for what it owns. - [`model_catalog.rs`](../crates/rocm-core/src/model_catalog.rs) — (ROCMAI-152, Phase 7c) the curated model catalog's types, built-in catalog JSON loading, signed external recipe indexes, and artifact cache-marker bookkeeping — see the module's own doc comment for what it owns. - [`model_readiness.rs`](../crates/rocm-core/src/model_readiness.rs) — the `rocm diagnose --model`/`rocm model --verbose` fit assessment: curated-recipe lookup against a host's measured GPU/RAM, shared between the two commands so they cannot disagree about whether a model fits. -- [`model_task.rs`](../crates/rocm-core/src/model_task.rs) — `ModelTask`, what a served model does (chat or speech-to-text): its client route, how readiness probes it, and which commands may use it. +- [`model_task.rs`](../crates/rocm-core/src/model_task.rs) — `ModelTask`, what a served model does (chat, speech-to-text or text-to-speech): its client route, how readiness probes it, and which commands may use it. - [`net.rs`](../crates/rocm-core/src/net.rs) — (ROCMAI-154, Phase 7b) networking/download/endpoint-readiness primitives. - [`paths.rs`](../crates/rocm-core/src/paths.rs) — (ROCMAI-154, Phase 7b) `AppPaths` and its filesystem-location primitives. - [`process.rs`](../crates/rocm-core/src/process.rs) — (ROCMAI-154, Phase 7b) process-spawn/terminate primitives, including `FileLock`, plus `run_with_timeout`: it drains both pipes on their own threads from the moment the child starts, so output size cannot stall a command into its own timeout, and it bounds how much of a timed-out child's output the error quotes, because that text reaches persisted automation logs. It is not yet the *only* such helper — [`therock.rs`](../apps/rocm/src/therock.rs)'s `wait_with_output_bounded`, [`examine.rs`](../crates/rocm-core/src/examine.rs)'s `run_raw`, and [`host_gpu.rs`](../crates/rocm-core/src/host_gpu.rs)'s `capture_optional_command_candidate_with_timeout` (plus its Windows-only `capture_diagnostic_command`) still join their reader threads with no time limit, so a grandchild holding the pipe can outlast their timeout; converging them onto `run_with_timeout` is follow-up work to ROCMAI-448 tracked in [#623](https://github.com/ROCm/rocm-cli/issues/623). @@ -159,7 +159,7 @@ These modules are private `mod` declarations — [`lib.rs`](../crates/e2e-report - [`install.rs`](../engines/lemonade/src/install.rs) — embeddable-runtime archive download/verify/extract, cache locking, and manifest persistence. - [`process.rs`](../engines/lemonade/src/process.rs) — `lemond`/llama-server spawn, env/path setup, and readiness/health polling. - [`runtime_dir.rs`](../engines/lemonade/src/runtime_dir.rs) — `XDG_RUNTIME_DIR` tier1/2/3 hardening for the spawned child. -- [`speech.rs`](../engines/lemonade/src/speech.rs) — speech-to-text serving through the `whispercpp` recipe: GPU backend selection, load, and the post-load transcription check. +- [`speech.rs`](../engines/lemonade/src/speech.rs) — speech model serving: speech-to-text through the `whispercpp` recipe and text-to-speech through `openmoss`, with GPU backend selection, load, and the post-load request check. - [`state.rs`](../engines/lemonade/src/state.rs) — service state file I/O, device-policy/GPU-selection parsing, and port/endpoint helpers. [`lib.rs`](../engines/lemonade/src/lib.rs) keeps only CLI/envelope dispatch and the service-lifecycle orchestration that calls across those modules. diff --git a/docs/engine-plugins.md b/docs/engine-plugins.md index 5310b1ddfb..2d2b10ed63 100644 --- a/docs/engine-plugins.md +++ b/docs/engine-plugins.md @@ -35,8 +35,9 @@ unsupported. rocm-cli does not use a CPU fallback for this path. ## Model tasks An engine advertises which kinds of model it can serve in the `tasks` field of -its capabilities: `chat` (`POST /v1/chat/completions`) and `transcription` -(speech-to-text, `POST /v1/audio/transcriptions`). An engine that does not +its capabilities: `chat` (`POST /v1/chat/completions`), `transcription` +(speech-to-text, `POST /v1/audio/transcriptions`), and `speech` (text-to-speech, +`POST /v1/audio/speech`). An engine that does not send the field — an external plugin written before it existed — is treated as serving `chat` only. `rocm serve` reads each model's task from its catalog recipe (a model outside the catalog is a chat model) and refuses, before any @@ -45,22 +46,25 @@ advertise its task. The task is passed to the engine as `task` in the `ResolveModel` and `Launch` requests, omitted for chat so a request for a chat model is unchanged. -`lemonade` advertises `chat` and `transcription`; `vllm` advertises `chat`. +`lemonade` advertises `chat`, `transcription`, and `speech`; `vllm` advertises +`chat`. Lemonade serves a speech-to-text model through its router with the -`whispercpp` recipe. The adapter names the backend explicitly — `rocm`, or -`vulkan` when Lemonade reports ROCm unsupported, never `cpu` — because -Lemonade's `auto` picks whichever whisper.cpp build is installed, which can be -the CPU one. Before reporting ready it checks that Lemonade loaded the model on -that backend and that a test transcription returns HTTP 200. It does not trust +`whispercpp` recipe, and a text-to-speech model with the `openmoss` recipe. The +adapter names the backend explicitly — `rocm`, or `vulkan` when Lemonade +reports ROCm unsupported, never `cpu` — because Lemonade's `auto` picks +whichever whisper.cpp build is installed, which can be the CPU one (OpenMOSS +has no CPU build). Before reporting ready it checks that Lemonade loaded the +model on that backend and that a test request for the task (a transcription, +or one spoken word) returns HTTP 200. It does not trust the per-model `device` field of Lemonade's health report: Lemonade 11.5.1 derives it from the backend name, labelling a ROCm Whisper model `cpu`, and a Vulkan one `gpu` even where whisper.cpp found no Vulkan device. `rocm serve`'s usable-AMD-GPU check, which runs before the engine starts, ensures an AMD GPU -is present, but not that a Vulkan whisper.cpp build can use it: where the -Vulkan driver exposes no device, that build may run on the CPU without -reporting it. This is a known gap, shared with the chat path's Vulkan -llama.cpp; the ROCm backend is not affected. +is present, but not that a Vulkan build of whisper.cpp or OpenMOSS can use it: +where the Vulkan driver exposes no device, that build may run on the CPU +without reporting it. This is a known gap, shared with the chat path's Vulkan +llama.cpp; the ROCm backends are not affected. ## Lemonade backend alignment on engine install diff --git a/docs/manual-testing.md b/docs/manual-testing.md index 346ae57e64..17bd69545a 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -309,6 +309,16 @@ Expected result: - `rocm serve whisper-tiny --engine vllm` is refused before any GPU check and names `rocm serve whisper-tiny --engine lemonade`. +Text-to-speech, on a GPU with about 14 GiB free (a 12.5 GB download): + +```powershell +rocm serve tts +curl.exe http://127.0.0.1:11435/v1/audio/speech -H "Content-Type: application/json" -d "{\"model\":\"OpenMOSS-TTS\",\"input\":\"Hello from ROCm.\",\"response_format\":\"wav\"}" -o hello.wav +``` + +Expected result: `hello.wav` plays the sentence, the summary names the +`text-to-speech` task, and `rocm services` lists a `text-to-speech endpoint`. + Confirm the GPU did the work by watching GPU utilization (`amd-smi metric` on Linux, Task Manager's GPU graph on Windows) while transcribing a recording of a minute or more. Do not rely on the `device` field of Lemonade's own @@ -659,6 +669,10 @@ Expected result: - For a speech-to-text server (`rocm serve whisper`), the page lists `POST /v1/audio/transcriptions` and no chat routes; choosing a WAV file under "Try it out" and pressing "Execute" returns its transcript. +- For a text-to-speech server (`rocm serve tts`), the page lists + `POST /v1/audio/speech`; "Execute" with the example body returns audio the + page offers as a download (`speech.wav`), and the downloaded file plays the + sentence. - For a server started with `--require-api-key` (an endpoint key), "Execute" is refused until the key is entered under "Authorize". - After `rocm services stop --yes`, reloading the page says the diff --git a/docs/testing.md b/docs/testing.md index c343d9fc29..b9a08bc139 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -333,8 +333,8 @@ through them is transcribed. The link `rocm serve` itself prints, and a try-it request through it to a real engine, are checked by the GPU serve canaries `serve-05` (vLLM) and `serve-07` -(Lemonade), and for a speech-to-text server by `serve-27`, which only run on -GPU lanes. On Linux Lemonade lanes `serve-07` is +(Lemonade), for a speech-to-text server by `serve-27`, and for a text-to-speech +server by the nightly `serve-29`, all of which only run on GPU lanes. On Linux Lemonade lanes `serve-07` is an expected failure for an unrelated known issue, so there a broken docs link would not turn the lane red: the Lemonade check is effective on the Windows Lemonade lanes, and `serve-05` on the vLLM lanes. @@ -1945,25 +1945,34 @@ The gate itself is held by unit tests in neighbours) and by the Windows manual check in [manual-testing.md](manual-testing.md), in the Ryzen 4000/5000 section. -## Speech-to-text Serving +## Speech Serving -Focused coverage for model tasks and speech-to-text serving: +Focused coverage for model tasks, speech-to-text and text-to-speech serving: ```bash cargo test -p rocm-core model_task cargo test -p rocm-core speech_to_text +cargo test -p rocm-core speech cargo test -p rocm-engine-protocol capabilities_from_an_engine_without_tasks -cargo test -p rocm-engine-lemonade speech:: +cargo test -p rocm-engine-lemonade speech cargo test -p rocm-engine-vllm refuse_a_speech_to_text_model -cargo test -p rocm --bin rocm speech_to_text +cargo test -p rocm --bin rocm speech cargo test -p rocm-dash-tui speech_to_text +cargo test -p rocmd text_to_speech ``` -The user-facing behaviour is in `serve-26` to `serve-28`, `chat-10`, and -`bench-07`. All but `serve-27` run on every lane: `serve-26` needs no server at -all, and the others run against a mock speech-to-text server. `serve-27` serves -Whisper-Tiny on a real GPU and needs the Lemonade GPU lanes -(`@requires-gpu @requires-engine:lemonade`). +`cargo test -p rocm-core speech` includes the probe timing: a text-to-speech +probe's longer budget, and that a failed one is not retried before twice that +budget has passed. + +The user-facing behaviour is in `serve-26` to `serve-28` and `serve-30`, +`chat-10`, `chat-11`, `bench-07`, and the nightly text-to-speech scenario +`serve-29`. All but `serve-27` and `serve-29` run against mock servers (or none, +for `serve-26` and `serve-30`) with no GPU; `chat-11` drives the interactive chat pane and so runs +on the Linux lanes only. `serve-27` serves Whisper-Tiny on a real GPU and needs +the Lemonade GPU lanes (`@requires-gpu @requires-engine:lemonade`); `serve-29` +serves OpenMOSS-TTS on those lanes too, but only in the nightly run +(`E2E_INCLUDE_NIGHTLY=1`), as the model is a 12.5 GB download. Running a subset with `cargo xtask e2e -- -n ` bypasses the capability filter, so a `@requires-gpu` scenario the pattern matches runs, and diff --git a/engines/lemonade/src/lib.rs b/engines/lemonade/src/lib.rs index eb6516eb79..d0e08b2147 100644 --- a/engines/lemonade/src/lib.rs +++ b/engines/lemonade/src/lib.rs @@ -108,7 +108,7 @@ enum CommandKind { engine_recipe_json: Option, #[arg(long)] gpu: Option, - /// What the model does: `chat` (default) or `transcription`. + /// What the model does: `chat` (default), `transcription` or `speech`. #[arg(long, default_value = "chat")] task: String, }, @@ -213,6 +213,16 @@ pub fn run_cli() -> Result<()> { Ok(()) } +/// How `/v1/health` names a loaded speech model's recipe and backend. +/// +/// Returns the recipe name and the `recipe_options` key that reports the +/// backend, for callers that judge a speech service's readiness from that +/// report; `None` for a task that is not served as a speech model. +#[must_use] +pub fn speech_model_health_keys(task: ModelTask) -> Option<(&'static str, &'static str)> { + crate::speech::speech_recipe(task).map(crate::speech::SpeechRecipe::health_keys) +} + pub fn builtin_handle_envelope(envelope: EngineRequestEnvelope) -> EngineResponseEnvelope { handle_envelope(envelope) } @@ -317,7 +327,7 @@ fn capabilities() -> EngineCapabilities { quantized_models: "GGUF through Lemonade llama.cpp (ROCm or Vulkan GPU backend auto-selected)".to_owned(), reasoning_parser: false, - tasks: vec![ModelTask::Chat, ModelTask::Transcription], + tasks: vec![ModelTask::Chat, ModelTask::Transcription, ModelTask::Speech], } } @@ -423,8 +433,9 @@ fn resolve_model_response(request: ResolveModelRequest) -> Result Result<()> { process_env.gpu_indices = request.gpu_indices.clone(); let log_path = request.log_path.as_deref(); crate::state::write_running_state(&request, &runtime, std::process::id(), None, "starting")?; - if request.task == ModelTask::Transcription { - return crate::speech::serve_transcription(&request, &runtime, &process_env, log_path); + if let Some(recipe) = crate::speech::speech_recipe(request.task) { + return crate::speech::serve_speech_model( + recipe, + &request, + &runtime, + &process_env, + log_path, + ); } // A canonical Hugging Face checkpoint (owner/repo:variant) cannot be served under its // exact name through Lemonade's model router (Lemonade renames registered models and @@ -730,18 +747,20 @@ fn healthcheck_service(request: HealthcheckRequest) -> Result { - crate::process::query_loaded_model_endpoint(endpoint, &model_ref, &backend) - } - ModelTask::Transcription => { - crate::speech::query_loaded_transcription_model(endpoint, &model_ref, &backend) - } - // A task this build cannot name was written by a newer one - // (`task_from_state`); it has no probe here, so it is never - // reported ready. - ModelTask::Unknown => Ok(false), - }) + .map( + |endpoint| match (task, crate::speech::speech_recipe(task)) { + (ModelTask::Chat, _) => { + crate::process::query_loaded_model_endpoint(endpoint, &model_ref, &backend) + } + (_, Some(recipe)) => crate::speech::query_loaded_speech_model( + recipe, endpoint, &model_ref, &backend, + ), + // A task this build cannot name was written by a newer one + // (`task_from_state`); it has no probe here, so it is never + // reported ready. + (_, None) => Ok(false), + }, + ) .transpose() .unwrap_or(None) .unwrap_or(false); @@ -852,6 +871,21 @@ fn print_json(value: &T) -> Result<()> { mod tests { use super::*; + #[test] + fn a_text_to_speech_model_resolves_to_the_openmoss_recipe() -> Result<()> { + let resolved = resolve_model_response(ResolveModelRequest { + model_ref: "OpenMOSS-TTS".to_owned(), + runtime_id: None, + device_policy: None, + recipe_override: None, + engine_recipe: None, + task: ModelTask::Speech, + })?; + assert_eq!(resolved.task, "speech"); + assert_eq!(resolved.loader, "openmoss"); + Ok(()) + } + #[test] fn a_task_from_a_newer_rocm_cli_is_not_resolved_as_any_known_task() { let error = resolve_model_response(ResolveModelRequest { diff --git a/engines/lemonade/src/speech.rs b/engines/lemonade/src/speech.rs index 44ac54c6ab..3ea4ec0969 100644 --- a/engines/lemonade/src/speech.rs +++ b/engines/lemonade/src/speech.rs @@ -2,29 +2,30 @@ // // SPDX-License-Identifier: MIT -//! Speech-to-text serving through Lemonade's `whispercpp` recipe. +//! Serving Lemonade's speech models: speech-to-text through its `whispercpp` +//! recipe and text-to-speech through its `openmoss` recipe. //! -//! A transcription model always runs behind Lemonade's router (`lemond`): the -//! direct `llama-server` path the chat flow can take on Linux has no audio -//! route. The backend is chosen the same way as for chat — ROCm first, Vulkan -//! when Lemonade reports ROCm unsupported, never CPU — and is named explicitly -//! on load, because Lemonade's `auto` resolves to whatever whisper.cpp build is -//! installed, which can be the CPU one. +//! These models always run behind Lemonade's router (`lemond`): the direct +//! `llama-server` path the chat flow can take on Linux has no audio routes. The +//! backend is chosen the same way as for chat — ROCm first, Vulkan when Lemonade +//! reports ROCm unsupported, never CPU — and is named explicitly on load, +//! because Lemonade's `auto` resolves to whatever build is installed, which for +//! whisper.cpp can be the CPU one. (OpenMOSS has no CPU build at all.) //! //! What this can and cannot verify: Lemonade 11.5.1 reports a loaded model's //! `device`, but derives it from the backend *name* — a `rocm` Whisper is //! reported as `cpu`, and a `vulkan` one as `gpu` even where whisper.cpp found //! no Vulkan device and ran on the CPU. So the field proves nothing either way //! and is not consulted. What is checked is the backend Lemonade actually loaded -//! (`recipe_options.whispercpp_backend`), exactly as the chat path checks +//! (`recipe_options._backend`), exactly as the chat path checks //! `llamacpp_backend`. `rocm serve`'s usable-AMD-GPU gate, which runs before //! any engine is started, ensures an AMD GPU is present, but not that a Vulkan -//! whisper.cpp build can use it: with no Vulkan device it may run on the CPU -//! unreported. That gap is shared with the chat path's Vulkan llama.cpp and is -//! documented in docs/engine-plugins.md. +//! build of the speech backend can use it: with no Vulkan device it may run on +//! the CPU unreported. That gap is shared with the chat path's Vulkan llama.cpp +//! and is documented in docs/engine-plugins.md. use anyhow::{Context, Result, anyhow, bail}; -use rocm_core::format_http_base_url; +use rocm_core::{ModelTask, format_http_base_url}; use rocm_engine_protocol::{DevicePolicy, EngineRecipeHint, ResolveModelResponse}; use serde_json::{Value, json}; use std::path::Path; @@ -34,60 +35,109 @@ use crate::backend_alignment::LemonadeRuntime; use crate::process::LemonadeProcessEnvironment; use crate::{DEFAULT_HOST, ServeHttpRequest}; -/// Lemonade's speech-to-text recipe. -pub(crate) const WHISPERCPP_RECIPE: &str = "whispercpp"; +/// How Lemonade serves one kind of speech model. +#[derive(Debug)] +pub(crate) struct SpeechRecipe { + pub(crate) task: ModelTask, + /// Lemonade's recipe name, as `lemonade backends` and `/v1/health` report it. + pub(crate) recipe: &'static str, + /// The `lemonade load` flag that names the backend. + load_flag: &'static str, + /// The `recipe_options` key that reports the backend a model was loaded on. + backend_option: &'static str, + /// The weight format, as the catalog records it. + dtype: &'static str, + /// What the user is told about the model's memory needs. + estimated_memory: &'static str, + /// Shown when the backend fails to install twice in a row. The download is + /// retried by every serve that finds the backend missing, so running the + /// same command again is the step that installs it. + install_advice: &'static str, +} + +pub(crate) const TRANSCRIPTION: SpeechRecipe = SpeechRecipe { + task: ModelTask::Transcription, + recipe: "whispercpp", + load_flag: "--whispercpp", + backend_option: "whispercpp_backend", + dtype: "ggml", + estimated_memory: "under 2 GiB for Whisper Large-v3-Turbo; well under 1 GiB for the \ + smaller Whisper models", + install_advice: "Lemonade could not install its speech-to-text (whisper.cpp) backend; \ + check the network connection and run the same `rocm serve` command \ + again, which retries the download", +}; + +pub(crate) const SPEECH: SpeechRecipe = SpeechRecipe { + task: ModelTask::Speech, + recipe: "openmoss", + load_flag: "--openmoss", + backend_option: "openmoss_backend", + dtype: "gguf", + estimated_memory: "about 14 GiB of GPU memory for OpenMOSS-TTS (8-bit)", + install_advice: "Lemonade could not install its text-to-speech (OpenMOSS) backend; \ + check the network connection and run the same `rocm serve` command \ + again, which retries the download", +}; + +impl SpeechRecipe { + pub(crate) const fn health_keys(&self) -> (&'static str, &'static str) { + (self.recipe, self.backend_option) + } +} + +/// The recipe a speech task is served with, or `None` for one that is not a +/// speech task. +pub(crate) const fn speech_recipe(task: ModelTask) -> Option<&'static SpeechRecipe> { + match task { + ModelTask::Transcription => Some(&TRANSCRIPTION), + ModelTask::Speech => Some(&SPEECH), + ModelTask::Chat | ModelTask::Unknown => None, + } +} /// GPU backends only, best first. `cpu` is deliberately absent: under the /// GPU-required policy a host without a usable GPU backend fails instead of -/// transcribing on the CPU. -const WHISPERCPP_BACKEND_PRIORITY: [&str; 2] = ["rocm", "vulkan"]; - -/// Shown when the whisper.cpp backend fails to install twice in a row. The -/// download is retried by every serve that finds the backend missing, so -/// running the same command again is the step that installs it. -const WHISPERCPP_INSTALL_ADVICE: &str = "Lemonade could not install its speech-to-text \ - (whisper.cpp) backend; check the network connection and run the same `rocm serve` \ - command again, which retries the download"; +/// running the model on the CPU. +const SPEECH_BACKEND_PRIORITY: [&str; 2] = ["rocm", "vulkan"]; /// How long `lemond` gets to answer its status query after it is spawned. const LEMOND_STARTUP_TIMEOUT: Duration = Duration::from_secs(30); -pub(crate) fn resolve_transcription_model( +pub(crate) fn resolve_speech_model( + recipe: &SpeechRecipe, canonical_model_id: String, device_policy: DevicePolicy, engine_recipe: Option, ) -> ResolveModelResponse { ResolveModelResponse { canonical_model_id, - task: rocm_engine_protocol::ModelTask::Transcription - .as_str() - .to_owned(), + task: recipe.task.as_str().to_owned(), source: "lemonade".to_owned(), revision: "main".to_owned(), - loader: WHISPERCPP_RECIPE.to_owned(), + loader: recipe.recipe.to_owned(), trust_remote_code: false, chat_template_mode: "none".to_owned(), - dtype: "ggml".to_owned(), + dtype: recipe.dtype.to_owned(), device_policy, - estimated_memory: "under 2 GiB for Whisper Large-v3-Turbo; well under 1 GiB for the \ - smaller Whisper models" - .to_owned(), + estimated_memory: recipe.estimated_memory.to_owned(), launch_defaults: json!({ "host": DEFAULT_HOST, "port": rocm_core::DEFAULT_LOCAL_PORT, "endpoint_mode": "openai" }), engine_recipe, - warnings: vec![ - "Lemonade runs Whisper on its ROCm whisper.cpp backend, or Vulkan when ROCm is \ - unsupported on this GPU; its CPU build is never selected" - .to_owned(), - ], + warnings: vec![format!( + "Lemonade runs this model on its ROCm {recipe} backend, or Vulkan when ROCm is \ + unsupported on this GPU; a CPU build is never selected", + recipe = recipe.recipe + )], } } -/// Serve a speech-to-text model until the router exits. -pub(crate) fn serve_transcription( +/// Serve a speech model until the router exits. +pub(crate) fn serve_speech_model( + recipe: &SpeechRecipe, request: &ServeHttpRequest, runtime: &LemonadeRuntime, process_env: &LemonadeProcessEnvironment, @@ -103,7 +153,7 @@ pub(crate) fn serve_transcription( Some(child.id()), "running", )?; - let prepared = prepare_transcription_model(request, runtime, process_env, log_path); + let prepared = prepare_speech_model(recipe, request, runtime, process_env, log_path); let backend = match prepared { Ok(backend) => backend, Err(error) => { @@ -116,6 +166,9 @@ pub(crate) fn serve_transcription( &request.state_path, &json!({ "status": "ready", + // The post-load smoke test was a real request answered with 200, + // so the CLI need not repeat it before calling the server ready. + rocm_core::INFERENCE_VERIFIED_STATE_KEY: rocm_core::unix_time_millis() as u64, "server_pid": child.id(), "server_start_ticks": rocm_core::process_start_ticks(child.id()), "backend_state": "ready", @@ -124,7 +177,7 @@ pub(crate) fn serve_transcription( "status": "loaded", "method": "lemonade-cli", "model_name": request.model_ref, - "whispercpp_backend": backend + recipe.backend_option: backend }, }), )?; @@ -145,9 +198,10 @@ pub(crate) fn serve_transcription( } /// Bring the router to the point of serving: pick and install a GPU backend, -/// load the model on it, and prove one transcription succeeds. Returns the +/// load the model on it, and prove one real request succeeds. Returns the /// backend in use. -fn prepare_transcription_model( +fn prepare_speech_model( + recipe: &SpeechRecipe, request: &ServeHttpRequest, runtime: &LemonadeRuntime, process_env: &LemonadeProcessEnvironment, @@ -164,16 +218,16 @@ fn prepare_transcription_model( ) .context("Lemonade server did not become ready")?; let listing = crate::backend_alignment::run_lemonade_backends_list(manifest, process_env)?; - let backend = select_whispercpp_backend(&listing)?; + let backend = select_backend(recipe, &listing)?; if backend.already_installed { eprintln!( - "Using installed Lemonade {WHISPERCPP_RECIPE}:{} backend.", - backend.name + "Using installed Lemonade {}:{} backend.", + recipe.recipe, backend.name ); } else { eprintln!( - "Installing Lemonade {WHISPERCPP_RECIPE}:{} backend...", - backend.name + "Installing Lemonade {}:{} backend...", + recipe.recipe, backend.name ); crate::backend_alignment::install_backend_with_retry( || { @@ -181,17 +235,18 @@ fn prepare_transcription_model( manifest, &request.host, request.port, - WHISPERCPP_RECIPE, + recipe.recipe, &backend.name, process_env, ) }, - WHISPERCPP_INSTALL_ADVICE, + recipe.install_advice, )?; } crate::process::run_lemonade_load_command( manifest, - &transcription_load_args( + &load_args( + recipe, &request.host, request.port, &request.model_ref, @@ -202,30 +257,38 @@ fn prepare_transcription_model( ) .with_context(|| { format!( - "failed to load {} with Lemonade {WHISPERCPP_RECIPE}:{}", - request.model_ref, backend.name + "failed to load {} with Lemonade {}:{}", + request.model_ref, recipe.recipe, backend.name ) })?; let health = crate::process::query_health_json(&request.host, request.port)?; - if !health_has_transcription_model(&health, &request.model_ref, &backend.name) { + if !health_has_model(recipe, &health, &request.model_ref, &backend.name) { bail!( - "Lemonade load completed but the server did not report {} loaded on its \ - {WHISPERCPP_RECIPE}:{} backend", + "Lemonade load completed but the server did not report {} loaded on its {}:{} \ + backend", request.model_ref, + recipe.recipe, backend.name ); } - let status = rocm_core::openai_transcription_status( + let status = rocm_core::openai_inference_status( + recipe.task, &format_http_base_url(&request.host, request.port), &request.model_ref, rocm_engine_protocol::resolve_endpoint_api_key().as_deref(), - rocm_core::INFERENCE_PROBE_TIMEOUT, + rocm_core::inference_probe_timeout(recipe.task), ) - .context("the post-load transcription smoke test did not complete")?; + .with_context(|| { + format!( + "the post-load {} smoke test did not complete", + recipe.task.display_name() + ) + })?; if status != 200 { bail!( - "{} loaded, but a test transcription returned HTTP {status}", - request.model_ref + "{} loaded, but a test {} request returned HTTP {status}", + request.model_ref, + recipe.task.display_name() ); } Ok(backend.name) @@ -237,16 +300,17 @@ struct SelectedBackend { already_installed: bool, } -/// Pick the whisper.cpp backend from `lemonade backends` output. -fn select_whispercpp_backend(listing: &str) -> Result { - let statuses = crate::backend_alignment::parse_backend_statuses(listing, WHISPERCPP_RECIPE); +/// Pick the recipe's backend from `lemonade backends` output. +fn select_backend(recipe: &SpeechRecipe, listing: &str) -> Result { + let statuses = crate::backend_alignment::parse_backend_statuses(listing, recipe.recipe); let (name, already_installed) = - crate::backend_alignment::select_best_backend(&statuses, &WHISPERCPP_BACKEND_PRIORITY) + crate::backend_alignment::select_best_backend(&statuses, &SPEECH_BACKEND_PRIORITY) .ok_or_else(|| { anyhow!( - "Lemonade reports no supported GPU speech-to-text backend for this host \ - (status: {}). The GPU-required policy does not fall back to CPU; check \ - the GPU driver with `rocm examine`.", + "Lemonade reports no supported GPU {} backend for this host (status: {}). \ + The GPU-required policy does not fall back to CPU; check the GPU driver \ + with `rocm examine`.", + recipe.task.display_name(), crate::backend_alignment::describe_backend_statuses(&statuses) ) })?; @@ -256,7 +320,13 @@ fn select_whispercpp_backend(listing: &str) -> Result { }) } -fn transcription_load_args(host: &str, port: u16, model_ref: &str, backend: &str) -> Vec { +fn load_args( + recipe: &SpeechRecipe, + host: &str, + port: u16, + model_ref: &str, + backend: &str, +) -> Vec { vec![ "--host".to_owned(), host.to_owned(), @@ -264,15 +334,15 @@ fn transcription_load_args(host: &str, port: u16, model_ref: &str, backend: &str port.to_string(), "load".to_owned(), model_ref.to_owned(), - "--whispercpp".to_owned(), + recipe.load_flag.to_owned(), backend.to_owned(), "--save-options".to_owned(), ] } -/// Whether Lemonade reports `model_ref` loaded through `whispercpp` on exactly +/// Whether Lemonade reports `model_ref` loaded through the recipe on exactly /// `backend`. -fn health_has_transcription_model(health: &Value, model_ref: &str, backend: &str) -> bool { +fn health_has_model(recipe: &SpeechRecipe, health: &Value, model_ref: &str, backend: &str) -> bool { let model_ref = model_ref.trim(); if model_ref.is_empty() { return false; @@ -285,17 +355,18 @@ fn health_has_transcription_model(health: &Value, model_ref: &str, backend: &str .any(|model| { let str_field = |key: &str| model.get(key).and_then(Value::as_str); str_field("model_name").is_some_and(|name| name.eq_ignore_ascii_case(model_ref)) - && str_field("recipe") == Some(WHISPERCPP_RECIPE) + && str_field("recipe") == Some(recipe.recipe) && model .get("recipe_options") - .and_then(|options| options.get("whispercpp_backend")) + .and_then(|options| options.get(recipe.backend_option)) .and_then(Value::as_str) == Some(backend) }) } -/// The healthcheck's "listed" signal for a speech-to-text service. -pub(crate) fn query_loaded_transcription_model( +/// The healthcheck's "listed" signal for a speech service. +pub(crate) fn query_loaded_speech_model( + recipe: &SpeechRecipe, endpoint_url: &str, model_ref: &str, backend: &str, @@ -303,7 +374,7 @@ pub(crate) fn query_loaded_transcription_model( let (host, port) = crate::state::parse_http_endpoint(endpoint_url) .with_context(|| format!("unsupported endpoint URL `{endpoint_url}`"))?; let health = crate::process::query_health_json(&host, port)?; - Ok(health_has_transcription_model(&health, model_ref, backend)) + Ok(health_has_model(recipe, &health, model_ref, backend)) } #[cfg(test)] @@ -336,7 +407,7 @@ whispercpp cpu installed v1.8.4 #[test] fn rocm_unsupported_selects_vulkan_even_when_the_cpu_build_is_installed() { assert_eq!( - select_whispercpp_backend(BACKENDS_NO_ROCM).expect("vulkan is usable"), + select_backend(&TRANSCRIPTION, BACKENDS_NO_ROCM).expect("vulkan is usable"), SelectedBackend { name: "vulkan".to_owned(), already_installed: false, @@ -351,7 +422,7 @@ whispercpp cpu installed v1.8.4 " rocm installed v1.8.4 -\n vulkan installable Backend is supported but not installed. lemonade backends install whispercpp:vulkan", ); assert_eq!( - select_whispercpp_backend(&listing).expect("rocm is usable"), + select_backend(&TRANSCRIPTION, &listing).expect("rocm is usable"), SelectedBackend { name: "rocm".to_owned(), already_installed: true, @@ -367,7 +438,7 @@ whispercpp cpu installed v1.8.4 rocm unsupported Unsupported GPU - vulkan unsupported Unsupported GPU - "; - let error = select_whispercpp_backend(listing) + let error = select_backend(&TRANSCRIPTION, listing) .expect_err("CPU must never be selected") .to_string(); assert!(error.contains("does not fall back to CPU"), "{error}"); @@ -379,8 +450,10 @@ whispercpp cpu installed v1.8.4 // `acestep` and `openmoss` are 11.5.1 recipes with a usable `vulkan` // row; were they unknown to the parser, those rows would be read as // continuation rows of the recipe above them. - let statuses = - crate::backend_alignment::parse_backend_statuses(BACKENDS_NO_ROCM, WHISPERCPP_RECIPE); + let statuses = crate::backend_alignment::parse_backend_statuses( + BACKENDS_NO_ROCM, + TRANSCRIPTION.recipe, + ); assert_eq!( statuses, vec![ @@ -419,7 +492,7 @@ whispercpp cpu installed v1.8.4 attempts += 1; anyhow::bail!("download interrupted") }, - WHISPERCPP_INSTALL_ADVICE, + TRANSCRIPTION.install_advice, ) .expect_err("both attempts fail"); assert_eq!(attempts, 2); @@ -431,19 +504,81 @@ whispercpp cpu installed v1.8.4 ); // The remedy holds because a serve that finds the backend missing // installs it: an `installable` backend is selected as not yet - // installed, which is what sends `prepare_transcription_model` down the + // installed, which is what sends `prepare_speech_model` down the // install path again. assert!( - !select_whispercpp_backend(BACKENDS_NO_ROCM) + !select_backend(&TRANSCRIPTION, BACKENDS_NO_ROCM) .expect("vulkan is usable") .already_installed ); } + #[test] + fn text_to_speech_uses_the_openmoss_gpu_backend_and_its_own_load_flag() { + assert_eq!( + select_backend(&SPEECH, BACKENDS_NO_ROCM).expect("vulkan is usable"), + SelectedBackend { + name: "vulkan".to_owned(), + already_installed: false, + } + ); + assert_eq!( + load_args(&SPEECH, "127.0.0.1", 13305, "OpenMOSS-TTS", "vulkan")[6..8], + ["--openmoss", "vulkan"] + ); + let loaded = json!({"all_models_loaded": [{ + "model_name": "OpenMOSS-TTS", + "recipe": "openmoss", + "recipe_options": {"openmoss_backend": "vulkan"} + }]}); + assert!(health_has_model(&SPEECH, &loaded, "OpenMOSS-TTS", "vulkan")); + assert!( + !health_has_model(&TRANSCRIPTION, &loaded, "OpenMOSS-TTS", "vulkan"), + "a text-to-speech model is not a loaded speech-to-text one" + ); + } + + #[test] + fn each_speech_task_reads_its_own_recipe_row() { + // ROCm usable for OpenMOSS but not for whisper.cpp: each task must pick + // from its own recipe's rows, not the other's. + let listing = BACKENDS_NO_ROCM.replace( + "openmoss rocm unsupported Unsupported GPU -", + "openmoss rocm installable Backend is supported but not installed. lemonade backends install openmoss:rocm", + ); + assert_ne!(listing, BACKENDS_NO_ROCM, "fixture edit applied"); + assert_eq!( + select_backend(&SPEECH, &listing) + .expect("rocm is usable") + .name, + "rocm" + ); + assert_eq!( + select_backend(&TRANSCRIPTION, &listing) + .expect("vulkan is usable") + .name, + "vulkan" + ); + } + + #[test] + fn only_speech_tasks_have_a_speech_recipe() { + assert_eq!( + speech_recipe(ModelTask::Transcription).map(|r| r.recipe), + Some("whispercpp") + ); + assert_eq!( + speech_recipe(ModelTask::Speech).map(|r| r.recipe), + Some("openmoss") + ); + assert!(speech_recipe(ModelTask::Chat).is_none()); + assert!(speech_recipe(ModelTask::Unknown).is_none()); + } + #[test] fn load_names_the_backend_explicitly_instead_of_auto() { assert_eq!( - transcription_load_args("127.0.0.1", 13305, "Whisper-Tiny", "rocm"), + load_args(&TRANSCRIPTION, "127.0.0.1", 13305, "Whisper-Tiny", "rocm"), vec![ "--host", "127.0.0.1", @@ -474,16 +609,23 @@ whispercpp cpu installed v1.8.4 #[test] fn loaded_model_must_be_on_the_requested_backend() { - assert!(health_has_transcription_model( + assert!(health_has_model( + &TRANSCRIPTION, &loaded("vulkan", "gpu"), "Whisper-Tiny", "vulkan" )); assert!( - !health_has_transcription_model(&loaded("cpu", "cpu"), "Whisper-Tiny", "vulkan"), + !health_has_model( + &TRANSCRIPTION, + &loaded("cpu", "cpu"), + "Whisper-Tiny", + "vulkan" + ), "a model Lemonade loaded on its CPU build is not the one we asked for" ); - assert!(!health_has_transcription_model( + assert!(!health_has_model( + &TRANSCRIPTION, &loaded("vulkan", "gpu"), "Whisper-Base", "vulkan" @@ -494,7 +636,8 @@ whispercpp cpu installed v1.8.4 fn the_device_label_is_not_trusted_either_way() { // Lemonade 11.5.1 labels a ROCm whisper.cpp model `cpu`; the backend // decides, not the label. - assert!(health_has_transcription_model( + assert!(health_has_model( + &TRANSCRIPTION, &loaded("rocm", "cpu"), "Whisper-Tiny", "rocm" diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 169c7cef2a..eb62839cb9 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -377,3 +377,24 @@ Feature: Model serving When the user lists running services Then the service is listed as speech-to-text with its transcription endpoint And the listed speech-to-text endpoint reaches that server's transcription route + + + # Text-to-speech on the GPU: Lemonade's OpenMOSS backend (ROCm, or Vulkan + # where ROCm is unsupported; it has no CPU build). Nightly because the model + # is a 12.5 GB download, which a per-PR lane cannot afford. + @id:serve-text-to-speech-model-speaks-on-the-printed-endpoint @requires-gpu @requires-engine:lemonade @nightly @serve-timeout:2400 + Scenario: serve-29 - A served text-to-speech model speaks on the endpoint the CLI prints + Given a managed runtime is active + When the user serves a text-to-speech model on Lemonade + Then the CLI prints the text-to-speech endpoint of the served model + And text sent to that endpoint comes back as spoken audio + And the running service is listed as text-to-speech + And the API docs link printed when it started opens the server's text-to-speech API docs + + # The text-to-speech counterpart of serve-26: refused before the GPU gate, so + # it runs on every lane. The command it recommends is the one serve-29 runs. + @id:serve-text-to-speech-model-refused-by-chat-only-engine + Scenario: serve-30 - A text-to-speech model sent to a chat-only engine is refused with a working alternative + When the user serves a text-to-speech model on an engine that only serves chat + Then serving is refused because that engine cannot serve text-to-speech models + And the user is told the command that serves the text-to-speech model with Lemonade diff --git a/tests/e2e-cucumber/tests/e2e/service_docs_steps.rs b/tests/e2e-cucumber/tests/e2e/service_docs_steps.rs index ebe9916cbc..e497b5bbc9 100644 --- a/tests/e2e-cucumber/tests/e2e/service_docs_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/service_docs_steps.rs @@ -337,6 +337,17 @@ async fn serve_printed_chat_docs_link_opens(world: &mut E2eWorld) { .await; } +#[then("the API docs link printed when it started opens the server's text-to-speech API docs")] +async fn serve_printed_tts_docs_link_opens(world: &mut E2eWorld) { + let stdout = world.serve_stdout.clone().expect("no serve output"); + serve_printed_docs_link_opens( + &stdout, + &["/v1/models", "/v1/audio/speech"], + &["/v1/chat/completions", "/v1/audio/transcriptions"], + ) + .await; +} + #[then("the API docs link printed when it started opens the server's speech-to-text API docs")] async fn serve_printed_speech_docs_link_opens(world: &mut E2eWorld) { let stdout = world.serve_stdout.clone().expect("no serve output"); diff --git a/tests/e2e-cucumber/tests/e2e/speech_steps.rs b/tests/e2e-cucumber/tests/e2e/speech_steps.rs index 60a6778d94..079c2eb798 100644 --- a/tests/e2e-cucumber/tests/e2e/speech_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/speech_steps.rs @@ -13,6 +13,9 @@ use crate::E2eWorld; /// The speech-to-text model the scenarios use: the smallest Whisper recipe, so /// the GPU scenario's download stays small. const SPEECH_MODEL_REF: &str = "whisper-tiny"; +/// The text-to-speech model serve-29 serves on a GPU lane and serve-30 names +/// in its refusal. +const TTS_MODEL_REF: &str = "tts"; const SPEECH_MODEL_ID: &str = "Whisper-Tiny"; fn cli_text(world: &E2eWorld) -> String { @@ -225,28 +228,27 @@ async fn serve_speech_model_on_chat_only_engine(world: &mut E2eWorld) { record_cli(world, result); } -/// Exactly the command the chat-only refusal recommends, so this scenario is -/// also the proof that the recommendation works. +/// Serve a speech model on Lemonade and wait until it is ready. /// /// The serve's own readiness wait is short, and a cold lane spends longer than -/// that downloading the model and the whisper.cpp backend, so a launch that -/// returns before it is ready is followed by polling `rocm services list`, -/// whose readiness check sends a transcription, until the service reports -/// ready or the scenario's serve timeout runs out. -#[when("the user serves a speech-to-text model with the engine the CLI recommended")] -async fn serve_speech_model_on_lemonade(world: &mut E2eWorld) { +/// that downloading the model and its backend, so a launch that returns before +/// it is ready is followed by polling `rocm services list` (whose readiness +/// check sends a real request for the task) until the service reports ready or +/// the scenario's serve timeout runs out. +async fn serve_speech_model(world: &mut E2eWorld, model_ref: &str, model_id: &str, task: &str) { let device_state = crate::e2e::serving_steps::ensure_serve_port_free().await; crate::e2e::service_docs_steps::give_api_docs_a_port_of_their_own(world); - let args = ["serve", SPEECH_MODEL_REF, "--engine", "lemonade"]; + let args = ["serve", model_ref, "--engine", "lemonade"]; let (stdout, stderr, rc) = crate::run_rocm(world, &args); let timeout = std::time::Duration::from_secs(crate::e2e::serving_steps::serve_timeout_for(world)); let started = std::time::Instant::now(); + let listed_task = format!("task: {task}"); let mut ready = rc == 0 && stdout.lines().any(|line| line.trim() == "readiness: ready"); while rc == 0 && !ready && started.elapsed() < timeout { tokio::time::sleep(std::time::Duration::from_secs(5)).await; let (listing, _, _) = crate::run_rocm(world, &["services", "list"]); - ready = listing.contains("status: ready") && listing.contains("task: speech-to-text"); + ready = listing.contains("status: ready") && listing.contains(&listed_task); } if !ready { let outcome = if rc == 0 { @@ -271,7 +273,19 @@ async fn serve_speech_model_on_lemonade(world: &mut E2eWorld) { } world.serve_stdout = Some(stdout.clone()); record_cli(world, (stdout, stderr, rc)); - world.model_name = Some(SPEECH_MODEL_ID.to_owned()); + world.model_name = Some(model_id.to_owned()); +} + +/// Exactly the command the chat-only refusal recommends, so this scenario is +/// also the proof that the recommendation works. +#[when("the user serves a speech-to-text model with the engine the CLI recommended")] +async fn serve_speech_model_on_lemonade(world: &mut E2eWorld) { + serve_speech_model(world, SPEECH_MODEL_REF, SPEECH_MODEL_ID, "speech-to-text").await; +} + +#[when("the user serves a text-to-speech model on Lemonade")] +async fn serve_tts_model_on_lemonade(world: &mut E2eWorld) { + serve_speech_model(world, TTS_MODEL_REF, "OpenMOSS-TTS", "text-to-speech").await; } #[when("the user asks the chat CLI to use the speech-to-text model")] @@ -293,6 +307,48 @@ async fn chat_with_speech_model(world: &mut E2eWorld) { // ── Then ─────────────────────────────────────────────────────────── +#[when("the user serves a text-to-speech model on an engine that only serves chat")] +async fn serve_tts_model_on_chat_only_engine(world: &mut E2eWorld) { + let result = crate::run_rocm(world, &["serve", TTS_MODEL_REF, "--engine", "vllm"]); + record_cli(world, result); +} + +#[then("serving is refused because that engine cannot serve text-to-speech models")] +async fn tts_refused_for_engine(world: &mut E2eWorld) { + let rc = world.cli_rc.expect("no serve rc recorded"); + let text = cli_text(world); + assert_ne!(rc, 0, "serving must be refused:\n{text}"); + assert!( + text.contains("is a text-to-speech model, which the vllm engine cannot serve"), + "the refusal must give the reason:\n{text}" + ); + assert!( + !text.to_lowercase().contains("no usable amd gpu"), + "the reason must not be hidden behind the GPU check:\n{text}" + ); +} + +#[then("the user is told the command that serves the text-to-speech model with Lemonade")] +async fn told_tts_lemonade_command(world: &mut E2eWorld) { + let text = cli_text(world); + // serve-29 runs this exact command on the nightly GPU lane and hears it. + assert!( + text.contains(&format!("rocm serve {TTS_MODEL_REF} --engine lemonade")), + "expected the working command in:\n{text}" + ); +} + +#[then("the running service is listed as text-to-speech")] +async fn running_service_listed_as_tts(world: &mut E2eWorld) { + let (listing, _, _) = crate::run_rocm(world, &["services", "list"]); + assert!( + listing.contains("task: text-to-speech") + && listing.contains("text-to-speech endpoint:") + && listing.contains("status: ready"), + "the served model must be listed as a ready text-to-speech service:\n{listing}" + ); +} + #[then("serving is refused because that engine cannot serve speech-to-text models")] async fn refused_for_engine(world: &mut E2eWorld) { let rc = world.cli_rc.expect("no serve rc recorded"); @@ -335,6 +391,54 @@ async fn prints_speech_endpoint(world: &mut E2eWorld) { world.endpoint = Some(url); } +#[then("the CLI prints the text-to-speech endpoint of the served model")] +async fn prints_tts_endpoint(world: &mut E2eWorld) { + let text = cli_text(world); + let url = printed_url( + world.cli_output.as_deref().unwrap_or_default(), + "text-to-speech endpoint", + ) + .unwrap_or_else(|| panic!("no text-to-speech endpoint printed:\n{text}")); + assert!( + url.ends_with("/v1/audio/speech"), + "the printed endpoint must be the speech route, got {url}" + ); + world.endpoint = Some(url); +} + +#[then("text sent to that endpoint comes back as spoken audio")] +async fn text_comes_back_as_audio(world: &mut E2eWorld) { + let url = world + .endpoint + .clone() + .expect("no printed endpoint recorded"); + let model = world.model_name.clone().expect("no model name recorded"); + let response = reqwest::Client::new() + .post(&url) + .json(&serde_json::json!({ + "model": model, + "input": "Hello from ROCm.", + "response_format": "wav" + })) + .timeout(std::time::Duration::from_mins(5)) + .send() + .await + .unwrap_or_else(|error| panic!("speech request to {url} failed: {error:?}")); + let status = response.status().as_u16(); + let audio = response.bytes().await.unwrap_or_default(); + assert_eq!( + status, + 200, + "speech at {url} failed: {}", + String::from_utf8_lossy(&audio) + ); + assert!( + audio.starts_with(b"RIFF") && audio.len() > 44, + "expected a WAV file with samples, got {} bytes", + audio.len() + ); +} + #[then("a recording sent to that endpoint is transcribed")] async fn recording_transcribed_at_printed_endpoint(world: &mut E2eWorld) { let url = world