Repository navigation
ROCMAI-863: refuse a variantless HF repo before Lemonade starts #919
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
8b51abe
15f9ce9
3fb7452
3cf9edd
866f4b1
c96bd06
12a6ef1
d55aff3
b45ff51
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -472,6 +472,16 @@ fn install_response(request: InstallRequest) -> Result<InstallResponse> { | |
| } | ||
|
|
||
| fn resolve_model_response(request: ResolveModelRequest) -> Result<ResolveModelResponse> { | ||
| // Refuse an unservable model reference on the pre-launch round trip `rocm serve` | ||
| // makes, not only in `serve_hf_checkpoint`. That backstop runs three layers down, | ||
| // inside the background engine process — which on Windows is spawned with no | ||
| // stdout or stderr, and whose exit status nobody reads — so its refusal is | ||
| // invisible there: the user gets an empty log, exit 0, and a service recorded as | ||
| // stopped, with nothing anywhere saying what was wrong. Answering here puts the | ||
| // reason on the user's terminal before any engine starts or any record is written. | ||
| if let Some(message) = crate::direct_llama::hf_checkpoint_variant_required(&request.model_ref) { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This guard now applies to every caller of |
||
| bail!("{message}"); | ||
| } | ||
| let device_policy = crate::device_policy::normalize_device_policy(request.device_policy)?; | ||
| let engine_recipe = crate::state::accepted_engine_recipe(request.engine_recipe)?; | ||
| let canonical_model_id = crate::state::resolve_lemonade_model_ref(&request.model_ref); | ||
|
|
@@ -889,4 +899,56 @@ mod tests { | |
| fn stop_scope_targets_only_the_recorded_lemonade_server() { | ||
| assert_eq!(STOP_SCOPE, rocm_core::KillScope::Single); | ||
| } | ||
|
|
||
| /// A `ResolveModel` request as `rocm serve` makes it for a plain | ||
| /// `rocm serve <model> --engine lemonade`: no runtime, no device policy, no | ||
| /// recipe, so only the model reference is under test. | ||
| fn resolve_model_request(model_ref: &str) -> ResolveModelRequest { | ||
| ResolveModelRequest { | ||
| model_ref: model_ref.to_owned(), | ||
| runtime_id: None, | ||
| device_policy: None, | ||
| recipe_override: None, | ||
| engine_recipe: None, | ||
| } | ||
| } | ||
|
|
||
| #[test] | ||
| fn resolving_a_hugging_face_repo_without_a_variant_is_refused() { | ||
| let error = resolve_model_response(resolve_model_request("Qwen/Qwen2.5-1.5B-Instruct")) | ||
| .expect_err("a bare owner/repo has no GGUF to serve and must be refused up front"); | ||
| let message = error.to_string(); | ||
| assert!( | ||
| message.contains("`owner/repo:<variant>`"), | ||
| "the refusal must show the form that would work, got:\n{message}" | ||
| ); | ||
| // The repository was never inspected, so naming a quantization would be a | ||
| // guess — and a wrong guess sends the user into a second failure. This | ||
| // repo, for one, ships only safetensors and has no `Q4_K_M` build. | ||
| assert!( | ||
| !message.contains("Q4_K_M") && !message.contains("Q4_0"), | ||
| "the refusal must not suggest a quantization the CLI has not seen, got:\n{message}" | ||
| ); | ||
| } | ||
|
|
||
| #[test] | ||
| fn resolving_a_servable_model_still_succeeds() { | ||
| // An `owner/repo:variant` is passed through verbatim: the user named an | ||
| // exact checkpoint and quantization, and nothing may substitute for it. | ||
| for model_ref in [ | ||
| "Qwen/Qwen2.5-1.5B-Instruct:Q4_K_M", | ||
| "unsloth/Qwen3-0.6B-GGUF:Q4_0", | ||
| ] { | ||
| let response = resolve_model_response(resolve_model_request(model_ref)) | ||
| .unwrap_or_else(|error| panic!("{model_ref} must still resolve: {error}")); | ||
| assert_eq!(response.canonical_model_id, model_ref); | ||
| } | ||
| // Short names are not checkpoints at all, so the guard must leave them to | ||
| // the alias expansion that has always handled them. | ||
| for model_ref in ["qwen", DEFAULT_MODEL] { | ||
| let response = resolve_model_response(resolve_model_request(model_ref)) | ||
| .unwrap_or_else(|error| panic!("{model_ref} must still resolve: {error}")); | ||
| assert_eq!(response.canonical_model_id, DEFAULT_MODEL); | ||
| } | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -34,17 +34,26 @@ | |
|
|
||
| # --- EAI-7219: short-name expansion (vLLM does not show an expanded name) --- | ||
| # What the lane logs show, so the scope below is not read as more than it is: | ||
| # lemonade DOES print the full name (`resolved model: Qwen/Qwen2.5-1.5B-Instruct`) | ||
| # on the Strix Halo Ubuntu and Windows lanes, and vLLM's stdout is empty -- the run | ||
| # stopped before it printed a serve plan, so the plan's name line was never | ||
| # reached. EAI-7219 as reported describes the other shape: a vLLM plan that prints | ||
| # `resolved model: qwen2.5`, the alias left unexpanded; the lane logs have not | ||
| # shown it. The steps used to discard stderr, so the cause of the empty stdout was | ||
| # not known. The first lane run that reported it (run 41811714290: Strix Halo | ||
| # Ubuntu release and nightly, and MI300X) shows vLLM stopping at "no active ROCm | ||
| # runtime is configured", before any alias handling, so on those lanes the rows are | ||
| # red for a missing runtime and not for this bug. On native Windows the stop is the | ||
| # up-front vLLM refusal (ROCMAI-823), a designed refusal and not this bug either. | ||
| # lemonade prints the full name on the Strix Halo Ubuntu and Windows lanes, and | ||
| # vLLM's stdout is empty -- the run stopped before it printed a serve plan, so the | ||
| # plan's name line was never reached. vLLM is therefore the whole of what keeps | ||
| # these rows red. | ||
| # | ||
| # ROCMAI-863's pre-launch refusal does NOT reach these rows, though an earlier | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: this comment narrates its own earlier revision ("an earlier revision of this comment said it did"). State only the current truth here and leave the history to the PR body. |
||
| # revision of this comment said it did. That was written when `qwen2.5` expanded | ||
| # to the bare `Qwen/Qwen2.5-1.5B-Instruct`, which ships no GGUF; #1044 then gave | ||
| # that recipe a per-engine ref, so lemonade is handed | ||
| # `Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M` with the variant already pinned and | ||
| # the refusal has nothing to fire on. The lemonade half still prints its name. | ||
| # EAI-7219 as reported describes | ||
| # the other shape: a vLLM plan that prints `resolved model: qwen2.5`, the alias | ||
| # left unexpanded; the lane logs have not shown it. The steps used to discard | ||
| # stderr, so the cause of the empty stdout was not known. The first lane run | ||
| # that reported it (run 41811714290: Strix Halo Ubuntu release and nightly, and | ||
| # MI300X) shows vLLM stopping at "no active ROCm runtime is configured", before | ||
| # any alias handling, so on those lanes the rows are red for a missing runtime | ||
| # and not for this bug. On native Windows the stop is the up-front vLLM refusal | ||
| # (ROCMAI-823), a designed refusal and not this bug either. | ||
| # The rows keep their all-hosts scope until the scenarios get a managed runtime or | ||
| # are gated on the engine and the rows are narrowed to the real bug; their reasons | ||
| # name both shapes so neither becomes a false claim. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -887,3 +887,22 @@ Feature: Model serving | |
| When the user sends a chat completion request | ||
| Then the response contains a model reply | ||
| And the response identifies the correct model | ||
|
|
||
| # ROCMAI-863: a Hugging Face repo named without a quantization variant has no | ||
| # GGUF file to serve, but serving started an engine for it anyway and left the | ||
| # objection to the background process — where, on Windows, nothing the user can | ||
| # see ever reports it: an empty log, a success exit, and a server recorded as | ||
| # stopped. The refusal now comes before anything starts. It also names no | ||
| # quantization of its own: nothing on this path reads the repository's files, so | ||
| # a suggested label is a guess, and following a guess fails a second time on a | ||
| # repo that ships no GGUF at all. @requires-gpu because a host with no usable | ||
| # GPU is turned away earlier, for that reason instead, and never reaches this | ||
| # one. No @requires-os deliberately — Windows is the lane the failure was silent | ||
| # on, so it is the one most worth watching. | ||
| @id:serve-missing-gguf-variant-refused @requires-gpu @requires-engine:lemonade | ||
| Scenario: serve-56 - Serving a Hugging Face repo with no quantization variant is refused | ||
| When the user serves a Hugging Face repo with no quantization variant on Lemonade | ||
| Then serving is refused before any engine starts | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does "serving is refused before any engine starts" assert that no engine process or log was created, or only a nonzero exit? The "no local server was recorded" step covers the record. The engine-not-started promise is the point of this ticket, so it deserves its own state assertion if this step only checks the exit code. |
||
| And the user is told the model needs an explicit GGUF variant | ||
| And no quantization is suggested for a repo the CLI has not inspected | ||
| And no local server was recorded | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Optional: the message hard-codes the placeholder
owner/repo:<variant>while the first clause interpolates the real ref.{model_ref}:<variant>would show exactly what to retype. The unit test and the serve-56 step currently pin the literal, so they would move with it.