Skip to content

ROCMAI-863: refuse a variantless HF repo before Lemonade starts - #919

Merged
r0x0r merged 9 commits into
mainfrom
fix/rocmai-863-lemonade-missing-gguf-variant
Nov 13, 2026
Merged

r0x0r merged 9 commits into
mainfrom
fix/rocmai-863-lemonade-missing-gguf-variant

Conversation

@r0x0r

@r0x0r r0x0r commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm serve Qwen/Qwen2.5-1.5B-Instruct --engine lemonade passed every check the CLI makes, started an engine process, and then failed inside that process. On Windows the failure was silent: the background engine is spawned with no stdout and no stderr, and nothing reads its exit status, so the user was left with a 0-byte service log, exit code 0, and a service recorded as stopped — with nothing anywhere saying what was wrong.

A bare Hugging Face owner/repo names no GGUF file, and Lemonade cannot serve one without a quantization variant. The choice would otherwise come from an interactive variant menu that this non-interactive path cannot answer, or from whatever GGUF happened to be cached, which risks serving a quantization the user never asked for.

rocm serve makes exactly one pre-launch validation round trip to the engine, ResolveModel (apps/rocm/src/serve_cmd.rs). Lemonade's resolve_model_response never asked whether the reference was servable at all. The only check lived in serve_hf_checkpoint, three layers down inside the background engine process — the one place whose output Windows discards.

The refusal that check produced was wrong in a second way. It advised `{model}:Q4_K_M`, a label nothing on that path had looked for: no code there reads the repository's file list. Qwen/Qwen2.5-1.5B-Instruct ships only model.safetensors and has no GGUF build at all, so a user who followed the advice hit a second failure — Lemonade's own Error fetching variants: No supported model files found in repository.

What changes

The refusal moves to the pre-launch ResolveModel, so the reason reaches the terminal on every platform, in both foreground and --managed modes, before any engine process starts and before any service record is written.

The in-process check in serve_hf_checkpoint stays as a backstop, and both callers now route through one helper beside the checkpoint parser. Wording written out twice would drift, and on any given platform only one of the two copies is the one the user ever sees — so the stale copy would be the visible one exactly where it mattered.

The message names no quantization. It gives the owner/repo:<variant> form, sends the user to the repository's .gguf files for the labels that actually exist, and says plainly that a repo with no .gguf files cannot be served by Lemonade under any variant. It is ASCII-only: a Windows console on the OEM code page renders non-ASCII punctuation as mojibake, and Windows is where this message is the user's only sight of the problem.

Ordering, and why this is not fail-fast

ResolveModel runs after the no-usable-GPU refusal and after Lemonade's own runtime preparation. On a host that does not yet have the Lemonade embeddable runtime, rocm serve <owner/repo> therefore installs that runtime and then refuses. That is a real improvement on installing, launching and dying invisibly, but it is not the earliest possible refusal, and the cost is a download the user did not need.

A syntax-only check in the CLI, before runtime preparation, would be earlier and was deliberately not taken: it puts one engine's model-reference grammar inside the CLI, where the next engine's rules would follow it, and the two copies would then have to agree forever. The engine stays the single thing that decides what it can serve.

Verification

Run on a Linux host with an AMD GPU, cold after cargo clean, against a tree verified byte-identical to this commit:

  • cargo test -p rocm-engine-lemonade — 165 passed, 0 failed (162 before this change, 3 new)
  • cargo test --workspace --all-targets --exclude e2e-cucumber — 5041 passed across 69 suites, 0 failed
  • cargo test -p e2e-cucumber --test feature_naming — 4 passed
  • cargo clippy -p e2e-cucumber --test e2e -- -D warnings — clean; this target is not reached by --all-targets and has to be named explicitly
  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo fmt --all -- --check — clean
  • local prek hooks: license headers, markdown links, whitespace, signature and DCO verification — all passed

Both halves of the change were shown to be load-bearing rather than asserted. Reverting only the lib.rs guard turns the rejection test red: the resolve hands back a success envelope for the bare reference. Separately restoring the old Q4_K_M wording turns the "names no quantization" tests red. Each was then restored and the suite went green again.

Not verified

Windows. It is the platform this change exists for, and its cfg(windows) paths were not compiled on either machine available here: the Linux host's x86_64-pc-windows-msvc target dies in a C build dependency (failed to find tool "lib.exe"), and macOS cannot build the lemonade crate at all. Windows coverage for this change is the windows-build-and-test job and nothing else.

serve-56 has not been executed against a real binary. @id:serve-missing-gguf-variant-refused is tagged @requires-gpu @requires-engine:lemonade, so it belongs to the self-hosted Strix Halo Ubuntu and Windows lanes in e2e-selfhosted.yml, which need a Lemonade runtime install. It type-checks, is clippy-clean under -D warnings, and all of its steps resolve, but it has not run. Its three substantive assertions — the refusal arrives before an engine starts, it names owner/repo:<variant>, it names no quantization — are each mirrored by a unit test that was run.

serve-56 is the scenario's current index in model_serving.feature, and tests/e2e-cucumber/tests/feature_naming.rs requires per-feature indexes to be sequential and in file order, so it is declared last in the file rather than beside the sibling refusal scenarios it most resembles. It was written as serve-42; merging main after #1044 added its own scenario ahead of this one and renumbered this to serve-56 (merge d55aff3c). Earlier revisions of this description said serve-42, which at the current head is an unrelated pre-existing WSL scenario.

Scope

tests/e2e-cucumber/expectations.toml carried a comment, and one xfail reason, written when this branch's refusal was expected to change what the qwen2.5 row does. It does not, and both have been corrected to say so. #1044 gave that recipe a per-engine ref, so lemonade is handed Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M with the variant already pinned and this change's guard never fires on it. vLLM stopping before it prints a serve plan is the whole of what keeps those rows red; the lemonade half still prints its name. Both rows still expect failure, for that reason rather than this one.

Two things are deliberately left alone:

  • Lemonade's own No supported model files found in repository error is still surfaced raw. It is a separate defect, and this change narrows rather than widens the path to it: it is now reachable only by naming an explicit owner/repo:variant against a repository that has no GGUF build.
  • The catalog mis-steer this change was originally written against is gone. The entry for Qwen/Qwen2.5-1.5B-Instruct declared preferred_engines: ["lemonade"] for a safetensors-only repository Lemonade can never serve; fix(catalog): point Lemonade at the small assistant's GGUF build #1044 fixed that at the source by giving the recipe a per-engine GGUF ref. The guard here is no longer what saves that model, and is now the backstop for an owner/repo the catalog has no recipe for at all — which is what serve-56 serves, since the original repository would no longer be refused.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. — searched; no row cites ROCMAI-863. Both EAI-7219 rows stay red. Only serve-short-name-consistent-across-engines serves lemonade, so only its reason changes; serve-short-name-expansion serves vLLM only and this change does not alter why it fails.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. — not applicable: no new subcommand or subsystem; the shared helper lives beside the checkpoint parser it depends on.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3. — the refusal is asserted as a returned error from resolve_model_response (no success envelope, nothing launched) and, in serve-56, as a refusal with no local server recorded. The "no invented quantization" half is asserted as an absence, so it cannot be satisfied by pinning whatever wording happens to be present.

r0x0r added 4 commits November 5, 2026 14:37
…CMAI-863)

`rocm serve` makes one pre-launch `ResolveModel` round trip to the engine
before it starts anything. Lemonade never checked there whether the model
reference was servable, so a bare Hugging Face `owner/repo` with no
`:variant` passed every check and the engine process started, only to fail
inside itself. On Windows that background process is spawned with no stdout
or stderr and nobody reads its exit status, so the refusal never surfaced:
the user saw a 0-byte log, exit code 0, and a service recorded as stopped,
with nothing anywhere saying what was wrong.

Answer on the pre-launch resolve instead, so the reason reaches the terminal
on every platform in both foreground and `--managed` modes, and route the
existing in-process backstop through the same shared message so the two
cannot drift apart.

The refusal also no longer offers `:Q4_K_M`. Nothing on this path reads the
repository's file list, so any example label is invented, and following
invented advice fails a second time against a repo that ships no `.gguf`
files at all. It now names no quantization and sends the user to look.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
…CMAI-863)

The ROCMAI-863 note above serve-short-name-expansion and
serve-short-name-consistent-across-engines said "These rows stay red for
that reason on the lemonade half", which reaches both rows. Only
serve-short-name-consistent-across-engines has a lemonade half: its step
loops over both engines, while serve-short-name-expansion hardcodes vLLM
and its reason never mentions lemonade.

Narrow the sentence to the row it is true of, so the note stops claiming
a lemonade failure for a scenario that never starts lemonade. Comment
text only: no when, bug or reason value changes.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
main took serve-42 for its own WSL ROCDXG scenario and now runs to
serve-49, so this branch's GGUF-variant refusal renumbers to serve-50.
Its only reference is the feature file itself; main's serve-42 citations
in docs/testing.md and service_launch.rs are left alone.

main also added `user_serves_with_automatic_port` to serving_steps.rs
beside this branch's new steps; both kept.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main (47 behind). Pushed as 3cf9edd0.

Renumbered: serve-42 → serve-50. main took serve-42 for its own WSL ROCDXG scenario while this branch was open, and now runs to serve-49. feature_naming ties the number to declaration position, so this branch's GGUF-variant refusal takes the next free one. Its only reference was the feature file itself; main's serve-42 citations in docs/testing.md and service_launch.rs belong to the WSL scenario and are untouched.

main also added user_serves_with_automatic_port to serving_steps.rs next to this branch's new steps; both kept.

Verified on a Linux MI300X box with the merged tree applied onto main: cargo clippy --workspace --all-targets -- -D warnings exit 0 · cargo clippy -p e2e-cucumber --test e2e -- -D warnings exit 0 · cargo test --workspace --all-targets exit 0, 77 suites, 0 failures · cargo fmt --all -- --check exit 0.

On the four red lanes

All four fail on the same scenario, and it is not this PR's: serve-stop-leaves-no-background-helper.

  • E2E tests (MI300X, current) — Reconciliation: 5 xfail, 0 XPASS, 1 unexpected failure → FAIL: 'serve-stop-leaves-no-background-helper'
  • both Strix Halo Ubuntu lanes — identical line
  • E2E consolidated report (self-hosted) is strictly downstream: a dead lane uploads no artifact, so the "every lane contributes" guard fires

The same scenario fails on main's own Strix Halo Ubuntu lanes (run 41982405007), and I have now seen it on MI300X too, so it is not Strix-specific. This branch touches only the Lemonade GGUF-variant refusal path and its scenario — nothing in serve-stop or the background helper.

Flagging it here because it is currently red on several open PRs at once and no per-PR work will clear it; it needs fixing on main.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	tests/e2e-cucumber/features/model_serving.feature
@r0x0r

r0x0r commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Merged origin/main to clear the conflict. Pushed as 866f4b19.

Scenario renumbered: serve-50 → serve-53. Main took 50/51/52 for the Anthropic-endpoint scenarios, and the feature_naming guard ties a scenario's number to its position in the file, so this had to move. The @id: tag is unchanged.

⚠️ Landing-order note: #854 also renumbered to serve-53/serve-54 against the same main. Whichever of the two merges first keeps its number and the other needs one more bump — the guard makes it impossible to pre-empt from either side.

Verification (Linux MI300X; the Mac cannot build rocm-engine-lemonade)

  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo clippy -p e2e-cucumber --test e2e -- -D warnings — clean (that target is harness = false, so check/test --all-targets give a false green on the step files)
  • cargo fmt --all --check — clean
  • cargo test -p e2e-cucumber --test feature_naming — 4/4, so the renumbering is consistent
  • cargo test --workspace — one failure, process::tests::released_loopback_lease_is_immediately_bindable_by_another_process. Pre-existing flake, not this PR: git diff origin/main...HEAD -- crates/rocm-core/src/process.rs is empty, and the test passes in isolation. It is one of a family of loopback-lease tests that bind real ports and race under parallel execution — a sibling in the same family failed the same way on fix(serve): log and fail managed-serve crashes instead of exiting 0 (ROCMAI-541) #854's run.

Not verified: anything cfg(windows) — the Linux box cannot build the MSVC target (ring/aws-lc-sys need a real Windows toolchain), so CI's windows-build-and-test lane is the only check there.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@r0x0r

r0x0r commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

CI note: the one red lane is not this PR.

E2E tests (Strix Halo, Ubuntu, nightly) reports Reconciliation: 2 xfail (failed as expected), 8 XPASS (8 flaky, 0 stale), 1 unexpected failure(s). The single unexpected failure is:

automations-11 - The background helper stops on its own once no watcher is enabled
  ✘ Then the CLI reports starting the background helper and it is running

Four reasons that is not attributable to this change:

  1. The diff cannot reach it. This PR touches engines/lemonade/{direct_llama,lib}.rs, model_serving.feature, serving_steps.rs and expectations.toml — nothing in the background automation helper's path.
  2. The expectations.toml edit adds and removes no rows. It changes two reason strings and a comment block, so it cannot flip any other scenario's reconciliation. (Worth saying explicitly, since that file does govern every scenario's xfail/flaky verdict — it was the one plausible mechanism and it does not hold.)
  3. The same lane passed concurrently on fix(serve): log and fail managed-serve crashes instead of exiting 0 (ROCMAI-541) #854 and feat(examine,serve): report real GPU/backend support instead of a static ROCm claim #678, both of which were running against the same main at the same time.
  4. The two other ✘ marks are the expected ones — serve-short-name-expansion and serve-short-name-consistent-across-engines, the EAI-7219 xfail rows whose reasons this PR updates. They are counted as xfail, not as failures.

The failing step starts a helper process and then asserts it is running, which is a timing-sensitive check on a shared self-hosted runner. I read it as a flake on that lane. A re-run is queued behind the still-running MI300X and W7900 lanes; I will re-run it and report the result rather than leaving this on an assertion.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	tests/e2e-cucumber/features/model_serving.feature
@r0x0r

r0x0r commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main again. Pushed as c96bd06a.

Scenario renumbered: serve-53 → serve-54. Main took 53 for the Anthropic tool-use scenario. That is the second renumber on this PR; #678 and #854 are in the same queue and all three are disjoint as of this push (678 → serve-54… see below).

⚠️ Correction to my earlier note: #678 also landed on serve-54 in the same window. These three PRs keep colliding because the feature_naming guard ties a scenario's number to its position in the file, so every one of us is appending to the same tail. Whichever merges first keeps its number; I will re-bump the others as they land rather than guessing now.

The Security scan reds are not this PR — generic-api-key hits on model names and runtime keys in other authors' commits, which the main..head scan range picks up. #997 and #998 fail identically. Main landed 2ce639bc ci(security): allow model names and runtime keys assigned in Rust sources in gitleaks (#1000), which allowlists exactly that, and this push merges it.

The earlier automations-11 failure (background helper) stands as analysed above: unrelated to this diff, and the same lane passed concurrently on #854 and #678. This push re-runs it from scratch.

Verification (Linux MI300X)

  • cargo clippy --workspace --all-targets -- -D warnings — clean
  • cargo clippy -p e2e-cucumber --test e2e -- -D warnings — clean
  • cargo fmt --all --check — clean
  • cargo test -p e2e-cucumber --test feature_naming — 4/4
  • cargo xtask check-architecture-doc / check-crate-edges / check-module-size — pass (module-size is now the blocking gate main added in 2d3e0cd5)

Not verified: anything cfg(windows) — the box cannot build the MSVC target.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>

# Conflicts:
#	tests/e2e-cucumber/features/model_serving.feature
@r0x0r

r0x0r commented Nov 11, 2026

Copy link
Copy Markdown
Collaborator Author

CI note: the WSL2 lane failure is infrastructure, not this change.

E2E tests (Strix Halo, WSL2) produced no results at all — the step died before any scenario ran, during dependency download:

unable to update registry `crates-io`
Caused by: download of an/st/anstyle-parse failed
Caused by: curl failed
Caused by: [16] Error in the HTTP2 framing layer

Hence the follow-on No files were found with the provided path: tests/e2e-cucumber/results/ — there was nothing to upload. Nothing in this PR can reach a crates.io transport error, and anstyle-parse is not a dependency it touches.

A re-run is queued behind the lanes still in flight; I'll confirm the result rather than leave it asserted.

For completeness on the earlier red lane: automations-11 on Strix Halo Ubuntu nightly was likewise unrelated (analysed above) — same lane passed concurrently on #854 and #678.

Current state

Merged with main, no blocking reviews, and the branch is green apart from the two infrastructure failures above. Scenario is now serve-55 after main took 53 and 54 for the Anthropic tool-use and streaming scenarios.

Gates re-run on this head (Linux MI300X): clippy --workspace --all-targets -D warnings, clippy -p e2e-cucumber --test e2e -D warnings, fmt --check, feature_naming 4/4, and xtask check-architecture-doc / check-crate-edges / check-module-size — all pass. Not verified: anything cfg(windows); the box cannot build the MSVC target.

@r0x0r

r0x0r commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

The two red checks are one infrastructure failure, not a code defect

Diagnosing rather than re-running, since this is a draft and the remedy burns a scarce self-hosted runner.

Root cause — E2E tests (Strix Halo, WSL2, current) (job 130310528756): the suite never ran. Cargo could not reach crates.io:

error: failed to get `anstyle-parse` as a dependency of package `anstream v1.0.0`
Caused by: unable to update registry `crates-io`
Caused by: download of an/st/anstyle-parse failed
Caused by: curl failed

Nothing was compiled, no scenario executed, and the step that uploads results then reported No files were found with the provided path: tests/e2e-cucumber/results/.

Consequence — E2E consolidated report (self-hosted) (job 130815743785): its step 6, "Check every lane that ran uploaded its report", failed with:

lane e2e-wsl ran (needs result: failure) but uploaded no e2e-gpu-strix-wsl-report,
so it is missing from the consolidated report

That is the gate working correctly — it noticed the WSL lane produced nothing. It is not a second, independent failure, and re-running the report job alone cannot fix it: the missing artifact has to come from the WSL lane. (The report job was already retried at 21:59 on 11 Nov and failed again for exactly that reason.)

Category: infrastructure (network / crates.io on the self-hosted Strix Halo WSL2 runner). Specifically not the known pre-existing Strix Halo flake — that one is serve-stop-leaves-no-background-helper on the Ubuntu lane and shows up in the Reconciliation: line. Here there is no Reconciliation: line at all, because the run died before cargo finished fetching dependencies.

What I did not do: I have not re-run it. The fix is a re-run of the whole 42829033774 run (not just the report job), and that occupies the shared Strix Halo runner. This PR is a draft at an unchanged head (12a6ef18d), so I did not think it was worth taking that hardware for. Say the word and I will trigger it.

Worth noting the same run is now ~2 days old (created 10 Nov 13:55); whenever this comes out of draft it will want a fresh run regardless.

#1044 added a `serve-55` for the recommended small assistant; main's
keeps that number (docs/testing.md and the feature preamble cite it) and
the ROCMAI-863 refusal scenario becomes `serve-56`.

The serve-02 xfail reason takes main's wording. This branch's version
said lemonade now refuses the expanded safetensors-only repo, but #1044
gave that recipe a per-engine GGUF ref with the variant pinned, so the
refusal no longer fires on that row.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Merged main after #1044 landed. Two conflicts, one of them a real change in meaning rather than a renumbering.

model_serving.feature — #1044 added its own serve-55. Main's keeps that number, since docs/testing.md and the feature file's own preamble cite it, and the ROCMAI-863 refusal scenario moved to serve-56. Its @id:serve-missing-gguf-variant-refused is unchanged, so expectations.toml and the step bindings were unaffected.

expectations.toml, the serve-02 xfail reason — worth a look, because this branch's wording is now wrong. It said:

...and lemonade now refuses the expanded safetensors-only repo before its plan prints (ROCMAI-863).

That was true when written: qwen2.5 expanded to the safetensors-only Qwen/Qwen2.5-1.5B-Instruct, which this branch's guard refuses. #1044 gave that recipe a per-engine ref, so Lemonade is now handed Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M — variant pinned, guard does not fire. I took main's wording instead:

...and the lemonade half is only reached once that stops happening.

So the guard this PR adds no longer has any effect on serve-02. That does not weaken the PR — serve-56 covers the guard directly with a repo that genuinely has no variant — but it does mean serve-02 is no longer indirect evidence for it.

Verified on Linux before pushing: cargo fmt --all --check, clippy --workspace --all-targets -D warnings, clippy -p e2e-cucumber --test e2e -D warnings, test -p e2e-cucumber --test feature_naming (all 4 naming guards pass, so the renumbering is positionally consistent), and xtask check-architecture-doc / check-crate-edges / check-module-size — all green.

…efusal

serve-56 asserts that Lemonade refuses a Hugging Face repo named without
a quantization variant. It served `Qwen/Qwen2.5-1.5B-Instruct`, the id
from the original report, which #1044 then gave a per-engine recipe
resolving to `Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M`. The variant now
arrives from the catalog, so the refusal correctly does not fire: the
Strix Halo Windows lane served the model and exited 0 where the scenario
expects a refusal.

Serve `Qwen/Qwen2.5-3B-Instruct` instead, which has no recipe, and assert
that it has none before running, so the scenario fails with the reason
rather than silently inverting if that id ever gains one.

Also corrects the EAI-7219 comment in expectations.toml, which still said
the ROCMAI-863 refusal is why the lemonade half of serve-02 prints no
name. #1044 removed that effect for the same reason; vLLM stopping before
its serve plan is the whole of what keeps those rows red.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
@r0x0r

r0x0r commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

The Windows lane caught a real inversion in serve-56 — fixed

The Strix Halo Windows run came back with one unexpected failure, and it is this PR's own scenario. It is worth spelling out because the scenario had stopped testing what it claims to test.

Scenario: serve-56 - Serving a Hugging Face repo with no quantization variant is refused
 ✔  When the user serves a Hugging Face repo with no quantization variant on Lemonade
 ✘  Then serving is refused before any engine starts
    expected serving to be refused, but it exited 0:
      requested model: Qwen/Qwen2.5-1.5B-Instruct
      resolved model:  Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M

The step served Qwen/Qwen2.5-1.5B-Instruct, the id from the original report, chosen because it ships safetensors only. #1044 then gave that recipe a per-engine ref, so Lemonade is now handed ...-GGUF:Q4_K_M with the variant already pinned. The guard has nothing to object to, serving proceeds, and the scenario inverts — it had become an assertion that a catalog-resolved model gets refused, which is the opposite of the intent.

So the CLI behaved correctly here and the test was wrong.

Fix. The scenario now serves Qwen/Qwen2.5-3B-Instruct, which the catalog has no recipe for, and the step asserts that up front:

assert!(
    !rocm_core::model_catalog::builtin_model_recipes()
        .iter()
        .any(|recipe| recipe.matches_ref(HF_REPO_WITHOUT_VARIANT)),
    "{HF_REPO_WITHOUT_VARIANT} now has a catalog recipe, which supplies the \
     quantization variant this scenario exists to see refused; pick an id \
     the catalog has no recipe for"
);

That is the part I care about more than the id swap. The failure mode here was silent: a catalog edit in a different PR turned this scenario into a tautology, and nothing said so until a GPU lane ran. The assertion turns that into a named failure at the first step, on every lane, instead of an exit-0 that looks like a product bug.

One more stale claim corrected. The EAI-7219 comment block in expectations.toml still said the ROCMAI-863 refusal is why the lemonade half of serve-02 prints no model name. #1044 removed that effect by the same mechanism. vLLM stopping before it prints a serve plan is the whole of what keeps those rows red; the lemonade half prints its name fine. I had fixed the reason string for this in the merge but left the comment above it contradicting the fix.

Verified on Linux: cargo fmt --all --check, clippy --workspace --all-targets -D warnings, clippy -p e2e-cucumber --test e2e -D warnings, test -p e2e-cucumber --test feature_naming, and xtask check-architecture-doc / check-crate-edges / check-module-size — all green. Qwen/Qwen2.5-3B-Instruct appears nowhere in model_catalog.json under normalize_model_ref's trim-and-lowercase, so the new assertion passes for the right reason.

@r0x0r
r0x0r marked this pull request as ready for review November 13, 2026 07:35
@r0x0r
r0x0r requested a review from a team as a code owner November 13, 2026 07:35
@r0x0r
r0x0r requested a review from volen-silo November 13, 2026 07:35

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head b45ff510
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (fan-out to independent workers failed on every attempt — see "What this covered" — so this was conducted as a single, thorough-as-possible context review instead).
Implements ROCMAI-863 in full: the title and scope match the ticket's own summary ("reject a Lemonade owner/repo with no variant before launching, and stop suggesting :Q4_K_M for repos without GGUF files") exactly.

Blocking

  • PR description cites a scenario id (serve-42) that is not this change's scenario — PR body ("Verification" and "Not verified" sections), tests/e2e-cucumber/features/model_serving.feature:727 vs :905.
    The description states serve-42 is "the next free index in model_serving.feature" and the id of the new refusal scenario, and that "serve-42 has not been executed against a real binary." At the PR's actual head (b45ff510), serve-42 is a pre-existing, unrelated scenario — "Serving on a WSL host whose ROCDXG library the loader cannot find is refused with WSL advice" (model_serving.feature:727, untouched by this diff). The scenario this PR actually adds is @id:serve-missing-gguf-variant-refused, named serve-56 (model_serving.feature:905), per this same branch's own merge commit d55aff3c ("the ROCMAI-863 refusal scenario becomes serve-56"). The PR body was never updated after that renumbering, so a reader following its own "Not verified" pointer lands on the wrong, already-covered scenario.
    Confidence 85 · mechanical · Fix: replace the serve-42 references in the PR body with serve-56, consistent with this repo's own AGENTS.md §5 ("write the PR body from the final diff ... re-read it after every review-fix commit").

Non-blocking

None

Decisions for the author

  • Refusing only at the engine's pre-launch ResolveModel, not with an earlier CLI-side syntax check — tradeoff
    An earlier CLI-side check of the owner/repo[:variant] grammar would refuse before Lemonade's runtime-prep step runs (avoiding an unneeded download on a host without the runtime yet), but would duplicate one engine's model-reference grammar inside the CLI, which the next engine would then have to follow too. The PR's own "Ordering, and why this is not fail-fast" section already lays out both sides; noted here only so it's visible as a deliberate choice.
  • The built-in catalog entry for Qwen/Qwen2.5-1.5B-Instruct still lists preferred_engines: ["lemonade"] for a safetensors-only repo Lemonade can never serve — tradeoff
    This PR turns that catalog mis-steer into an immediate, legible refusal instead of a silent one (a real improvement) but leaves the mis-steer itself in place, as the PR text acknowledges ("arguably the deeper bug and deserves a ticket of its own"). Confirmed directly against crates/rocm-core/src/model_catalog.json: this is the only lemonade-preferred recipe lacking either an inline variant or a model_id_override supplying one — every other lemonade-preferred recipe is either a bare registry name (never reaches this checkpoint path) or already variant-pinned, so no other recipe is affected by leaving this alone.

Positive signals

  • The new guard was checked against every lemonade-preferred recipe in crates/rocm-core/src/model_catalog.json: each one either has no / in its id (a bare registry name such as Qwen3-4B-Instruct-2507-GGUF, which never reaches parse_hf_checkpoint) or already carries a :variant (inline in canonical_model_id, e.g. unsloth/Qwen3.6-35B-A3B-GGUF:Q4_K_M, or via model_id_override, e.g. the Qwen2.5-1.5B entry). The new refusal cannot regress any currently-servable catalog model.
  • The "ASCII only" property claimed in hf_checkpoint_variant_required's doc comment is structurally guaranteed, not merely asserted: the only variable fragment interpolated into the message is model_ref itself, which reaches this function only after parse_hf_checkpoint has validated every component against is_safe_hf_component's [A-Za-z0-9._-] charset — the invariant can't silently regress without that charset check itself changing too.
  • Routing both call sites (the pre-launch RPC in lib.rs::resolve_model_response and the in-process backstop in direct_llama.rs::serve_hf_checkpoint) through one shared helper is exactly the right fix for the wording-drift problem the PR describes, and no duplicated inline refusal string is left behind at either call site.

Deployment notes

  • User-visible behavior change for rocm serve <owner/repo> --engine lemonade with no :variant: this previously could install the Lemonade runtime and attempt to serve, succeeding only by accident on whatever GGUF happened to be cached (or failing invisibly on Windows); it now fails fast with an explicit refusal before any engine starts. A correctly-specified owner/repo:variant reference is unaffected.

What this covered

DEGRADED REVIEW — fan-out to independent workers was attempted four times (two in parallel, two sequential) and failed every time with a concurrency-limit error from the agent-spawning tool (shared across other review tasks running concurrently), not a lack of capability; retries were stopped per that error's own guidance. This is therefore a single-context pass rather than independent fan-out passes, and is marked degraded per the review method's own rule. Recommend a fan-out re-run once spawn capacity is available, particularly a fresh independent pass over tests/e2e-cucumber/tests/e2e/serving_steps.rs (3,564 lines total; only the new ~65-line step block and its closest neighbors got close reading here) and tests/e2e-cucumber/features/model_serving.feature (889 lines; same caveat).

Despite the degraded mode, read the full content of all five changed files at the PR head (base 691834a0…head b45ff510): engines/lemonade/src/direct_llama.rs, engines/lemonade/src/lib.rs, tests/e2e-cucumber/expectations.toml, tests/e2e-cucumber/features/model_serving.feature, tests/e2e-cucumber/tests/e2e/serving_steps.rs; plus surrounding/calling code needed to judge the change in place: engines/lemonade/src/state.rs, apps/rocm/src/serve_cmd.rs (serve_model_ref_for_engine, recipe_preferred_engine, the ResolveModel/Launch call sites), crates/rocm-core/src/model_catalog.rs and the live data in crates/rocm-core/src/model_catalog.json, and tests/e2e-cucumber/tests/feature_naming.rs's sequential-id rule. Checked CI: all 49 check runs at this head report success (two unrelated jobs report skipped); none failing. Checked git log/blame on the touched regions: all attributable to this branch's own commits. Ticket ROCMAI-863 was retrieved and its summary matches this PR's title and scope.

Did not run: a "prior changes" pass (no access here to historical review-comment threads on earlier changes to these files, only git log/git blame, which showed nothing of concern); a live local cargo test/cargo clippy run (relied on CI's green result and the PR's own stated local verification instead).
Not reconciled against any PR discussion or review thread; only the PR's own description and its named ticket were read, and only after the findings above were reached.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@r0x0r

r0x0r commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

PR body corrected — and it was stale in three ways, not one

The description states serve-42 is "the next free index"... At the PR's actual head, serve-42 is a pre-existing, unrelated scenario.

Confirmed and fixed. You were right that a reader following the description's own "Not verified" pointer landed on an unrelated WSL scenario. Re-reading the whole body against the current head rather than only patching the id turned up two more claims that #1044 invalidated, so all three are corrected:

1. The scenario id. Now serve-56 throughout. I kept one explicit mention of the old id, because "it was written as serve-42 and renumbered by merge d55aff3c" is more useful to a reviewer than a silent replacement that leaves earlier review threads referring to a number the body no longer contains.

2. The expectations.toml claim was backwards. The body said this change makes the qwen2.5 row's lemonade half stop printing a resolved name, because qwen2.5 expands to the safetensors-only repo and gets refused. That stopped being true when #1044 gave the recipe a per-engine ref: lemonade is handed Qwen/Qwen2.5-1.5B-Instruct-GGUF:Q4_K_M with the variant already pinned, so this change's guard never fires there. vLLM stopping before its serve plan is the whole of what keeps those rows red. I corrected the file itself earlier; the body had not caught up.

3. Your second "decision for the author" is now moot, and the body says so. You flagged that the catalog still lists preferred_engines: ["lemonade"] for a safetensors-only repo. #1044 fixed that at the source. So the guard here is no longer what rescues that model — it is the backstop for an owner/repo the catalog has no recipe for at all.

That last point had a consequence worth flagging, because it is the reason this PR's own scenario was briefly broken: serve-56 used to serve Qwen/Qwen2.5-1.5B-Instruct, and once the catalog supplied the variant the refusal correctly stopped firing — the Strix Halo Windows lane caught it serving and exiting 0 where the scenario expects a refusal. It now serves Qwen/Qwen2.5-3B-Instruct, which has no recipe, and the step asserts that up front:

assert!(
    !rocm_core::model_catalog::builtin_model_recipes()
        .iter()
        .any(|recipe| recipe.matches_ref(HF_REPO_WITHOUT_VARIANT)),
    "{HF_REPO_WITHOUT_VARIANT} now has a catalog recipe, which supplies the \
     quantization variant this scenario exists to see refused; pick an id \
     the catalog has no recipe for"
);

so a future catalog edit fails the scenario by name at the first step instead of silently turning it into a tautology.

On your first "decision for the author" — refusing at the engine's pre-launch ResolveModel rather than with a CLI-side grammar check — no change; as you note, the PR text already lays out both sides, and duplicating one engine's model-reference grammar in the CLI is the cost I did not want to pay.

Nothing in the diff changed with this push; it is description-only.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The guard sits in resolve_model_response behind one shared helper, the serve_hf_checkpoint backstop uses the same message, and the wording names no quantization it hasn't inspected. That matches the ticket's recommended design. The unit tests and serve-56 cover the case, and CI is green on this head, apart from the GPU lane that runs serve-56 itself.

Nothing blocking. A few small things, mostly questions, left inline:

  • The message prints the literal owner/repo:<variant> rather than the user's own ref.
  • The guard now applies to every caller of resolve_model_response, not just rocm serve. The ticket asks to confirm the service-reuse ResolveModel call in serve_cmd.rs reports the new error cleanly, and I couldn't see that evidenced.
  • Does "serving is refused before any engine starts" assert that no engine process or log was created, or only the exit code?
  • The expectations.toml comment narrates its own revision history.

This is a read of the diff plus CI results; I didn't build or run it locally.

checkpoint.variant.is_none().then(|| {
format!(
"`{model_ref}` needs an explicit GGUF variant to serve; use \
`owner/repo:<variant>` with a quantization listed in the repo's \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional: the message hard-codes the placeholder owner/repo:<variant> while the first clause interpolates the real ref. {model_ref}:<variant> would show exactly what to retype. The unit test and the serve-56 step currently pin the literal, so they would move with it.

// invisible there: the user gets an empty log, exit 0, and a service recorded as
// stopped, with nothing anywhere saying what was wrong. Answering here puts the
// reason on the user's terminal before any engine starts or any record is written.
if let Some(message) = crate::direct_llama::hf_checkpoint_variant_required(&request.model_ref) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This guard now applies to every caller of resolve_model_response, not only rocm serve's pre-launch call. Could you list the other call sites (the service-reuse ResolveModel in serve_cmd.rs, anything else) and confirm each surfaces this error cleanly? The ticket asks for that check explicitly.

@id:serve-missing-gguf-variant-refused @requires-gpu @requires-engine:lemonade
Scenario: serve-56 - Serving a Hugging Face repo with no quantization variant is refused
When the user serves a Hugging Face repo with no quantization variant on Lemonade
Then serving is refused before any engine starts

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does "serving is refused before any engine starts" assert that no engine process or log was created, or only a nonzero exit? The "no local server was recorded" step covers the record. The engine-not-started promise is the point of this ticket, so it deserves its own state assertion if this step only checks the exit code.

# plan's name line was never reached. vLLM is therefore the whole of what keeps
# these rows red.
#
# ROCMAI-863's pre-launch refusal does NOT reach these rows, though an earlier

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this comment narrates its own earlier revision ("an earlier revision of this comment said it did"). State only the current truth here and leave the history to the PR body.

@r0x0r
r0x0r added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit e2e5c5b Nov 13, 2026
56 of 57 checks passed
@r0x0r
r0x0r deleted the fix/rocmai-863-lemonade-missing-gguf-variant branch November 13, 2026 11:02
r0x0r added a commit that referenced this pull request Nov 13, 2026
PR #919 (e2e5c5b) added serve-56
`@id:serve-missing-gguf-variant-refused` as the last scenario in
model_serving.feature, colliding with this branch's own serve-56 and
serve-57. Cucumber scenario numbers must equal their position in the
file (enforced by tests/e2e-cucumber/tests/feature_naming.rs), so
main's already-merged serve-56 keeps its number and this branch's two
shift up:

  serve-56 serve-scripted-death-after-startup        -> serve-57
  serve-57 serve-scripted-external-stop-while-attached -> serve-58

The `@id:` tags are unchanged, so expectations.toml and the step
bindings are unaffected. References updated to match: the two
scenarios' cross-references in the feature file's own comments,
docs/testing.md's "Managed-engine crash vs. stop (ROCMAI-541)"
section, and the doc comment in
apps/rocm/src/service_launch/readiness.rs.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 13, 2026
PR #919 landed on main and took the next free scenario number in
tests/e2e-cucumber/features/model_serving.feature: its
@id:serve-missing-gguf-variant-refused scenario is serve-56. This branch
already had a serve-56 of its own,
@id:serve-accepts-an-apu-the-kernel-gives-no-compute-node, so both sides
claimed the same number.

feature_naming requires a scenario's number to equal its position in the
file, and main's scenario comes first, so main keeps serve-56 and this
branch's APU scenario moves to serve-57. Two lines changed for it:

  * the Scenario: line in model_serving.feature
  * the doc comment on unsupported_host_gpu_message in
    apps/rocm/src/therock/family.rs, which cites the scenario by number

The @id: tags are untouched on both scenarios -- expectations.toml and
the step bindings key on those, not on the numbers. docs/testing.md
refers only to serve-55, which did not move. Apart from those two lines
this branch's diff against main is identical to its diff against the
merge base, and main's side is taken whole.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
r0x0r added a commit that referenced this pull request Nov 17, 2026
PR #919 landed on main and took the next free scenario number in
tests/e2e-cucumber/features/model_serving.feature: its
@id:serve-missing-gguf-variant-refused scenario is serve-56. This branch
already had a serve-56 of its own,
@id:serve-accepts-an-apu-the-kernel-gives-no-compute-node, so both sides
claimed the same number.

feature_naming requires a scenario's number to equal its position in the
file, and main's scenario comes first, so main keeps serve-56 and this
branch's APU scenario moves to serve-57. Two lines changed for it:

  * the Scenario: line in model_serving.feature
  * the doc comment on unsupported_host_gpu_message in
    apps/rocm/src/therock/family.rs, which cites the scenario by number

The @id: tags are untouched on both scenarios -- expectations.toml and
the step bindings key on those, not on the numbers. docs/testing.md
refers only to serve-55, which did not move. Apart from those two lines
this branch's diff against main is identical to its diff against the
merge base, and main's side is taken whole.

Signed-off-by: Roman Sirokov <roman.sirokov@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants