Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -907,6 +907,60 @@ one model across multiple GPUs is not supported. Because selection uses the
`amd-smi` ordinal but is applied through `HIP_VISIBLE_DEVICES`, ROCm CLI warns
when `ROCR_VISIBLE_DEVICES` is set, because the two orderings can diverge.

#### Reuse is exempt from the no-GPU refusal

Under the default `gpu_required` policy, `rocm serve` on a host with no usable
AMD GPU fails fast with `no usable AMD GPU detected`, before any engine is
prepared or downloaded. A `rocm serve` that matches an **already-running
managed service for the same engine and model** now skips that refusal and
prints the reuse summary instead. This is a real change to a refusal you may be
relying on: on a GPU-less host, `rocm serve <model>` can now succeed where it
previously always failed.

The exemption is limited to that case, and nothing else about `gpu_required`
changes. The reused service was vetted against the policy at its own launch,
and the reusing invocation launches no process and does no GPU work. A live
service for a different engine does not soften the refusal, and neither does one
for a model whose name does not overlap — those still fail fast with the usual
message and prepare nothing.

Matching the engine and model is necessary but not sufficient. The exemption
applies only when the service would *actually* be reused, and that is decided by
exactly the predicate the launch itself applies: the recipe settings must match
(the sampling-control rule under [Model serving](#model-serving) above), a
`--require-api-key` request must be satisfiable by a service that already has a
key, and the service's recorded device policy must satisfy this one — a
`gpu_required` serve is not reused from a service recorded as `cpu_only`. That
is the only policy pair refused; any other combination, including a record that
predates the field and so names no policy at all, is accepted. (No `rocm serve`
can record a `cpu_only` service today — `--device cpu_only` is refused outright,
since CPU is not a fallback path here — so that term is a guard, not a refusal
you will meet.) A live service that fails any of those is not reusable, so it
buys no bypass: `rocm serve` refuses on the GPU instead, which is the accurate
message and leaves your running server alone. (Telling you to stop it and re-run
would not help: the re-run would have nothing left to reuse and would hit the
same GPU refusal.)

Two limits are worth stating exactly, because they are easy to read as stronger
than they are:

- **Model names are matched leniently.** The check that decides whether reuse is
even worth asking about uses the same two-way substring match the service
listings use, so a live `qwen3-8b-instruct` service opens it for
`rocm serve qwen`. On a GPU-less host that invocation then does engine
preparation — possibly printing `Preparing <engine> for GPU serving...` and
installing — *before* the refusal it ultimately gets, and if that preparation
fails you get its error rather than the GPU one. Leniency is deliberate:
tightening it would silently break legitimate reuse by short name. Only a
model name that does not overlap is guaranteed to refuse first with nothing
prepared.
- **The exemption does not apply with an explicit `--gpu <index>`.** It skips
the no-usable-GPU bail, not the pinned-index check that follows it — and that
check refuses *every* index on a host with no usable GPU, because there is no
visible ordinal for one to name. So `rocm serve <model> --gpu 0` is refused on
a GPU-less host whether or not a reusable service is running, and so is
`--gpu 1`, and so on. The exemption applies under the default `--gpu auto`.

Manage background servers started with `--managed`:

```
Expand Down
14 changes: 14 additions & 0 deletions apps/rocm/src/engine_runtime_select.rs
Original file line number Diff line number Diff line change
Expand Up @@ -572,6 +572,20 @@ fn service_model_reports_rocm_backend(model: &serde_json::Value) -> bool {
.is_some_and(|backend| backend.trim().to_ascii_lowercase().starts_with("rocm"))
}

/// Whether two model names refer to the same model, leniently.
///
/// Deliberately a bidirectional `contains` after suffix/case normalisation, so
/// the abbreviated names people actually type match the canonical ones the
/// engines report. The bound that buys: it admits any *overlapping* name, so a
/// live service for `qwen3-8b-instruct` satisfies a query for `qwen`. On the
/// serve reuse pre-gate that means the `ResolveModel` round-trip and any
/// self-managed engine install can run ahead of the no-usable-GPU bail for a
/// service this invocation cannot actually reuse.
///
/// Left lenient on purpose: tightening it would trade that bounded, visible
/// cost for false negatives that silently break legitimate reuse, which is the
/// worse failure. Recorded here rather than implied, since four call sites
/// share this one relation.
pub(crate) fn service_model_names_match(left: &str, right: &str) -> bool {
let left = left.trim();
let right = right.trim();
Expand Down
15 changes: 15 additions & 0 deletions apps/rocm/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -505,6 +505,21 @@ rocm model --verbose"
/// logs in this terminal instead. When streaming, press Ctrl-D to detach and leave the
/// server running, or Ctrl-C to stop it. Inspect or stop servers later with
/// `rocm services`.
///
/// Under --device gpu_required (the default) this command refuses on a host with no
/// usable AMD GPU. Reusing an already-running managed server is the one exception: it
/// launches nothing, so it is allowed through. The exception applies only when the
/// server would actually be reused: same engine and model, matching recipe settings,
/// a key already present if --require-api-key is asked for, and a device policy at
/// least as strong as this one. A live server for a different engine, or for a model
/// whose name does not overlap, does not soften the refusal, and nothing is prepared
/// before it. A same-model server this run turns out not to be able to reuse does not
/// soften it either, but engine preparation has already run by the time that is known.
/// Model names are matched leniently, so a live server whose name overlaps the one you
/// asked for (`qwen` for a running `qwen3-8b-instruct`) does let engine preparation run
/// before the refusal.
/// With an explicit --gpu the index is still validated after the exception, so a
/// pinned GPU that is not visible is refused even when a reusable server exists.
#[command(after_help = "EXAMPLES:\n \
rocm serve qwen\n \
rocm serve qwen --engine vllm --port 8000\n \
Expand Down
Loading
Loading