Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1348,6 +1348,25 @@ directory that only your user account can read. Set `ROCM_SERVE_API_KEY` rather
passing `--api-key`, so the secret stays out of your shell history and the
process table.

With the vLLM engine, an endpoint bound to a public interface

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding that a key-protected vLLM service that is already running keeps its old launch (without the every-route check) until it is restarted. rocm serve reuses it because the middleware is not part of the reuse match, so after upgrading, users should restart keyed vLLM services.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that it's real: managed_service_reuse_refusal in apps/rocm/src/service_launch/launch_transaction.rs matches on the recipe, requires_api_key and device policy, so a keyed vLLM service started before the upgrade is reused without the middleware until it is restarted. I've left the README unchanged in this PR so the approved diff stays as reviewed; the PR description carries it as an upgrade note ("Restart key-protected vLLM services after upgrading") for the release notes. The latest push (fb7fa46) only merges main and changes no behaviour.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: a key-protected vLLM service already running before an upgrade keeps its old launch (no every-route middleware) until it is restarted — rocm serve's reuse check (apps/rocm/src/service_launch/launch_transaction.rs:329, if require_api_key && !existing.requires_api_key) is a bare boolean "did the existing record ask for a key" test with no awareness of whether that process actually has this middleware. Nothing in README.md, docs/testing.md or docs/manual-testing.md tells an operator to restart a keyed vLLM service after upgrading — only the PR description's own "Upgrade note (not in the docs)" mentions it. Suggest adding one sentence here (or in the new "vLLM Endpoint Key Coverage" manual-testing section) telling operators to restart already-running key-protected vLLM services after upgrading.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same point as the thread above, and the same answer: confirmed against managed_service_reuse_refusal, kept out of this PR's docs so the approved diff is unchanged, and stated as an upgrade note in the PR description. fb7fa46 is a merge of main only.

(`--allow-public-bind`), or served with `--require-api-key`, answers only
clients that present its API key as an `Authorization: Bearer` header, on every
route the server answers, not just the OpenAI `/v1` API — including vLLM's own
`/invocations`, `/tokenize` and `/metrics` — except `GET`/`HEAD /health` and
CORS preflights: the `/health` liveness check stays open. The AIM engine runs
the same vLLM server inside its container but does not install this check, so
a key-protected AIM endpoint has only vLLM's own `VLLM_API_KEY` check, which
leaves routes such as `/invocations`, `/tokenize` and `/metrics` answering
without the key.

Because `/metrics` is covered, `rocm dash` cannot read engine metrics from a
key-protected vLLM server: it leaves that server's KV-cache, queue, throughput
and latency figures empty and keeps showing a warning such as
`vllm scrape: 1 failed (last: transport: GET http://<host>:<port>/metrics: status 401 Unauthorized)`.

`rocm bench load` sends no API key, so it cannot benchmark a key-protected
endpoint on any engine.

`--temperature` (>= 0.0), `--top-p` (0.0-1.0), and `--max-tokens` (> 0) set
server-wide sampling defaults for the launched engine. They apply only to
`vllm` and `lemonade`; other engines reject them. For vLLM they are folded
Expand Down
5 changes: 3 additions & 2 deletions apps/rocm/src/service_launch/endpoint_auth.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,9 @@ pub(crate) fn drop_orphaned_endpoint_key_on_already_running(
/// The one such case today: Windows managed Lemonade. Its server reads the
/// value-typed `LEMONADE_API_KEY` env var, but the Windows detached-spawn
/// primitive only carries path-valued env overrides, so the key never reaches it.
/// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds
/// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter
/// The vllm engine enforces auth wherever it runs (`VLLM_API_KEY` plus the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"wherever it runs" conflicts with docs/testing.md, which now says Linux and WSL. vLLM serving is Linux/WSL-only, so I'd say that here (this check exempts vllm on every platform, and the middleware tests are cfg(linux)).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed. The sentence names the vllm engine, and that engine only runs on Linux and WSL: engines/vllm/src/runtime.rs refuses native Windows, and docs/testing.md says "vLLM (Linux and WSL, ...)" in the same paragraph. "Wherever it runs" is true as written; this function exempts vllm on every platform because there is no platform where it starts without the key check. fb7fa46 (merge of main) leaves it as is.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: "enforces auth wherever it runs" overstates against this change's own Linux/WSL-only scoping elsewhere (docs/testing.md's new text, and the #[cfg_attr(not(target_os = "linux"), ignore)] comment on the middleware's Rust test). Not functionally wrong — vLLM is already rejected on native Windows elsewhere (engines/vllm/src/runtime.rs:176), so there's no real gap — but the wording is inconsistent with the rest of the PR. Suggest "wherever it runs (Linux and WSL)" to match.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the thread above: left as is, because the vllm engine runs only on Linux and WSL (engines/vllm/src/runtime.rs rejects native Windows), so "wherever it runs" already means that. No change in fb7fa46, which only merges main.

/// every-route middleware in `engines/vllm/src/endpoint_auth.rs`), and loopback
/// binds (`public_bind == false`) need no key — both pass. `is_windows` is a parameter
/// so both branches are unit-testable off-Windows.
pub(crate) fn ensure_public_bind_engine_supported(
engine: &str,
Expand Down
1 change: 1 addition & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -692,6 +692,7 @@ The vLLM engine adapter (`rocm-engine-vllm`): installs vLLM into a managed TheRo
| [`lib.rs`](../engines/vllm/src/lib.rs) | Crate root: CLI and envelope dispatch and the few items shared across the modules, including `verified_torchaudio_release` for the `rocm` binary, a `pub fn` here that delegates to the `pub(crate)` one in [`wheel_builds.rs`](../engines/vllm/src/wheel_builds.rs) (it is not a `pub use`). | flat |
| [`main.rs`](../engines/vllm/src/main.rs) | The `rocm-engine-vllm` binary's entry point; it only calls `rocm_engine_vllm::run_cli()`. | flat |
| [`discover_builds.rs`](../engines/vllm/src/discover_builds.rs) | The ROCm 10.x discovery table (`VLLM_ROCM_DISCOVER_BUILD_TABLE`, one row per supported `major.minor` line), its lookup and the series matcher. | domain |
| [`endpoint_auth.rs`](../engines/vllm/src/endpoint_auth.rs) | The endpoint-key middleware for a keyed launch: writes [`rocm_cli_vllm_auth.py`](../engines/vllm/src/rocm_cli_vllm_auth.py) (compiled in with `include_str!`) under the engine's data directory, prepends that directory to the server's `PYTHONPATH` and passes `--middleware rocm_cli_vllm_auth.RequireEndpointApiKey`, which requires the bearer key on every HTTP and WebSocket route except `GET`/`HEAD /health` and CORS preflights. `VLLM_API_KEY` alone covers only a fixed set of path prefixes. | domain |
| [`install.rs`](../engines/vllm/src/install.rs) | ROCm and vLLM install execution through `uv`, by the static-pin route or the ROCm 10.x discovery route, then a check that vLLM's `tensorizer` imports (upgrading it when it does not). | domain |
| [`process.rs`](../engines/vllm/src/process.rs) | The resolve-model, launch, serve and stop handlers: recipe validation, device-policy and GPU-index resolution, the vLLM server spawn and launch environment, and readiness polling. | domain |
| [`repair.rs`](../engines/vllm/src/repair.rs) | Whether an already-resolvable managed vLLM environment violates vLLM's pinned dependencies and must be reinstalled (`assess_runtime_repair`), treating the SDK's torch build, a user-chosen torch stack (torch, torchvision, torchaudio, triton) with alignment disabled, and the tensorizer upgrade as intended. | domain |
Expand Down
55 changes: 43 additions & 12 deletions docs/manual-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -1528,7 +1528,38 @@ On some GPU families `rocm install sdk` installs vLLM itself
the `rocm engines install vllm` that follows then shows the block without the
warning, and nothing more needs doing.

## 7. ComfyUI Verification
## 7. vLLM Endpoint Key Coverage

Needs a Linux or WSL host where the vLLM engine can start. Check that an
endpoint key covers every route the vLLM server answers, not only `/v1`:

```bash
export ROCM_SERVE_API_KEY=manual-test-key
rocm serve Qwen/Qwen3.5-0.8B --engine vllm --require-api-key --managed
PORT=<the port the deployment summary's endpoint shows>
curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://127.0.0.1:$PORT/invocations" \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen3.5-0.8B","messages":[{"role":"user","content":"hi"}]}'
curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://127.0.0.1:$PORT/tokenize" \
-H 'Content-Type: application/json' -d '{"model":"Qwen/Qwen3.5-0.8B","prompt":"hi"}'
curl -s -o /dev/null -w '%{http_code}\n' "http://127.0.0.1:$PORT/health"
```

Expected result:

- The `/invocations` and `/tokenize` calls print `401`.
- Repeating them with `-H "Authorization: Bearer $ROCM_SERVE_API_KEY"` prints
`200`.
- The `/health` call prints `200` without the key. `GET`/`HEAD /health` and
CORS preflights are the only requests the server answers without it.

Stop the service afterwards (`rocm services stop <service-id> --yes`). The
automated scenario is the `@nightly` `serve-vllm-key-guards-every-route` in
[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature),
which runs after merge in the nightly runs on Instinct GPUs (MI300X, and MI350P
on the shared GPU queue).

## 8. ComfyUI Verification

ComfyUI is managed as an app surface. It should start a local web server and
show the URL to open:
Expand Down Expand Up @@ -1607,7 +1638,7 @@ python scripts\comfyui_therock_gpu_test.py
This test may download a small checkpoint and submit a cat image workflow
through the ComfyUI HTTP API.

## 8. Optional Cloud Provider Key
## 9. Optional Cloud Provider Key

Local ROCm use does not need a cloud provider key. If you want to test OpenAI or
Anthropic provider setup, save the key through stdin so it does not land in
Expand All @@ -1632,7 +1663,7 @@ To remove the saved key:
rocm config clear-provider-key openai
```

## 9. Optional Provider-Assisted Planning
## 10. Optional Provider-Assisted Planning

Most users should leave this off. To test ambiguity resolution with an already
running local provider service:
Expand Down Expand Up @@ -1660,7 +1691,7 @@ To turn provider-assisted planning back off:
rocm config clear-planner-provider
```

## 10. Uninstall
## 11. Uninstall

Run this last, on a machine installed with the release installer (not a
`cargo` build, whose binaries uninstall leaves alone):
Expand Down Expand Up @@ -1736,7 +1767,7 @@ Expected result:
either case. The same prompt is shared by `rocm fix` and the SDK-overwrite
prompt, so the same answers behave the same way there.

## 11. AIM (Docker) Engine Verification
## 12. AIM (Docker) Engine Verification

Linux/WSL2 only, and requires a reachable Docker daemon (`docker info` must
succeed). The catalog entry used below has profiles for Radeon PRO W7900 and
Expand Down Expand Up @@ -1801,7 +1832,7 @@ Expected result:
- On native Windows, the same command fails explicitly instead of falling
back to another engine.

## 11. Background Automation Helper
## 12. Background Automation Helper

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this renumber (inserting the new section 7 and shifting everything after it by one) carries forward a pre-existing duplicate-number bug rather than fixing it. The base branch already had two ## 11. headings ("AIM (Docker) Engine Verification" and "Background Automation Helper"); after this diff they are both ## 12. (line 1770 and this line, 1835). Since the diff directly edits both of these exact headings to renumber them, this would be a one-line fix to take while already touching the lines: bump "Background Automation Helper" (and everything after it) to ## 13. onward.


The helper (`rocm daemon`) runs only while an automation watcher is enabled.
Start from a fresh config (all watchers off) and use a scratch
Expand Down Expand Up @@ -1832,7 +1863,7 @@ Expected result:
seconds before the helper exits) still ends with a helper running: enable
waits for an exiting helper to finish, then starts a new one.

## 12. Context-Size Estimate for a GGUF Model
## 13. Context-Size Estimate for a GGUF Model

Needs a host with a Radeon GPU (a discrete card: an APU answers `undetermined`
by design) and network access to Hugging Face. The point is to check the
Expand Down Expand Up @@ -1895,7 +1926,7 @@ Finally, `rocm diagnose --report --model <owner>/<repo>:Q4_K_M --context-size
8192` prints the report with no model section, and stderr says that `--model,
--context-size were ignored` and nothing about the model was fetched.

## 13. Pasting An Error Into `rocm diagnose`
## 14. Pasting An Error Into `rocm diagnose`

`rocm diagnose --symptom-file` takes a pasted error without shell quoting.
Copy a line that would break an argument, for example:
Expand Down Expand Up @@ -1933,7 +1964,7 @@ Expected result:
paste with a non-ASCII character in it (`é`, `→`, a typographic quote)
reports the same causes as in PowerShell 7.

## 14. Which Causes a Pasted Error Contributed To
## 15. Which Causes a Pasted Error Contributed To

`rocm diagnose --symptom` scores the catalog from the examined host's state and
the pasted text together, so a finding about the host can outrank the entry the
Expand Down Expand Up @@ -1961,7 +1992,7 @@ HSA_OVERRIDE_GFX_VERSION=11.0.0 rocm diagnose --json
The e2e scenarios `diagnose-62`, `diagnose-63` and `diagnose-64` run these three
checks on every lane.

## 15. Dashboard Chat With A Small-Context Local Model
## 16. Dashboard Chat With A Small-Context Local Model

Needs a local OpenAI-compatible endpoint serving a model with a small context
— llama.cpp's default `n_ctx` is 32768, which is the case this checks. The
Expand Down Expand Up @@ -2003,7 +2034,7 @@ Expected result:
context.` — and `rocm-cli.log` should still carry the provider's own body
with `exceed_context_size_error` in it.

## 16. Serve Smoke Test Failure
## 17. Serve Smoke Test Failure

Needs any engine `rocm serve` can start (the behaviour is not vLLM-specific) and
an interactive terminal: the smoke test and its summary run only there. The
Expand Down Expand Up @@ -2047,7 +2078,7 @@ becomes ready:
content is not a failure: the summary stays `ready` with the two metric rows
`n/a` and no smoke-test note.

## 17. `rocm fix` When A Step Does Not Finish Or A Read Fails
## 18. `rocm fix` When A Step Does Not Finish Or A Read Fails

The automated scenarios (`diagnose-15`, `diagnose-68` on bare-metal Linux;
`diagnose-39`, `diagnose-69`, `diagnose-70` on Windows) cover these; repeat
Expand Down
73 changes: 65 additions & 8 deletions docs/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -2497,9 +2497,11 @@ audit log. Only the serving user can read the key file: it is mode 0600 on Unix
and has a protected DACL granting only that user on Windows, and every store
replaces it with a freshly created owner-only file, so permissions loosened in
place do not survive a re-store. Verifying that the running server actually
*rejects* an unauthenticated request requires a live engine; that end-to-end
assertion is a deferred follow-up and is **not yet** exercised by the GPU
acceptance scripts (`scripts/vllm_therock_gpu_test.py`). The unit tests cover
*rejects* an unauthenticated request requires a live engine. For vLLM that is
the `@nightly` GPU scenario `serve-vllm-key-guards-every-route` (serve-61),
which runs in the nightly runs on Instinct GPUs (see "vLLM endpoint-key
coverage" below). For the other engines, AIM included, no test yet asserts that
the running server refuses a keyless request. The unit tests cover
key policy, generation, the owner-only key file (including one whose
permissions were loosened before a re-store, and on the Windows build job the
DACL read back from disk), and key-file resolution. The tests that fail
Expand Down Expand Up @@ -2637,8 +2639,9 @@ via `spawn_hidden_console_with_log`, whose env-override API is path-valued only,
so it cannot receive the value-typed `LEMONADE_API_KEY` that Lemonade's server
reads. Rather than launch an unauthenticated public server, `rocm serve` **fails
closed** on that exact combination (public bind + `--engine lemonade` + Windows)
and directs the user to `--engine vllm` or a loopback host. vLLM (all platforms,
via `VLLM_API_KEY`), direct llama-server serving (via `--api-key-file`),
and directs the user to `--engine vllm` or a loopback host. vLLM (Linux and WSL,
via `VLLM_API_KEY` plus the `rocm_cli_vllm_auth` middleware described below),
direct llama-server serving (via `--api-key-file`),
and Lemonade on non-Windows are all fully supported. Teaching the Windows
detached-spawn primitive to carry a value-typed override (so Windows Lemonade can
serve public with auth) is tracked as a separate follow-up.
Expand All @@ -2663,6 +2666,60 @@ lemonade` against the shared tree) reaches the restore only on a runner whose
shared tree still holds a stale pin; on a healthy runner the install succeeds and
the restore never runs, so a green lane does not show that it works.

vLLM endpoint-key coverage: `VLLM_API_KEY` on its own makes vLLM check the key
only under a fixed set of path prefixes (`/v1` and a few others), while the
server also answers routes such as `/invocations`, `/tokenize` and `/metrics`.
So a keyed vLLM launch also passes `--middleware
rocm_cli_vllm_auth.RequireEndpointApiKey`, an ASGI middleware compiled into the
engine adapter
([rocm_cli_vllm_auth.py](../engines/vllm/src/rocm_cli_vllm_auth.py)), written
under the engine's data directory at launch and put on the server's
`PYTHONPATH`. It
requires the key on every route except `GET`/`HEAD /health` and CORS
preflights, and raises instead of constructing when the key is empty, so a
server that loaded it never answers a request unchecked.

```bash
cargo test -p rocm-engine-vllm endpoint_auth
cargo test -p rocm-engine-vllm vllm_serve_args
```

The `endpoint_auth` tests run the shipped middleware under `python3` with a
standard-library ASGI harness (they fail, rather than skip, when `python3` is
missing on Linux), so the per-route behavior is covered on every PR without a
GPU. The `vllm_server_command` tests assert, on every PR, that a keyed launch
gets the `--middleware` argument, `VLLM_API_KEY`, and the middleware directory
at the front of `PYTHONPATH` ahead of the runtime's own entry, together, and
that an unkeyed launch gets none of them:

```bash
cargo test -p rocm-engine-vllm vllm_server_command
```

What no unit test covers is the call in `spawn_vllm_server` that hands the
resolved endpoint key to `vllm_server_command`: that spawn path probes the host
(OpenMPI, libatomic, libnuma) before it builds the command, so only a real
launch exercises it.

That a real vLLM server loads the middleware and refuses keyless calls is
covered by the `@nightly` GPU scenario `serve-vllm-key-guards-every-route`
([model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature)),
which runs in the scheduled nightly runs on Instinct GPUs, where vLLM is the
serving engine: the MI300X job in [nightly.yml](../.github/workflows/nightly.yml)
and the shared GPU queue's Instinct lanes, MI350P among them
([nightly-on-the-queue.md](nightly-on-the-queue.md)). Those run after merge, not
per PR. It is not tagged `@merge-queue`:
the merge queue runs no lane where vLLM is the engine (the MI300X lane does not
run on `merge_group`), so under that tag it would run nowhere. To run it on a
branch before merge, dispatch `e2e-selfhosted.yml` on the MI300X lane with a
`name_filter`, which bypasses the harness's tag gates (`@nightly` included), so
the scenario runs without `include_nightly`:

```bash
gh workflow run e2e-selfhosted.yml --ref <branch> \
-f platform=app-dev-gpu -f name_filter='^serve-61 '
```

Lemonade backend selection and alignment focused tests (reading the
`llamacpp` rows out of `lemonade backends` when other recipes have indented
rows of their own, and the second alignment attempt choosing the newest
Expand Down Expand Up @@ -5231,9 +5288,9 @@ hold the behaviour end to end, GPU or not:
runs only the bash half, on Linux, and ci.yml's Windows lane does not run
this scenario.
- `diagnose-60` (Linux and WSL lanes only, like every pseudo-terminal
scenario here; the Windows key is checked by hand in
[manual-testing.md](manual-testing.md) section 13) runs
`rocm diagnose --symptom-file -` on a pseudo-terminal tall enough to keep
scenario here; the Windows key is checked by hand in the "Pasting An Error
Into `rocm diagnose`" section of [manual-testing.md](manual-testing.md))
runs `rocm diagnose --symptom-file -` on a pseudo-terminal tall enough to keep
the whole exchange on screen (it has no scrollback), requires the hint
telling the user how to end the paste, then types the paste and Ctrl-D, the
Linux end-of-input key the hint names, and requires the screen, which also
Expand Down
Loading
Loading