Repository navigation
fix(vllm): require the endpoint key on every server route #974
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1348,6 +1348,25 @@ directory that only your user account can read. Set `ROCM_SERVE_API_KEY` rather | |
| passing `--api-key`, so the secret stays out of your shell history and the | ||
| process table. | ||
|
|
||
| With the vLLM engine, an endpoint bound to a public interface | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Non-blocking: a key-protected vLLM service already running before an upgrade keeps its old launch (no every-route middleware) until it is restarted —
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same point as the thread above, and the same answer: confirmed against |
||
| (`--allow-public-bind`), or served with `--require-api-key`, answers only | ||
| clients that present its API key as an `Authorization: Bearer` header, on every | ||
| route the server answers, not just the OpenAI `/v1` API — including vLLM's own | ||
| `/invocations`, `/tokenize` and `/metrics` — except `GET`/`HEAD /health` and | ||
| CORS preflights: the `/health` liveness check stays open. The AIM engine runs | ||
| the same vLLM server inside its container but does not install this check, so | ||
| a key-protected AIM endpoint has only vLLM's own `VLLM_API_KEY` check, which | ||
| leaves routes such as `/invocations`, `/tokenize` and `/metrics` answering | ||
| without the key. | ||
|
|
||
| Because `/metrics` is covered, `rocm dash` cannot read engine metrics from a | ||
| key-protected vLLM server: it leaves that server's KV-cache, queue, throughput | ||
| and latency figures empty and keeps showing a warning such as | ||
| `vllm scrape: 1 failed (last: transport: GET http://<host>:<port>/metrics: status 401 Unauthorized)`. | ||
|
|
||
| `rocm bench load` sends no API key, so it cannot benchmark a key-protected | ||
| endpoint on any engine. | ||
|
|
||
| `--temperature` (>= 0.0), `--top-p` (0.0-1.0), and `--max-tokens` (> 0) set | ||
| server-wide sampling defaults for the launched engine. They apply only to | ||
| `vllm` and `lemonade`; other engines reject them. For vLLM they are folded | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -98,8 +98,9 @@ pub(crate) fn drop_orphaned_endpoint_key_on_already_running( | |
| /// The one such case today: Windows managed Lemonade. Its server reads the | ||
| /// value-typed `LEMONADE_API_KEY` env var, but the Windows detached-spawn | ||
| /// primitive only carries path-valued env overrides, so the key never reaches it. | ||
| /// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds | ||
| /// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter | ||
| /// The vllm engine enforces auth wherever it runs (`VLLM_API_KEY` plus the | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "wherever it runs" conflicts with
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not changed. The sentence names the vllm engine, and that engine only runs on Linux and WSL:
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Non-blocking: "enforces auth wherever it runs" overstates against this change's own Linux/WSL-only scoping elsewhere (docs/testing.md's new text, and the
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as the thread above: left as is, because the vllm engine runs only on Linux and WSL ( |
||
| /// every-route middleware in `engines/vllm/src/endpoint_auth.rs`), and loopback | ||
| /// binds (`public_bind == false`) need no key — both pass. `is_windows` is a parameter | ||
| /// so both branches are unit-testable off-Windows. | ||
| pub(crate) fn ensure_public_bind_engine_supported( | ||
| engine: &str, | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1528,7 +1528,38 @@ On some GPU families `rocm install sdk` installs vLLM itself | |
| the `rocm engines install vllm` that follows then shows the block without the | ||
| warning, and nothing more needs doing. | ||
|
|
||
| ## 7. ComfyUI Verification | ||
| ## 7. vLLM Endpoint Key Coverage | ||
|
|
||
| Needs a Linux or WSL host where the vLLM engine can start. Check that an | ||
| endpoint key covers every route the vLLM server answers, not only `/v1`: | ||
|
|
||
| ```bash | ||
| export ROCM_SERVE_API_KEY=manual-test-key | ||
| rocm serve Qwen/Qwen3.5-0.8B --engine vllm --require-api-key --managed | ||
| PORT=<the port the deployment summary's endpoint shows> | ||
| curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://127.0.0.1:$PORT/invocations" \ | ||
| -H 'Content-Type: application/json' \ | ||
| -d '{"model":"Qwen/Qwen3.5-0.8B","messages":[{"role":"user","content":"hi"}]}' | ||
| curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://127.0.0.1:$PORT/tokenize" \ | ||
| -H 'Content-Type: application/json' -d '{"model":"Qwen/Qwen3.5-0.8B","prompt":"hi"}' | ||
| curl -s -o /dev/null -w '%{http_code}\n' "http://127.0.0.1:$PORT/health" | ||
| ``` | ||
|
|
||
| Expected result: | ||
|
|
||
| - The `/invocations` and `/tokenize` calls print `401`. | ||
| - Repeating them with `-H "Authorization: Bearer $ROCM_SERVE_API_KEY"` prints | ||
| `200`. | ||
| - The `/health` call prints `200` without the key. `GET`/`HEAD /health` and | ||
| CORS preflights are the only requests the server answers without it. | ||
|
|
||
| Stop the service afterwards (`rocm services stop <service-id> --yes`). The | ||
| automated scenario is the `@nightly` `serve-vllm-key-guards-every-route` in | ||
| [model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature), | ||
| which runs after merge in the nightly runs on Instinct GPUs (MI300X, and MI350P | ||
| on the shared GPU queue). | ||
|
|
||
| ## 8. ComfyUI Verification | ||
|
|
||
| ComfyUI is managed as an app surface. It should start a local web server and | ||
| show the URL to open: | ||
|
|
@@ -1607,7 +1638,7 @@ python scripts\comfyui_therock_gpu_test.py | |
| This test may download a small checkpoint and submit a cat image workflow | ||
| through the ComfyUI HTTP API. | ||
|
|
||
| ## 8. Optional Cloud Provider Key | ||
| ## 9. Optional Cloud Provider Key | ||
|
|
||
| Local ROCm use does not need a cloud provider key. If you want to test OpenAI or | ||
| Anthropic provider setup, save the key through stdin so it does not land in | ||
|
|
@@ -1632,7 +1663,7 @@ To remove the saved key: | |
| rocm config clear-provider-key openai | ||
| ``` | ||
|
|
||
| ## 9. Optional Provider-Assisted Planning | ||
| ## 10. Optional Provider-Assisted Planning | ||
|
|
||
| Most users should leave this off. To test ambiguity resolution with an already | ||
| running local provider service: | ||
|
|
@@ -1660,7 +1691,7 @@ To turn provider-assisted planning back off: | |
| rocm config clear-planner-provider | ||
| ``` | ||
|
|
||
| ## 10. Uninstall | ||
| ## 11. Uninstall | ||
|
|
||
| Run this last, on a machine installed with the release installer (not a | ||
| `cargo` build, whose binaries uninstall leaves alone): | ||
|
|
@@ -1736,7 +1767,7 @@ Expected result: | |
| either case. The same prompt is shared by `rocm fix` and the SDK-overwrite | ||
| prompt, so the same answers behave the same way there. | ||
|
|
||
| ## 11. AIM (Docker) Engine Verification | ||
| ## 12. AIM (Docker) Engine Verification | ||
|
|
||
| Linux/WSL2 only, and requires a reachable Docker daemon (`docker info` must | ||
| succeed). The catalog entry used below has profiles for Radeon PRO W7900 and | ||
|
|
@@ -1801,7 +1832,7 @@ Expected result: | |
| - On native Windows, the same command fails explicitly instead of falling | ||
| back to another engine. | ||
|
|
||
| ## 11. Background Automation Helper | ||
| ## 12. Background Automation Helper | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Non-blocking: this renumber (inserting the new section 7 and shifting everything after it by one) carries forward a pre-existing duplicate-number bug rather than fixing it. The base branch already had two |
||
|
|
||
| The helper (`rocm daemon`) runs only while an automation watcher is enabled. | ||
| Start from a fresh config (all watchers off) and use a scratch | ||
|
|
@@ -1832,7 +1863,7 @@ Expected result: | |
| seconds before the helper exits) still ends with a helper running: enable | ||
| waits for an exiting helper to finish, then starts a new one. | ||
|
|
||
| ## 12. Context-Size Estimate for a GGUF Model | ||
| ## 13. Context-Size Estimate for a GGUF Model | ||
|
|
||
| Needs a host with a Radeon GPU (a discrete card: an APU answers `undetermined` | ||
| by design) and network access to Hugging Face. The point is to check the | ||
|
|
@@ -1895,7 +1926,7 @@ Finally, `rocm diagnose --report --model <owner>/<repo>:Q4_K_M --context-size | |
| 8192` prints the report with no model section, and stderr says that `--model, | ||
| --context-size were ignored` and nothing about the model was fetched. | ||
|
|
||
| ## 13. Pasting An Error Into `rocm diagnose` | ||
| ## 14. Pasting An Error Into `rocm diagnose` | ||
|
|
||
| `rocm diagnose --symptom-file` takes a pasted error without shell quoting. | ||
| Copy a line that would break an argument, for example: | ||
|
|
@@ -1933,7 +1964,7 @@ Expected result: | |
| paste with a non-ASCII character in it (`é`, `→`, a typographic quote) | ||
| reports the same causes as in PowerShell 7. | ||
|
|
||
| ## 14. Which Causes a Pasted Error Contributed To | ||
| ## 15. Which Causes a Pasted Error Contributed To | ||
|
|
||
| `rocm diagnose --symptom` scores the catalog from the examined host's state and | ||
| the pasted text together, so a finding about the host can outrank the entry the | ||
|
|
@@ -1961,7 +1992,7 @@ HSA_OVERRIDE_GFX_VERSION=11.0.0 rocm diagnose --json | |
| The e2e scenarios `diagnose-62`, `diagnose-63` and `diagnose-64` run these three | ||
| checks on every lane. | ||
|
|
||
| ## 15. Dashboard Chat With A Small-Context Local Model | ||
| ## 16. Dashboard Chat With A Small-Context Local Model | ||
|
|
||
| Needs a local OpenAI-compatible endpoint serving a model with a small context | ||
| — llama.cpp's default `n_ctx` is 32768, which is the case this checks. The | ||
|
|
@@ -2003,7 +2034,7 @@ Expected result: | |
| context.` — and `rocm-cli.log` should still carry the provider's own body | ||
| with `exceed_context_size_error` in it. | ||
|
|
||
| ## 16. Serve Smoke Test Failure | ||
| ## 17. Serve Smoke Test Failure | ||
|
|
||
| Needs any engine `rocm serve` can start (the behaviour is not vLLM-specific) and | ||
| an interactive terminal: the smoke test and its summary run only there. The | ||
|
|
@@ -2047,7 +2078,7 @@ becomes ready: | |
| content is not a failure: the summary stays `ready` with the two metric rows | ||
| `n/a` and no smoke-test note. | ||
|
|
||
| ## 17. `rocm fix` When A Step Does Not Finish Or A Read Fails | ||
| ## 18. `rocm fix` When A Step Does Not Finish Or A Read Fails | ||
|
|
||
| The automated scenarios (`diagnose-15`, `diagnose-68` on bare-metal Linux; | ||
| `diagnose-39`, `diagnose-69`, `diagnose-70` on Windows) cover these; repeat | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider adding that a key-protected vLLM service that is already running keeps its old launch (without the every-route check) until it is restarted.
rocm servereuses it because the middleware is not part of the reuse match, so after upgrading, users should restart keyed vLLM services.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed that it's real:
managed_service_reuse_refusalinapps/rocm/src/service_launch/launch_transaction.rsmatches on the recipe,requires_api_keyand device policy, so a keyed vLLM service started before the upgrade is reused without the middleware until it is restarted. I've left the README unchanged in this PR so the approved diff stays as reviewed; the PR description carries it as an upgrade note ("Restart key-protected vLLM services after upgrading") for the release notes. The latest push (fb7fa46) only merges main and changes no behaviour.