diff --git a/README.md b/README.md index 9719b59b1..1af112fe7 100644 --- a/README.md +++ b/README.md @@ -1348,6 +1348,25 @@ directory that only your user account can read. Set `ROCM_SERVE_API_KEY` rather passing `--api-key`, so the secret stays out of your shell history and the process table. +With the vLLM engine, an endpoint bound to a public interface +(`--allow-public-bind`), or served with `--require-api-key`, answers only +clients that present its API key as an `Authorization: Bearer` header, on every +route the server answers, not just the OpenAI `/v1` API — including vLLM's own +`/invocations`, `/tokenize` and `/metrics` — except `GET`/`HEAD /health` and +CORS preflights: the `/health` liveness check stays open. The AIM engine runs +the same vLLM server inside its container but does not install this check, so +a key-protected AIM endpoint has only vLLM's own `VLLM_API_KEY` check, which +leaves routes such as `/invocations`, `/tokenize` and `/metrics` answering +without the key. + +Because `/metrics` is covered, `rocm dash` cannot read engine metrics from a +key-protected vLLM server: it leaves that server's KV-cache, queue, throughput +and latency figures empty and keeps showing a warning such as +`vllm scrape: 1 failed (last: transport: GET http://:/metrics: status 401 Unauthorized)`. + +`rocm bench load` sends no API key, so it cannot benchmark a key-protected +endpoint on any engine. + `--temperature` (>= 0.0), `--top-p` (0.0-1.0), and `--max-tokens` (> 0) set server-wide sampling defaults for the launched engine. They apply only to `vllm` and `lemonade`; other engines reject them. For vLLM they are folded diff --git a/apps/rocm/src/service_launch/endpoint_auth.rs b/apps/rocm/src/service_launch/endpoint_auth.rs index 6a647314b..2de5247a0 100644 --- a/apps/rocm/src/service_launch/endpoint_auth.rs +++ b/apps/rocm/src/service_launch/endpoint_auth.rs @@ -98,8 +98,9 @@ pub(crate) fn drop_orphaned_endpoint_key_on_already_running( /// The one such case today: Windows managed Lemonade. Its server reads the /// value-typed `LEMONADE_API_KEY` env var, but the Windows detached-spawn /// primitive only carries path-valued env overrides, so the key never reaches it. -/// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds -/// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter +/// The vllm engine enforces auth wherever it runs (`VLLM_API_KEY` plus the +/// every-route middleware in `engines/vllm/src/endpoint_auth.rs`), and loopback +/// binds (`public_bind == false`) need no key — both pass. `is_windows` is a parameter /// so both branches are unit-testable off-Windows. pub(crate) fn ensure_public_bind_engine_supported( engine: &str, diff --git a/docs/architecture.md b/docs/architecture.md index b929c3165..aacea5ec1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -692,6 +692,7 @@ The vLLM engine adapter (`rocm-engine-vllm`): installs vLLM into a managed TheRo | [`lib.rs`](../engines/vllm/src/lib.rs) | Crate root: CLI and envelope dispatch and the few items shared across the modules, including `verified_torchaudio_release` for the `rocm` binary, a `pub fn` here that delegates to the `pub(crate)` one in [`wheel_builds.rs`](../engines/vllm/src/wheel_builds.rs) (it is not a `pub use`). | flat | | [`main.rs`](../engines/vllm/src/main.rs) | The `rocm-engine-vllm` binary's entry point; it only calls `rocm_engine_vllm::run_cli()`. | flat | | [`discover_builds.rs`](../engines/vllm/src/discover_builds.rs) | The ROCm 10.x discovery table (`VLLM_ROCM_DISCOVER_BUILD_TABLE`, one row per supported `major.minor` line), its lookup and the series matcher. | domain | +| [`endpoint_auth.rs`](../engines/vllm/src/endpoint_auth.rs) | The endpoint-key middleware for a keyed launch: writes [`rocm_cli_vllm_auth.py`](../engines/vllm/src/rocm_cli_vllm_auth.py) (compiled in with `include_str!`) under the engine's data directory, prepends that directory to the server's `PYTHONPATH` and passes `--middleware rocm_cli_vllm_auth.RequireEndpointApiKey`, which requires the bearer key on every HTTP and WebSocket route except `GET`/`HEAD /health` and CORS preflights. `VLLM_API_KEY` alone covers only a fixed set of path prefixes. | domain | | [`install.rs`](../engines/vllm/src/install.rs) | ROCm and vLLM install execution through `uv`, by the static-pin route or the ROCm 10.x discovery route, then a check that vLLM's `tensorizer` imports (upgrading it when it does not). | domain | | [`process.rs`](../engines/vllm/src/process.rs) | The resolve-model, launch, serve and stop handlers: recipe validation, device-policy and GPU-index resolution, the vLLM server spawn and launch environment, and readiness polling. | domain | | [`repair.rs`](../engines/vllm/src/repair.rs) | Whether an already-resolvable managed vLLM environment violates vLLM's pinned dependencies and must be reinstalled (`assess_runtime_repair`), treating the SDK's torch build, a user-chosen torch stack (torch, torchvision, torchaudio, triton) with alignment disabled, and the tensorizer upgrade as intended. | domain | diff --git a/docs/manual-testing.md b/docs/manual-testing.md index d425d7cd0..4feab8755 100644 --- a/docs/manual-testing.md +++ b/docs/manual-testing.md @@ -1528,7 +1528,38 @@ On some GPU families `rocm install sdk` installs vLLM itself the `rocm engines install vllm` that follows then shows the block without the warning, and nothing more needs doing. -## 7. ComfyUI Verification +## 7. vLLM Endpoint Key Coverage + +Needs a Linux or WSL host where the vLLM engine can start. Check that an +endpoint key covers every route the vLLM server answers, not only `/v1`: + +```bash +export ROCM_SERVE_API_KEY=manual-test-key +rocm serve Qwen/Qwen3.5-0.8B --engine vllm --require-api-key --managed +PORT= +curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://127.0.0.1:$PORT/invocations" \ + -H 'Content-Type: application/json' \ + -d '{"model":"Qwen/Qwen3.5-0.8B","messages":[{"role":"user","content":"hi"}]}' +curl -s -o /dev/null -w '%{http_code}\n' -X POST "http://127.0.0.1:$PORT/tokenize" \ + -H 'Content-Type: application/json' -d '{"model":"Qwen/Qwen3.5-0.8B","prompt":"hi"}' +curl -s -o /dev/null -w '%{http_code}\n' "http://127.0.0.1:$PORT/health" +``` + +Expected result: + +- The `/invocations` and `/tokenize` calls print `401`. +- Repeating them with `-H "Authorization: Bearer $ROCM_SERVE_API_KEY"` prints + `200`. +- The `/health` call prints `200` without the key. `GET`/`HEAD /health` and + CORS preflights are the only requests the server answers without it. + +Stop the service afterwards (`rocm services stop --yes`). The +automated scenario is the `@nightly` `serve-vllm-key-guards-every-route` in +[model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature), +which runs after merge in the nightly runs on Instinct GPUs (MI300X, and MI350P +on the shared GPU queue). + +## 8. ComfyUI Verification ComfyUI is managed as an app surface. It should start a local web server and show the URL to open: @@ -1607,7 +1638,7 @@ python scripts\comfyui_therock_gpu_test.py This test may download a small checkpoint and submit a cat image workflow through the ComfyUI HTTP API. -## 8. Optional Cloud Provider Key +## 9. Optional Cloud Provider Key Local ROCm use does not need a cloud provider key. If you want to test OpenAI or Anthropic provider setup, save the key through stdin so it does not land in @@ -1632,7 +1663,7 @@ To remove the saved key: rocm config clear-provider-key openai ``` -## 9. Optional Provider-Assisted Planning +## 10. Optional Provider-Assisted Planning Most users should leave this off. To test ambiguity resolution with an already running local provider service: @@ -1660,7 +1691,7 @@ To turn provider-assisted planning back off: rocm config clear-planner-provider ``` -## 10. Uninstall +## 11. Uninstall Run this last, on a machine installed with the release installer (not a `cargo` build, whose binaries uninstall leaves alone): @@ -1736,7 +1767,7 @@ Expected result: either case. The same prompt is shared by `rocm fix` and the SDK-overwrite prompt, so the same answers behave the same way there. -## 11. AIM (Docker) Engine Verification +## 12. AIM (Docker) Engine Verification Linux/WSL2 only, and requires a reachable Docker daemon (`docker info` must succeed). The catalog entry used below has profiles for Radeon PRO W7900 and @@ -1801,7 +1832,7 @@ Expected result: - On native Windows, the same command fails explicitly instead of falling back to another engine. -## 11. Background Automation Helper +## 12. Background Automation Helper The helper (`rocm daemon`) runs only while an automation watcher is enabled. Start from a fresh config (all watchers off) and use a scratch @@ -1832,7 +1863,7 @@ Expected result: seconds before the helper exits) still ends with a helper running: enable waits for an exiting helper to finish, then starts a new one. -## 12. Context-Size Estimate for a GGUF Model +## 13. Context-Size Estimate for a GGUF Model Needs a host with a Radeon GPU (a discrete card: an APU answers `undetermined` by design) and network access to Hugging Face. The point is to check the @@ -1895,7 +1926,7 @@ Finally, `rocm diagnose --report --model /:Q4_K_M --context-size 8192` prints the report with no model section, and stderr says that `--model, --context-size were ignored` and nothing about the model was fetched. -## 13. Pasting An Error Into `rocm diagnose` +## 14. Pasting An Error Into `rocm diagnose` `rocm diagnose --symptom-file` takes a pasted error without shell quoting. Copy a line that would break an argument, for example: @@ -1933,7 +1964,7 @@ Expected result: paste with a non-ASCII character in it (`é`, `→`, a typographic quote) reports the same causes as in PowerShell 7. -## 14. Which Causes a Pasted Error Contributed To +## 15. Which Causes a Pasted Error Contributed To `rocm diagnose --symptom` scores the catalog from the examined host's state and the pasted text together, so a finding about the host can outrank the entry the @@ -1961,7 +1992,7 @@ HSA_OVERRIDE_GFX_VERSION=11.0.0 rocm diagnose --json The e2e scenarios `diagnose-62`, `diagnose-63` and `diagnose-64` run these three checks on every lane. -## 15. Dashboard Chat With A Small-Context Local Model +## 16. Dashboard Chat With A Small-Context Local Model Needs a local OpenAI-compatible endpoint serving a model with a small context — llama.cpp's default `n_ctx` is 32768, which is the case this checks. The @@ -2003,7 +2034,7 @@ Expected result: context.` — and `rocm-cli.log` should still carry the provider's own body with `exceed_context_size_error` in it. -## 16. Serve Smoke Test Failure +## 17. Serve Smoke Test Failure Needs any engine `rocm serve` can start (the behaviour is not vLLM-specific) and an interactive terminal: the smoke test and its summary run only there. The @@ -2047,7 +2078,7 @@ becomes ready: content is not a failure: the summary stays `ready` with the two metric rows `n/a` and no smoke-test note. -## 17. `rocm fix` When A Step Does Not Finish Or A Read Fails +## 18. `rocm fix` When A Step Does Not Finish Or A Read Fails The automated scenarios (`diagnose-15`, `diagnose-68` on bare-metal Linux; `diagnose-39`, `diagnose-69`, `diagnose-70` on Windows) cover these; repeat diff --git a/docs/testing.md b/docs/testing.md index f2e113962..ded11dd15 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -2497,9 +2497,11 @@ audit log. Only the serving user can read the key file: it is mode 0600 on Unix and has a protected DACL granting only that user on Windows, and every store replaces it with a freshly created owner-only file, so permissions loosened in place do not survive a re-store. Verifying that the running server actually -*rejects* an unauthenticated request requires a live engine; that end-to-end -assertion is a deferred follow-up and is **not yet** exercised by the GPU -acceptance scripts (`scripts/vllm_therock_gpu_test.py`). The unit tests cover +*rejects* an unauthenticated request requires a live engine. For vLLM that is +the `@nightly` GPU scenario `serve-vllm-key-guards-every-route` (serve-61), +which runs in the nightly runs on Instinct GPUs (see "vLLM endpoint-key +coverage" below). For the other engines, AIM included, no test yet asserts that +the running server refuses a keyless request. The unit tests cover key policy, generation, the owner-only key file (including one whose permissions were loosened before a re-store, and on the Windows build job the DACL read back from disk), and key-file resolution. The tests that fail @@ -2637,8 +2639,9 @@ via `spawn_hidden_console_with_log`, whose env-override API is path-valued only, so it cannot receive the value-typed `LEMONADE_API_KEY` that Lemonade's server reads. Rather than launch an unauthenticated public server, `rocm serve` **fails closed** on that exact combination (public bind + `--engine lemonade` + Windows) -and directs the user to `--engine vllm` or a loopback host. vLLM (all platforms, -via `VLLM_API_KEY`), direct llama-server serving (via `--api-key-file`), +and directs the user to `--engine vllm` or a loopback host. vLLM (Linux and WSL, +via `VLLM_API_KEY` plus the `rocm_cli_vllm_auth` middleware described below), +direct llama-server serving (via `--api-key-file`), and Lemonade on non-Windows are all fully supported. Teaching the Windows detached-spawn primitive to carry a value-typed override (so Windows Lemonade can serve public with auth) is tracked as a separate follow-up. @@ -2663,6 +2666,60 @@ lemonade` against the shared tree) reaches the restore only on a runner whose shared tree still holds a stale pin; on a healthy runner the install succeeds and the restore never runs, so a green lane does not show that it works. +vLLM endpoint-key coverage: `VLLM_API_KEY` on its own makes vLLM check the key +only under a fixed set of path prefixes (`/v1` and a few others), while the +server also answers routes such as `/invocations`, `/tokenize` and `/metrics`. +So a keyed vLLM launch also passes `--middleware +rocm_cli_vllm_auth.RequireEndpointApiKey`, an ASGI middleware compiled into the +engine adapter +([rocm_cli_vllm_auth.py](../engines/vllm/src/rocm_cli_vllm_auth.py)), written +under the engine's data directory at launch and put on the server's +`PYTHONPATH`. It +requires the key on every route except `GET`/`HEAD /health` and CORS +preflights, and raises instead of constructing when the key is empty, so a +server that loaded it never answers a request unchecked. + +```bash +cargo test -p rocm-engine-vllm endpoint_auth +cargo test -p rocm-engine-vllm vllm_serve_args +``` + +The `endpoint_auth` tests run the shipped middleware under `python3` with a +standard-library ASGI harness (they fail, rather than skip, when `python3` is +missing on Linux), so the per-route behavior is covered on every PR without a +GPU. The `vllm_server_command` tests assert, on every PR, that a keyed launch +gets the `--middleware` argument, `VLLM_API_KEY`, and the middleware directory +at the front of `PYTHONPATH` ahead of the runtime's own entry, together, and +that an unkeyed launch gets none of them: + +```bash +cargo test -p rocm-engine-vllm vllm_server_command +``` + +What no unit test covers is the call in `spawn_vllm_server` that hands the +resolved endpoint key to `vllm_server_command`: that spawn path probes the host +(OpenMPI, libatomic, libnuma) before it builds the command, so only a real +launch exercises it. + +That a real vLLM server loads the middleware and refuses keyless calls is +covered by the `@nightly` GPU scenario `serve-vllm-key-guards-every-route` +([model_serving.feature](../tests/e2e-cucumber/features/model_serving.feature)), +which runs in the scheduled nightly runs on Instinct GPUs, where vLLM is the +serving engine: the MI300X job in [nightly.yml](../.github/workflows/nightly.yml) +and the shared GPU queue's Instinct lanes, MI350P among them +([nightly-on-the-queue.md](nightly-on-the-queue.md)). Those run after merge, not +per PR. It is not tagged `@merge-queue`: +the merge queue runs no lane where vLLM is the engine (the MI300X lane does not +run on `merge_group`), so under that tag it would run nowhere. To run it on a +branch before merge, dispatch `e2e-selfhosted.yml` on the MI300X lane with a +`name_filter`, which bypasses the harness's tag gates (`@nightly` included), so +the scenario runs without `include_nightly`: + +```bash +gh workflow run e2e-selfhosted.yml --ref \ + -f platform=app-dev-gpu -f name_filter='^serve-61 ' +``` + Lemonade backend selection and alignment focused tests (reading the `llamacpp` rows out of `lemonade backends` when other recipes have indented rows of their own, and the second alignment attempt choosing the newest @@ -5231,9 +5288,9 @@ hold the behaviour end to end, GPU or not: runs only the bash half, on Linux, and ci.yml's Windows lane does not run this scenario. - `diagnose-60` (Linux and WSL lanes only, like every pseudo-terminal - scenario here; the Windows key is checked by hand in - [manual-testing.md](manual-testing.md) section 13) runs - `rocm diagnose --symptom-file -` on a pseudo-terminal tall enough to keep + scenario here; the Windows key is checked by hand in the "Pasting An Error + Into `rocm diagnose`" section of [manual-testing.md](manual-testing.md)) + runs `rocm diagnose --symptom-file -` on a pseudo-terminal tall enough to keep the whole exchange on screen (it has no scrollback), requires the hint telling the user how to end the paste, then types the paste and Ctrl-D, the Linux end-of-input key the hint names, and requires the screen, which also diff --git a/engines/vllm/src/endpoint_auth.rs b/engines/vllm/src/endpoint_auth.rs new file mode 100644 index 000000000..6fd0a7635 --- /dev/null +++ b/engines/vllm/src/endpoint_auth.rs @@ -0,0 +1,433 @@ +// Copyright © Advanced Micro Devices, Inc., or its affiliates. +// +// SPDX-License-Identifier: MIT + +//! Endpoint API-key enforcement for the vLLM server. +//! +//! Setting `VLLM_API_KEY` alone does not protect a vLLM endpoint: vLLM's own +//! check guards only a fixed set of path prefixes (`/v1` and a few others), +//! while the server also registers inference-capable routes outside them — +//! `/invocations`, `/tokenize`, `/detokenize` and more. So whenever an endpoint +//! key is in force, the server is additionally started with +//! `--middleware rocm_cli_vllm_auth.RequireEndpointApiKey`: a small ASGI +//! middleware, shipped inside this binary and written next to the engine's +//! data at launch, that requires the key on every route except +//! `GET`/`HEAD /health` and CORS preflights. +//! +//! vLLM imports the middleware by module path, so the directory it is written +//! to is prepended to the server's `PYTHONPATH`. A failed import aborts vLLM's +//! startup, so a launch that asked for a key fails closed rather than coming up +//! with only the partial check. + +use anyhow::{Context, Result}; +use std::ffi::OsString; +use std::fs; +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::process::Command as ProcessCommand; + +/// Python module name the middleware is imported as. +const MODULE_NAME: &str = "rocm_cli_vllm_auth"; + +/// Source of the middleware module, compiled into the binary so the copy vLLM +/// imports always matches the adapter that launched it. +const MODULE_SOURCE: &str = include_str!("rocm_cli_vllm_auth.py"); + +/// The `--middleware` import path vLLM is given. +pub(crate) const MIDDLEWARE_IMPORT_PATH: &str = "rocm_cli_vllm_auth.RequireEndpointApiKey"; + +/// Environment variable carrying the key to both vLLM's own check and the +/// middleware. The middleware reads this exact name (`KEY_ENV` in the module). +pub(crate) const KEY_ENV: &str = "VLLM_API_KEY"; + +/// Extra `vllm serve` arguments that install the middleware. +pub(crate) fn middleware_args() -> [String; 2] { + ["--middleware".to_owned(), MIDDLEWARE_IMPORT_PATH.to_owned()] +} + +/// Directory the middleware module is written to under the engine's data +/// directory. Owned by the invoking user like the rest of that tree, so only +/// they can replace the code vLLM imports. +pub(crate) fn module_dir(paths: &rocm_core::AppPaths) -> PathBuf { + paths.engine_dir(crate::ENGINE_NAME).join("python") +} + +/// Write the middleware module into `dir`, replacing any earlier copy. +/// +/// Written to a temporary file and renamed into place, so a concurrent launch +/// importing the module never reads a half-written file. +pub(crate) fn write_module(dir: &Path) -> Result { + fs::create_dir_all(dir).with_context(|| format!("failed to create {}", dir.display()))?; + let path = dir.join(format!("{MODULE_NAME}.py")); + let mut staged = tempfile::NamedTempFile::new_in(dir) + .with_context(|| format!("failed to stage a file in {}", dir.display()))?; + staged + .write_all(MODULE_SOURCE.as_bytes()) + .with_context(|| format!("failed to write {}", path.display()))?; + staged + .persist(&path) + .with_context(|| format!("failed to write {}", path.display()))?; + Ok(path) +} + +/// Require `api_key` on every route of the vLLM server `command` will start. +/// +/// Sets the key, writes the middleware module into `module_dir` and prepends +/// that directory to `PYTHONPATH`. Call after any other `PYTHONPATH` change +/// (see `apply_therock_env`): an entry `command` already sets is extended +/// rather than replaced. The `--middleware` argument itself comes from +/// [`middleware_args`] via `vllm_serve_args`, so the argv stays a pure +/// function of the launch request. +pub(crate) fn protect_endpoint( + command: &mut ProcessCommand, + api_key: &str, + module_dir: &Path, +) -> Result<()> { + write_module(module_dir)?; + command.env(KEY_ENV, api_key); + let pythonpath = middleware_pythonpath( + module_dir, + command_env(command, "PYTHONPATH"), + std::env::var_os("PYTHONPATH"), + )?; + command.env("PYTHONPATH", pythonpath); + Ok(()) +} + +/// How a [`ProcessCommand`] treats one environment variable. +#[derive(Debug, PartialEq, Eq)] +enum EnvOverride { + /// The command leaves it alone; the child inherits this process's value. + Inherited, + /// The command removes it (`env_remove`). + Removed, + /// The command sets it to this value. + Set(OsString), +} + +/// The server's `PYTHONPATH`: `module_dir` first, then whatever the server +/// would otherwise have seen — the value the command sets, nothing if it +/// removes the variable, or `inherited` when it leaves it alone. Dropping the +/// inherited value would silently change which packages the server imports. +fn middleware_pythonpath( + module_dir: &Path, + command_value: EnvOverride, + inherited: Option, +) -> Result { + let current = match command_value { + EnvOverride::Inherited => inherited, + EnvOverride::Removed => None, + EnvOverride::Set(value) => Some(value), + }; + crate::process::prepend_path_entries(&[module_dir.to_path_buf()], current) +} + +/// How `command` treats the environment variable `key`. +fn command_env(command: &ProcessCommand, key: &str) -> EnvOverride { + command + .get_envs() + .find_map(|(name, value)| (name == key).then_some(value)) + .map_or(EnvOverride::Inherited, |value| match value { + Some(value) => EnvOverride::Set(value.to_owned()), + None => EnvOverride::Removed, + }) +} + +/// The value `command` sets `key` to, if it sets one. Shared by this module's +/// tests and the launch-command tests in `process.rs`. +#[cfg(test)] +pub(crate) fn env_of(command: &ProcessCommand, key: &str) -> Option { + match command_env(command, key) { + EnvOverride::Set(value) => Some(value), + EnvOverride::Inherited | EnvOverride::Removed => None, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::process::Stdio; + + fn entries(value: &OsString) -> Vec { + std::env::split_paths(value).collect() + } + + #[test] + fn middleware_pythonpath_keeps_the_inherited_value_when_the_command_sets_none() -> Result<()> { + let module_dir = PathBuf::from("/data/python"); + let composed = middleware_pythonpath( + &module_dir, + EnvOverride::Inherited, + Some("/user/site".into()), + )?; + assert_eq!( + entries(&composed), + vec![module_dir, PathBuf::from("/user/site")], + "the server must still see the PYTHONPATH it would have inherited" + ); + Ok(()) + } + + #[test] + fn middleware_pythonpath_prefers_the_command_value_over_the_inherited_one() -> Result<()> { + let module_dir = PathBuf::from("/data/python"); + let composed = middleware_pythonpath( + &module_dir, + EnvOverride::Set("/sdk/amd_smi".into()), + Some("/user/site".into()), + )?; + assert_eq!( + entries(&composed), + vec![module_dir, PathBuf::from("/sdk/amd_smi")] + ); + Ok(()) + } + + #[test] + fn middleware_pythonpath_honours_a_command_that_removes_the_variable() -> Result<()> { + let module_dir = PathBuf::from("/data/python"); + let composed = + middleware_pythonpath(&module_dir, EnvOverride::Removed, Some("/user/site".into()))?; + assert_eq!(entries(&composed), vec![module_dir]); + Ok(()) + } + + #[test] + fn command_env_distinguishes_inherited_removed_and_set() { + let mut command = ProcessCommand::new("vllm"); + assert_eq!(command_env(&command, "PYTHONPATH"), EnvOverride::Inherited); + command.env_remove("PYTHONPATH"); + assert_eq!(command_env(&command, "PYTHONPATH"), EnvOverride::Removed); + command.env("PYTHONPATH", "/x"); + assert_eq!( + command_env(&command, "PYTHONPATH"), + EnvOverride::Set("/x".into()) + ); + } + + #[test] + fn protect_endpoint_sets_key_and_makes_the_middleware_importable() -> Result<()> { + let temp = tempfile::tempdir()?; + let module_dir = temp.path().join("python"); + let mut command = ProcessCommand::new("vllm"); + command.env("PYTHONPATH", "/opt/existing"); + + protect_endpoint(&mut command, "secret-key", &module_dir)?; + + assert_eq!(env_of(&command, KEY_ENV), Some("secret-key".into())); + let pythonpath = env_of(&command, "PYTHONPATH").expect("PYTHONPATH set"); + let entries: Vec = std::env::split_paths(&pythonpath).collect(); + assert_eq!( + entries, + vec![module_dir.clone(), PathBuf::from("/opt/existing")], + "the middleware directory goes first and an earlier entry is kept" + ); + let written = fs::read_to_string(module_dir.join(format!("{MODULE_NAME}.py")))?; + assert_eq!(written, MODULE_SOURCE); + Ok(()) + } + + #[test] + fn write_module_replaces_a_stale_copy() -> Result<()> { + let temp = tempfile::tempdir()?; + let path = temp.path().join(format!("{MODULE_NAME}.py")); + fs::write(&path, "class RequireEndpointApiKey: pass\n")?; + assert_eq!(write_module(temp.path())?, path); + assert_eq!(fs::read_to_string(&path)?, MODULE_SOURCE); + Ok(()) + } + + #[test] + fn import_path_names_the_shipped_module_and_class() { + let (module, class) = MIDDLEWARE_IMPORT_PATH + .rsplit_once('.') + .expect("module.Class"); + assert_eq!(module, MODULE_NAME); + assert!(MODULE_SOURCE.contains(&format!("\nclass {class}:"))); + assert!(MODULE_SOURCE.contains(&format!("KEY_ENV = \"{KEY_ENV}\""))); + } + + /// Drives the shipped middleware with a stdlib-only ASGI harness, the same + /// way vLLM's `--middleware` loader mounts it (`app.add_middleware(cls)` + /// wraps the app in `cls(app)`). Proves the routes vLLM leaves outside its + /// own key check — `/invocations`, `/tokenize`, ... — answer 401 without + /// the key, and that only `GET`/`HEAD /health` and CORS preflights stay + /// open. + /// + /// Needs `python3` on PATH and fails loudly without it rather than passing + /// vacuously. Linux-only: vLLM serving is Linux/WSL-only in rocm-cli, so + /// the module never runs anywhere else. + #[test] + #[cfg_attr( + not(target_os = "linux"), + ignore = "vLLM serving (and so this middleware) is Linux/WSL-only" + )] + fn middleware_requires_the_key_on_every_route_but_health() -> Result<()> { + let temp = tempfile::tempdir()?; + let module_dir = temp.path().join("python"); + write_module(&module_dir)?; + let harness = temp.path().join("harness.py"); + fs::write(&harness, ASGI_HARNESS)?; + + let output = ProcessCommand::new("python3") + .arg(&harness) + .env("PYTHONPATH", &module_dir) + .env(KEY_ENV, "secret-key") + .stdin(Stdio::null()) + .output() + .context("python3 is required to exercise the vLLM auth middleware")?; + let stdout = String::from_utf8_lossy(&output.stdout); + let stderr = String::from_utf8_lossy(&output.stderr); + assert!( + output.status.success(), + "middleware harness failed\nstdout:\n{stdout}\nstderr:\n{stderr}" + ); + assert!(stdout.contains("ALL OK"), "harness stdout:\n{stdout}"); + Ok(()) + } + + /// Construction must fail when the key is empty, so a protected launch + /// whose key went missing cannot answer a request without checking one. + #[test] + #[cfg_attr( + not(target_os = "linux"), + ignore = "vLLM serving (and so this middleware) is Linux/WSL-only" + )] + fn middleware_refuses_to_start_without_a_key() -> Result<()> { + let temp = tempfile::tempdir()?; + write_module(temp.path())?; + let output = ProcessCommand::new("python3") + .args([ + "-c", + "import rocm_cli_vllm_auth as m; m.RequireEndpointApiKey(lambda *a: None)", + ]) + .env("PYTHONPATH", temp.path()) + .env_remove(KEY_ENV) + .stdin(Stdio::null()) + .output() + .context("python3 is required to exercise the vLLM auth middleware")?; + assert!( + !output.status.success(), + "constructing without a key must fail" + ); + assert!( + String::from_utf8_lossy(&output.stderr).contains("VLLM_API_KEY is empty"), + "stderr: {}", + String::from_utf8_lossy(&output.stderr) + ); + Ok(()) + } + + const ASGI_HARNESS: &str = r#" +import asyncio +import rocm_cli_vllm_auth as m + +async def inner(scope, receive, send): + if scope["type"] == "websocket": + await send({"type": "websocket.accept"}) + return + await send({"type": "http.response.start", "status": 200, "headers": []}) + await send({"type": "http.response.body", "body": b"inner"}) + +async def call(app, method, path, headers=(), kind="http", root_path=""): + sent = [] + async def receive(): + return {"type": "http.request", "body": b"", "more_body": False} + async def send(message): + sent.append(message) + scope = {"type": kind, "method": method, "path": path, "root_path": root_path, + "headers": [(k.lower().encode(), v.encode()) for k, v in headers]} + await app(scope, receive, send) + first = sent[0] + if first["type"] == "http.response.start": + return first["status"] + return first["type"] + +GOOD = [("Authorization", "Bearer secret-key")] +CASES = [ + # Routes outside vLLM's own guarded prefixes must need the key too. + ("POST", "/invocations", (), 401), + ("POST", "/tokenize", (), 401), + ("POST", "/detokenize", (), 401), + ("POST", "/generative_scoring", (), 401), + ("POST", "/scale_elastic_ep", (), 401), + ("GET", "/metrics", (), 401), + ("GET", "/ping", (), 401), + ("POST", "/v1/chat/completions", (), 401), + ("GET", "/some/future/route", (), 401), + # Wrong, malformed or non-bearer credentials. + ("POST", "/invocations", [("Authorization", "Bearer wrong")], 401), + ("POST", "/invocations", [("Authorization", "Bearer secret-key-and-more")], 401), + ("POST", "/invocations", [("Authorization", "Basic secret-key")], 401), + ("POST", "/invocations", [("Authorization", "secret-key")], 401), + ("POST", "/invocations", [("X-Api-Key", "secret-key")], 401), + # Only GET/HEAD /health is open, and only at exactly that path. + ("GET", "/health", (), 200), + ("HEAD", "/health", (), 200), + ("POST", "/health", (), 401), + ("GET", "/health/", (), 401), + ("GET", "/healthz", (), 401), + # A CORS preflight (OPTIONS carrying both Origin and + # Access-Control-Request-Method) is the only other keyless request. Each + # part alone is not one, and neither are those headers on another method. + ("OPTIONS", "/invocations", (), 401), + ("OPTIONS", "/invocations", + [("Origin", "http://x"), ("Access-Control-Request-Method", "POST")], 200), + ("OPTIONS", "/invocations", [("Origin", "http://x")], 401), + ("OPTIONS", "/invocations", [("Access-Control-Request-Method", "POST")], 401), + ("POST", "/invocations", + [("Origin", "http://x"), ("Access-Control-Request-Method", "POST")], 401), + # The right key opens every route, with any bearer-scheme casing. + ("POST", "/invocations", GOOD, 200), + ("POST", "/tokenize", GOOD, 200), + ("GET", "/metrics", GOOD, 200), + ("POST", "/v1/chat/completions", [("Authorization", "bearer secret-key")], 200), +] + +async def main(): + app = m.RequireEndpointApiKey(inner) + failures = [] + for method, path, headers, want in CASES: + got = await call(app, method, path, headers) + if got != want: + failures.append(f"{method} {path} {headers!r}: want {want}, got {got}") + # root_path is stripped before matching, as vLLM's own check does. + got = await call(app, "GET", "/prefix/health", root_path="/prefix") + if got != 200: + failures.append(f"root_path /health: want 200, got {got}") + # Only a path that starts with root_path has it stripped. A path of the same + # length as root_path but outside it is not the health check, and neither is + # a path that holds root_path anywhere but at its start. + for path in ("/xxxxxx/health", "/health/prefix"): + got = await call(app, "GET", path, root_path="/prefix") + if got != 401: + failures.append(f"{path} outside root_path: want 401, got {got}") + # WebSocket routes are closed before the handshake without the key. + got = await call(app, None, "/v1/realtime", kind="websocket") + if got != "websocket.close": + failures.append(f"websocket without key: want close, got {got}") + got = await call(app, None, "/v1/realtime", GOOD, kind="websocket") + if got != "websocket.accept": + failures.append(f"websocket with key: want accept, got {got}") + # The /health exemption is for HTTP GET/HEAD only, not a WebSocket there. + # This catches an exemption keyed on the path alone (for any scope type). + # Dropping only the scope-type guard in `_exempt` would still pass here, + # because a WebSocket scope carries no method for the GET/HEAD check. + got = await call(app, None, "/health", kind="websocket") + if got != "websocket.close": + failures.append(f"websocket /health without key: want close, got {got}") + # Lifespan events pass straight through. + seen = [] + async def lifespan_inner(scope, receive, send): + seen.append(scope["type"]) + await m.RequireEndpointApiKey(lifespan_inner)({"type": "lifespan"}, None, None) + if seen != ["lifespan"]: + failures.append(f"lifespan not passed through: {seen}") + if failures: + print("\n".join(failures)) + raise SystemExit(1) + print("ALL OK") + +asyncio.run(main()) +"#; +} diff --git a/engines/vllm/src/lib.rs b/engines/vllm/src/lib.rs index 80ad31eca..836976797 100644 --- a/engines/vllm/src/lib.rs +++ b/engines/vllm/src/lib.rs @@ -18,6 +18,7 @@ use std::io::{Read, Write}; use std::path::PathBuf; mod discover_builds; +mod endpoint_auth; mod install; mod process; mod repair; diff --git a/engines/vllm/src/process.rs b/engines/vllm/src/process.rs index 87f3d4f9a..b1d6a9033 100644 --- a/engines/vllm/src/process.rs +++ b/engines/vllm/src/process.rs @@ -235,25 +235,19 @@ the ROCm SDK's bundled numa uses renamed symbol versions and cannot satisfy it, .with_context(|| format!("failed to create {}", parent.display()))?; } - let mut command = ProcessCommand::new(&runtime.command); - command - .args(vllm_serve_args( - &request.model_ref, - &request.host, - request.port, - vllm_enforce_eager_enabled(), - request.engine_recipe.as_ref(), - )) - .stdin(Stdio::null()); - // When `rocm serve` protects a public endpoint, the key arrives via the - // environment / key file (never argv). Hand it to vLLM as `VLLM_API_KEY` so - // its OpenAI server rejects unauthenticated requests; passing it through the - // env keeps it out of the process table. - if let Some(api_key) = rocm_engine_protocol::resolve_endpoint_api_key() { - command.env("VLLM_API_KEY", api_key); - } - apply_therock_env(&mut command, runtime)?; - rocm_engine_protocol::apply_gpu_visibility(&mut command, &request.gpu_indices); + // When `rocm serve` protects an endpoint, the key arrives via the key file + // (never argv). It reaches vLLM through the environment, which keeps it out + // of the process table, and is enforced on every route by the middleware + // `endpoint_auth` installs: vLLM's own `VLLM_API_KEY` check alone leaves + // routes such as `/invocations` open. + let api_key = rocm_engine_protocol::resolve_endpoint_api_key(); + let mut command = vllm_server_command( + request, + runtime, + api_key.as_deref(), + || Ok(crate::endpoint_auth::module_dir(&AppPaths::discover()?)), + vllm_enforce_eager_enabled(), + )?; if let Some(log_path) = log_path { let log = fs::File::create(log_path) .with_context(|| format!("failed to create {}", log_path.display()))?; @@ -274,6 +268,40 @@ the ROCm SDK's bundled numa uses renamed symbol versions and cannot satisfy it, } }) } +/// The `vllm serve` command for `request`, short of where its output goes: +/// argv, TheRock environment and GPU visibility, plus, with an `api_key`, the +/// endpoint-key protection: the `--middleware` argument, `VLLM_API_KEY`, and +/// the middleware module written to `module_dir()` and put at the front of +/// `PYTHONPATH`. `--middleware` depends on the other two: vLLM has to import +/// the module from `PYTHONPATH`, and the middleware refuses to construct +/// without `VLLM_API_KEY`. So all three are built in one place and asserted +/// together by one test. `module_dir` is consulted only for a keyed launch. +fn vllm_server_command( + request: &ServeHttpRequest, + runtime: &VllmRuntime, + api_key: Option<&str>, + module_dir: impl FnOnce() -> Result, + enforce_eager: bool, +) -> Result { + let mut command = ProcessCommand::new(&runtime.command); + command + .args(vllm_serve_args( + &request.model_ref, + &request.host, + request.port, + enforce_eager, + request.engine_recipe.as_ref(), + api_key.is_some(), + )) + .stdin(Stdio::null()); + apply_therock_env(&mut command, runtime)?; + // After `apply_therock_env`, which may set its own `PYTHONPATH` entry. + if let Some(api_key) = api_key { + crate::endpoint_auth::protect_endpoint(&mut command, api_key, &module_dir()?)?; + } + rocm_engine_protocol::apply_gpu_visibility(&mut command, &request.gpu_indices); + Ok(command) +} /// Explain a spawn that failed on a script whose `#!` interpreter is gone. /// /// The kernel reports the missing *interpreter* as ENOENT against the *script*, @@ -516,7 +544,8 @@ fn apply_vllm_rocm10_discover_env( command.env("FLASH_ATTENTION_TRITON_AMD_ENABLE", "TRUE"); Ok(()) } -/// Full argument vector for `vllm serve`, as spawned by [`spawn_vllm_server`]. +/// Full argument vector for `vllm serve`, as built into the launch command by +/// [`vllm_server_command`]. /// /// Kept as a pure function so the exact argv can be asserted in tests: memory /// and graph-mode flags materially change how much VRAM vLLM claims, and a @@ -526,12 +555,16 @@ fn apply_vllm_rocm10_discover_env( /// supply a default — vLLM's own default applies unless the user asks for a /// specific fraction via `rocm serve --gpu-memory-utilization`, which arrives /// here through the engine recipe's `required_flags`. +/// +/// `require_api_key` adds the `--middleware` that enforces the endpoint key on +/// every route (see [`crate::endpoint_auth`]). fn vllm_serve_args( model_ref: &str, host: &str, port: u16, enforce_eager: bool, engine_recipe: Option<&EngineRecipeHint>, + require_api_key: bool, ) -> Vec { let mut args = vec![ "serve".to_owned(), @@ -549,6 +582,9 @@ fn vllm_serve_args( args.push("--enforce-eager".to_owned()); } args.extend(engine_recipe_launch_args(engine_recipe)); + if require_api_key { + args.extend(crate::endpoint_auth::middleware_args()); + } args } /// Whether to launch vLLM with `--enforce-eager` (CUDA graphs disabled). @@ -583,7 +619,10 @@ fn resolve_vllm_ready_timeout(override_value: Option) -> Duration { .map_or(DEFAULT_VLLM_READY_TIMEOUT, Duration::from_secs) } -fn prepend_path_entries(entries: &[PathBuf], current: Option) -> Result { +pub(crate) fn prepend_path_entries( + entries: &[PathBuf], + current: Option, +) -> Result { let mut parts = Vec::new(); for entry in entries { if !entry.as_os_str().is_empty() && !parts.iter().any(|part: &PathBuf| part == entry) { @@ -815,7 +854,7 @@ mod tests { #[test] fn vllm_serve_args_omit_gpu_memory_utilization_without_override() { - let args = vllm_serve_args("facebook/opt-125m", "127.0.0.1", 8000, false, None); + let args = vllm_serve_args("facebook/opt-125m", "127.0.0.1", 8000, false, None, false); assert_eq!( args, @@ -842,7 +881,14 @@ mod tests { ..EngineRecipeHint::default() }; - let args = vllm_serve_args("facebook/opt-125m", "127.0.0.1", 8000, true, Some(&hint)); + let args = vllm_serve_args( + "facebook/opt-125m", + "127.0.0.1", + 8000, + true, + Some(&hint), + false, + ); let position = args .iter() @@ -852,6 +898,178 @@ mod tests { assert!(args.contains(&"--enforce-eager".to_owned())); } #[test] + fn vllm_serve_args_install_the_auth_middleware_when_a_key_is_required() { + let args = vllm_serve_args("facebook/opt-125m", "0.0.0.0", 8000, false, None, true); + + let position = args + .iter() + .position(|arg| arg == "--middleware") + .expect("a keyed endpoint must be launched with the auth middleware"); + assert_eq!( + args.get(position + 1).map(String::as_str), + Some(crate::endpoint_auth::MIDDLEWARE_IMPORT_PATH) + ); + assert!( + !args.iter().any(|arg| arg.contains("api-key")), + "the key itself must never reach argv: {args:?}" + ); + } + #[test] + fn vllm_serve_args_omit_the_auth_middleware_without_a_key() { + let args = vllm_serve_args("facebook/opt-125m", "127.0.0.1", 8000, false, None, false); + assert!( + !args.iter().any(|arg| arg == "--middleware"), + "an unkeyed endpoint must not load the middleware, which refuses to construct \ + without a key: {args:?}" + ); + } + fn keyed_launch_request() -> ServeHttpRequest { + ServeHttpRequest { + service_id: "svc-vllm".to_owned(), + model_ref: "facebook/opt-125m".to_owned(), + host: "0.0.0.0".to_owned(), + port: 8000, + device_policy: DevicePolicy::GpuRequired, + gpu_indices: vec![1], + runtime_id: None, + env_id: None, + state_path: PathBuf::from("state.json"), + log_path: None, + engine_recipe: None, + } + } + #[test] + fn keyed_vllm_server_command_installs_middleware_key_and_pythonpath_together() -> Result<()> { + let temp = tempfile::tempdir()?; + let module_dir = temp.path().join("python"); + // A ROCm 10 discover runtime, whose TheRock environment sets its own + // `PYTHONPATH` entry (`amd_smi` in the venv). The middleware directory + // has to be prepended after that, or the TheRock value replaces it. + let venv = temp.path().join("venv"); + let site_packages = venv.join("lib").join("python3.12").join("site-packages"); + fs::create_dir_all(&site_packages)?; + let runtime = test_vllm_runtime( + Some(venv.join("bin").join("python")), + Some("10.0.0".to_owned()), + ); + let command = vllm_server_command( + &keyed_launch_request(), + &runtime, + Some("secret-key"), + || Ok(module_dir.clone()), + false, + )?; + + let args: Vec = command + .get_args() + .map(|arg| arg.to_string_lossy().into_owned()) + .collect(); + let position = args + .iter() + .position(|arg| arg == "--middleware") + .expect("a keyed launch must pass the auth middleware to vllm serve"); + assert_eq!( + args.get(position + 1).map(String::as_str), + Some(crate::endpoint_auth::MIDDLEWARE_IMPORT_PATH) + ); + assert!( + !args.iter().any(|arg| arg.contains("secret-key")), + "the key must never reach argv: {args:?}" + ); + assert!( + !args.iter().any(|arg| arg == "--enforce-eager"), + "enforce_eager was false: {args:?}" + ); + assert_eq!( + crate::endpoint_auth::env_of(&command, "VLLM_API_KEY"), + Some("secret-key".into()), + "the middleware and vLLM's own check both read the key from VLLM_API_KEY" + ); + let pythonpath = crate::endpoint_auth::env_of(&command, "PYTHONPATH") + .expect("a keyed launch must put the middleware directory on PYTHONPATH"); + let entries: Vec = std::env::split_paths(&pythonpath).collect(); + let amd_smi = site_packages + .join("_rocm_sdk_core") + .join("share") + .join("amd_smi"); + assert_eq!( + entries.get(..2), + Some(&[module_dir.clone(), amd_smi][..]), + "vLLM imports the middleware by module name, so its directory goes first, \ + ahead of the runtime's own entry, which is kept: {entries:?}" + ); + assert_eq!( + crate::endpoint_auth::env_of(&command, "HIP_VISIBLE_DEVICES"), + Some("1".into()), + "the GPU the request selected must still be applied to a keyed launch" + ); + assert!( + module_dir.join("rocm_cli_vllm_auth.py").is_file(), + "the middleware module must be written where PYTHONPATH points" + ); + Ok(()) + } + #[test] + fn vllm_server_command_forwards_enforce_eager() -> Result<()> { + let command = vllm_server_command( + &keyed_launch_request(), + &VllmRuntime::for_test(), + None, + || panic!("an unkeyed launch must not need the middleware directory"), + true, + )?; + assert!( + command.get_args().any(|arg| arg == "--enforce-eager"), + "enforce_eager was true: {:?}", + command.get_args().collect::>() + ); + Ok(()) + } + #[test] + fn unkeyed_vllm_server_command_carries_no_key_protection() -> Result<()> { + // The same TheRock runtime as the keyed test, so `PYTHONPATH` has a value + // of its own that a stray middleware entry would be prepended to. + let temp = tempfile::tempdir()?; + let venv = temp.path().join("venv"); + let site_packages = venv.join("lib").join("python3.12").join("site-packages"); + fs::create_dir_all(&site_packages)?; + let runtime = test_vllm_runtime( + Some(venv.join("bin").join("python")), + Some("10.0.0".to_owned()), + ); + let command = vllm_server_command( + &keyed_launch_request(), + &runtime, + None, + || panic!("an unkeyed launch must not need the middleware directory"), + false, + )?; + assert!( + !command.get_args().any(|arg| arg == "--middleware"), + "an unkeyed launch must not load the middleware, which refuses to start without a key" + ); + assert_eq!(crate::endpoint_auth::env_of(&command, "VLLM_API_KEY"), None); + let pythonpath = crate::endpoint_auth::env_of(&command, "PYTHONPATH") + .expect("the TheRock runtime sets its own PYTHONPATH entry"); + let entries: Vec = std::env::split_paths(&pythonpath).collect(); + let amd_smi = site_packages + .join("_rocm_sdk_core") + .join("share") + .join("amd_smi"); + // The runtime's own value: its entry ahead of whatever PYTHONPATH this + // process inherited. + let expected: Vec = std::env::split_paths(&prepend_path_entries( + &[amd_smi], + std::env::var_os("PYTHONPATH"), + )?) + .collect(); + assert_eq!( + entries, expected, + "an unkeyed launch keeps the runtime's PYTHONPATH exactly, with no middleware directory" + ); + Ok(()) + } + #[test] fn resolve_model_echoes_matching_engine_recipe() -> Result<()> { let hint = test_engine_recipe(ENGINE_NAME, ENGINE_RECIPE_CONTRACT_VERSION); let response = resolve_model_response(ResolveModelRequest { diff --git a/engines/vllm/src/rocm_cli_vllm_auth.py b/engines/vllm/src/rocm_cli_vllm_auth.py new file mode 100644 index 000000000..38f408f81 --- /dev/null +++ b/engines/vllm/src/rocm_cli_vllm_auth.py @@ -0,0 +1,125 @@ +# Copyright © Advanced Micro Devices, Inc., or its affiliates. +# +# SPDX-License-Identifier: MIT + +"""Require the endpoint API key on every route of a vLLM OpenAI server. + +rocm-cli passes ``--middleware rocm_cli_vllm_auth.RequireEndpointApiKey`` to +``vllm serve`` whenever it protects an endpoint with an API key. vLLM's own +``VLLM_API_KEY`` check guards only a fixed set of path prefixes (``/v1`` and +a few others), while the server also registers inference-capable routes +outside them (``/invocations``, ``/tokenize``, ...). This middleware closes +that gap by checking the key on every HTTP and WebSocket request, whatever +its path, so a route vLLM adds in a later release is covered too. + +Only ``GET``/``HEAD /health`` and CORS preflight requests pass without the +key: the first is the liveness probe rocm-cli advertises, the second carries +no credentials by design and is answered by vLLM's CORS middleware before +any route runs. + +The key is read from ``VLLM_API_KEY`` -- the same variable vLLM's own check +reads, set by rocm-cli on the server process -- so both checks always agree. +Construction fails when the variable is empty, so a launch that asked for +protection cannot come up unprotected. + +Pure ASGI and standard library only: it must import in whichever vLLM +release the runtime ships, with no assumption about its Starlette version. +This file is written out by the rocm-cli vLLM engine adapter at launch; +edit the copy in the rocm-cli source tree, not the one on disk. +""" + +import hashlib +import hmac +import os + +KEY_ENV = "VLLM_API_KEY" + +# Exact paths (after any ASGI root_path) that answer without the key, and the +# only methods they answer to. Everything else requires the key. +UNAUTHENTICATED_ROUTES = {"/health": frozenset({"GET", "HEAD"})} + +_UNAUTHORIZED_BODY = b'{"error":"Unauthorized"}' + +# RFC 6455 policy-violation close code. Closing before accepting makes the +# server reject the WebSocket handshake with HTTP 403. +_WS_POLICY_VIOLATION = 1008 + + +def _digest(value): + return hashlib.sha256(value).digest() + + +class RequireEndpointApiKey: + """ASGI middleware rejecting any request that lacks the bearer key.""" + + def __init__(self, app): + key = os.environ.get(KEY_ENV, "") + if not key: + raise RuntimeError( + f"{KEY_ENV} is empty; refusing to start an endpoint that was " + "asked to require an API key without one" + ) + self.app = app + self._expected = _digest(key.encode("utf-8")) + + def _authorized(self, scope): + authorization = None + for name, value in scope.get("headers") or (): + if name.lower() == b"authorization": + authorization = value + break + if authorization is None: + return False + scheme, _, token = authorization.decode("latin-1").partition(" ") + if scheme.lower() != "bearer": + return False + # Compare fixed-length digests so the comparison time depends on + # neither the key's length nor how much of it matched. + return hmac.compare_digest(_digest(token.encode("utf-8")), self._expected) + + @staticmethod + def _route_path(scope): + path = scope.get("path", "") + root_path = scope.get("root_path", "") + if root_path and path.startswith(root_path): + path = path[len(root_path) :] + return path + + @staticmethod + def _is_cors_preflight(scope): + if scope.get("method") != "OPTIONS": + return False + names = {name.lower() for name, _ in scope.get("headers") or ()} + return b"origin" in names and b"access-control-request-method" in names + + def _exempt(self, scope): + if scope["type"] != "http": + return False + if self._is_cors_preflight(scope): + return True + methods = UNAUTHENTICATED_ROUTES.get(self._route_path(scope)) + return methods is not None and scope.get("method") in methods + + async def __call__(self, scope, receive, send): + kind = scope["type"] + if kind not in ("http", "websocket") or self._exempt(scope): + await self.app(scope, receive, send) + return + if self._authorized(scope): + await self.app(scope, receive, send) + return + if kind == "websocket": + await send({"type": "websocket.close", "code": _WS_POLICY_VIOLATION}) + return + await send( + { + "type": "http.response.start", + "status": 401, + "headers": [ + (b"content-type", b"application/json"), + (b"content-length", str(len(_UNAUTHORIZED_BODY)).encode()), + (b"www-authenticate", b"Bearer"), + ], + } + ) + await send({"type": "http.response.body", "body": _UNAUTHORIZED_BODY}) diff --git a/tests/e2e-cucumber/features/diagnose.feature b/tests/e2e-cucumber/features/diagnose.feature index c5afa35f5..94962ec51 100644 --- a/tests/e2e-cucumber/features/diagnose.feature +++ b/tests/e2e-cucumber/features/diagnose.feature @@ -915,7 +915,8 @@ Feature: Diagnosing failures and listing fixes # paste is diagnosed (not investigated further; the console is started with # cursor inheritance, which waits on a reply the screen emulator does not # send). That leaves the Windows half of the hint (Ctrl-Z then Enter) without - # an e2e check; docs/manual-testing.md section 13 covers it by hand. + # an e2e check; docs/manual-testing.md's "Pasting An Error Into `rocm diagnose`" + # section covers it by hand. @id:diagnose-a-paste-at-the-terminal-is-explained-and-diagnosed @requires-os:linux Scenario: diagnose-60 - Pasting the error at the terminal is explained, and the paste is diagnosed when the user ends it Given a pasted error containing quotes, a command substitution and a leading minus sign diff --git a/tests/e2e-cucumber/features/model_serving.feature b/tests/e2e-cucumber/features/model_serving.feature index 096bd0825..e6fd7b0eb 100644 --- a/tests/e2e-cucumber/features/model_serving.feature +++ b/tests/e2e-cucumber/features/model_serving.feature @@ -980,3 +980,27 @@ Feature: Model serving And the deployment summary names the logs command for that service And the launch's service record is left as ready And no background automation daemon was left behind + + # An API key on a vLLM endpoint (`--require-api-key`, a public bind, or + # `rocm remote serve`) has to cover the whole server, not only the OpenAI `/v1` + # API: vLLM's own key check guards a fixed set of path prefixes, while the + # server also answers inference and tokenizer routes outside them. rocm-cli + # starts vLLM with a middleware that demands the key on every route except + # `GET`/`HEAD /health` and CORS preflights. The middleware itself is + # unit-tested on every PR (`endpoint_auth` in the vLLM engine crate); this + # proves it is actually wired into a real vLLM server. A full extra vLLM + # serve, so `@nightly` rather than per-PR: it runs after merge in the nightly + # runs on Instinct GPUs, which serve with vLLM: the MI300X job in + # `nightly.yml` and the shared GPU queue's Instinct lanes (MI350P among them). + # Not `@merge-queue`: the merge queue runs no lane where vLLM is the engine + # (the MI300X lane does not run on `merge_group`), so under that tag it + # skipped everywhere. + @id:serve-vllm-key-guards-every-route @requires-gpu @requires-engine:vllm @nightly + Scenario: serve-61 - A key-protected vLLM endpoint refuses keyless requests outside the OpenAI API + Given a managed runtime is active + And a model is being served on GPU behind an API key + When a client calls the server's routes without the key + Then every route outside the OpenAI API refuses the call as unauthorized + And the health check answers without the key + When the client repeats those calls with the key + Then every one of those calls succeeds diff --git a/tests/e2e-cucumber/tests/e2e.rs b/tests/e2e-cucumber/tests/e2e.rs index 3c2cc9788..11d44472b 100644 --- a/tests/e2e-cucumber/tests/e2e.rs +++ b/tests/e2e-cucumber/tests/e2e.rs @@ -173,6 +173,9 @@ pub struct E2eWorld { /// from. Consumed by [`run_rocm_with_scenario_env`] like `command_env`; `None` /// leaves the harness's own working directory. pub command_dir: Option, + /// `(request, HTTP status)` for each server route a step called directly, + /// in call order, for a later Then step to judge. + pub route_statuses: Vec<(String, u16)>, /// The interactive dash/chat TUI spawned under a pseudo-terminal for this /// scenario, if any (see `e2e::tui_driver`). Torn down in `Drop` before the /// mock server and isolated directory so the child process never outlives @@ -405,6 +408,7 @@ impl Default for E2eWorld { expect_xfail: false, command_env: Vec::new(), command_dir: None, + route_statuses: Vec::new(), tui: None, chat_use_mock: false, comfyui_baseline_torch_stack: None, @@ -985,6 +989,18 @@ pub fn run_rocm_bounded( world: &E2eWorld, args: &[&str], timeout: std::time::Duration, +) -> (String, String, i32) { + run_rocm_bounded_with_env(world, args, &[], timeout) +} + +/// Like [`run_rocm_bounded`], but sets `envs` on the child on top of the +/// scenario's isolated env, for a value that must not appear in the recorded +/// argv (an endpoint API key). +pub fn run_rocm_bounded_with_env( + world: &E2eWorld, + args: &[&str], + envs: &[(&str, &str)], + timeout: std::time::Duration, ) -> (String, String, i32) { use std::process::Stdio; @@ -995,6 +1011,7 @@ pub fn run_rocm_bounded( let mut cmd = std::process::Command::new(&binary); cmd.args(args); world.isolate_cmd(&mut cmd); + cmd.envs(envs.iter().copied()); cmd.stdin(Stdio::null()); cmd.stdout(Stdio::from( std::fs::File::create(&out_path).expect("stdout file"), diff --git a/tests/e2e-cucumber/tests/e2e/serving_steps.rs b/tests/e2e-cucumber/tests/e2e/serving_steps.rs index 990bb6b18..c649d4de6 100644 --- a/tests/e2e-cucumber/tests/e2e/serving_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/serving_steps.rs @@ -95,9 +95,25 @@ pub(crate) async fn model_is_ready( expect_model: Option<&str>, timeout_secs: u64, ) -> bool { + model_is_ready_with_key(models_url, expect_model, timeout_secs, None).await +} + +/// [`model_is_ready`] against an endpoint protected by `api_key`, which a +/// keyless listing would only ever see refused. +async fn model_is_ready_with_key( + models_url: &str, + expect_model: Option<&str>, + timeout_secs: u64, + api_key: Option<&str>, +) -> bool { + let client = reqwest::Client::new(); let deadline = Instant::now() + Duration::from_secs(timeout_secs); while Instant::now() < deadline { - if let Ok(resp) = reqwest::get(models_url).await + let mut request = client.get(models_url); + if let Some(key) = api_key { + request = request.bearer_auth(key); + } + if let Ok(resp) = request.send().await && resp.status().is_success() { match expect_model { @@ -628,6 +644,145 @@ async fn setup_gpu_model(world: &mut E2eWorld) { serve_gpu_model(world).await; } +/// The key the keyed-serve scenario protects its endpoint with. A fixed value: +/// what that scenario checks is whether the key is enforced, not how it is minted. +const PROTECTED_ENDPOINT_KEY: &str = "e2e-protected-endpoint-key-7f3a91c2"; + +#[given("a model is being served on GPU behind an API key")] +async fn setup_gpu_model_behind_key(world: &mut E2eWorld) { + serve_gpu_model_with_key(world, Some(PROTECTED_ENDPOINT_KEY)).await; +} + +/// Server routes outside the OpenAI `/v1` API that can run the model, read its +/// tokenizer or report on it. Each must demand the key like `/v1` does; the +/// bodies are valid requests, so a refusal can only come from authentication. +fn routes_outside_the_openai_api( + model: &str, +) -> Vec<(reqwest::Method, &'static str, serde_json::Value)> { + vec![ + ( + reqwest::Method::POST, + "/invocations", + serde_json::json!({ + "model": model, + "messages": [{"role": "user", "content": "Hello"}], + "max_tokens": 8, + }), + ), + ( + reqwest::Method::POST, + "/tokenize", + serde_json::json!({"model": model, "prompt": "Hello"}), + ), + ( + reqwest::Method::POST, + "/detokenize", + serde_json::json!({"model": model, "tokens": [9707]}), + ), + (reqwest::Method::GET, "/metrics", serde_json::Value::Null), + ] +} + +/// The liveness probe, the one route the middleware answers without the key. +const HEALTH_ROUTE: &str = "GET /health"; + +/// Call every route in [`routes_outside_the_openai_api`] and then `GET /health`, +/// with or without the key, recording each status in `world.route_statuses`. +async fn call_routes_outside_the_openai_api(world: &mut E2eWorld, api_key: Option<&str>) { + let model = world.model_name.clone().expect("no model served"); + let client = reqwest::Client::builder() + .timeout(Duration::from_mins(2)) + .build() + .expect("http client"); + let base = format!("http://127.0.0.1:{SERVE_PORT}"); + world.route_statuses.clear(); + for (method, path, body) in routes_outside_the_openai_api(&model) { + let mut request = client.request(method.clone(), format!("{base}{path}")); + if !body.is_null() { + request = request.json(&body); + } + if let Some(key) = api_key { + request = request.bearer_auth(key); + } + let status = request + .send() + .await + .unwrap_or_else(|e| panic!("{method} {base}{path} failed: {e}")) + .status() + .as_u16(); + world + .route_statuses + .push((format!("{method} {path}"), status)); + } + let mut health = client.get(format!("{base}/health")); + if let Some(key) = api_key { + health = health.bearer_auth(key); + } + let status = health + .send() + .await + .unwrap_or_else(|e| panic!("GET {base}/health failed: {e}")) + .status() + .as_u16(); + world.route_statuses.push((HEALTH_ROUTE.to_owned(), status)); +} + +#[when("a client calls the server's routes without the key")] +async fn client_calls_routes_without_key(world: &mut E2eWorld) { + call_routes_outside_the_openai_api(world, None).await; +} + +#[when("the client repeats those calls with the key")] +async fn client_repeats_calls_with_key(world: &mut E2eWorld) { + call_routes_outside_the_openai_api(world, Some(PROTECTED_ENDPOINT_KEY)).await; +} + +#[then("every route outside the OpenAI API refuses the call as unauthorized")] +async fn assert_other_routes_refused(world: &mut E2eWorld) { + let others: Vec<_> = world + .route_statuses + .iter() + .filter(|(route, _)| route != HEALTH_ROUTE) + .collect(); + assert!(!others.is_empty(), "no routes were called"); + let unrefused: Vec<_> = others + .into_iter() + .filter(|(_, status)| *status != 401) + .collect(); + assert!( + unrefused.is_empty(), + "routes answered a client without the key instead of 401: {unrefused:?}" + ); +} + +#[then("the health check answers without the key")] +async fn assert_health_answers(world: &mut E2eWorld) { + let health = world + .route_statuses + .iter() + .find(|(route, _)| route == HEALTH_ROUTE) + .expect("the health check was not called"); + assert!( + (200..300).contains(&health.1), + "{HEALTH_ROUTE} answered {}; the liveness probe must stay open", + health.1 + ); +} + +#[then("every one of those calls succeeds")] +async fn assert_all_routes_succeed(world: &mut E2eWorld) { + assert!(!world.route_statuses.is_empty(), "no routes were called"); + let failed: Vec<_> = world + .route_statuses + .iter() + .filter(|(_, status)| !(200..300).contains(status)) + .collect(); + assert!( + failed.is_empty(), + "routes refused a client presenting the key: {failed:?}" + ); +} + #[when("the user serves a model on GPU from the installed runtime")] async fn user_serves_gpu_model_from_installed_runtime(world: &mut E2eWorld) { // Unlike `setup_active_runtime`, this scenario intentionally keeps the @@ -638,6 +793,12 @@ async fn user_serves_gpu_model_from_installed_runtime(world: &mut E2eWorld) { } async fn serve_gpu_model(world: &mut E2eWorld) { + serve_gpu_model_with_key(world, None).await; +} + +/// Serve the host's GPU model, protecting the endpoint with `api_key` when one +/// is given (`--require-api-key`, so even this loopback bind demands it). +async fn serve_gpu_model_with_key(world: &mut E2eWorld, api_key: Option<&str>) { // Serve by the canonical HuggingFace ID (not the `qwen2.5` alias) with an // explicit engine matching this host. This step is a *precondition* for // scenarios that test inference/chat behavior, so it must not fail for @@ -676,12 +837,24 @@ async fn serve_gpu_model(world: &mut E2eWorld) { for attempt in 1..=attempts { made = attempt; let device_state = ensure_serve_port_free().await; - let (stdout, stderr, rc) = crate::run_rocm_bounded( + let mut args = vec!["serve", model, "--engine", engine, "--managed"]; + // The key goes through `ROCM_SERVE_API_KEY`, as `rocm serve --help` + // advises, not `--api-key`: argv is recorded verbatim in the uploaded + // `results/commands.jsonl`. + let mut envs = Vec::new(); + if let Some(key) = api_key { + args.push("--require-api-key"); + envs.push(("ROCM_SERVE_API_KEY", key)); + } + let (stdout, stderr, rc) = crate::run_rocm_bounded_with_env( world, - &["serve", model, "--engine", engine, "--managed"], + &args, + &envs, e2e_cucumber::serve_log::serve_cli_budget(timeout_secs), ); - if rc == 0 && model_is_ready(&MODELS_URL, Some(ready_substr), timeout_secs).await { + if rc == 0 + && model_is_ready_with_key(&MODELS_URL, Some(ready_substr), timeout_secs, api_key).await + { world.endpoint = Some(SERVE_BASE_URL.to_string()); world.model_name = Some(model.to_string()); return; diff --git a/tests/e2e-cucumber/tests/e2e/symptom_file_steps.rs b/tests/e2e-cucumber/tests/e2e/symptom_file_steps.rs index 10d3fd7e4..a3c4dc638 100644 --- a/tests/e2e-cucumber/tests/e2e/symptom_file_steps.rs +++ b/tests/e2e-cucumber/tests/e2e/symptom_file_steps.rs @@ -306,8 +306,9 @@ async fn told_how_to_end(world: &mut E2eWorld) { // The hint's own words. The end-of-input key it names is the one the When // step pressed, and the CLI finished because of it -- for the Linux key: // the scenario runs on Linux and WSL only, and the Windows key (Ctrl-Z then - // Enter) is checked by hand in docs/manual-testing.md section 13. The hint - // is wider than the terminal, so where it wraps is ignored. + // Enter) is checked by hand in docs/manual-testing.md's "Pasting An Error + // Into `rocm diagnose`" section. The hint is wider than the terminal, so + // where it wraps is ignored. assert!( squeeze(screen).contains(&squeeze( "Paste the error text, then end it with Ctrl-D at the start of a line \ diff --git a/xtask/src/ascii_output_contract.rs b/xtask/src/ascii_output_contract.rs index 18827f11d..d4d802aef 100644 --- a/xtask/src/ascii_output_contract.rs +++ b/xtask/src/ascii_output_contract.rs @@ -162,6 +162,12 @@ mod tests { "The `©` is in a `#` shell comment of the installer `rocm remote \ provision` uploads. The shell never echoes a comment.", ), + ( + "engines/vllm/src/rocm_cli_vllm_auth.py", + "The `©` is in the `#` copyright comment of the auth middleware \ + module `rocm serve` writes for vLLM to import. Python never prints \ + a comment.", + ), ]; /// Individual literals that are out of scope, as