Skip to content

fix(vllm): require the endpoint key on every server route - #974

Queued
rominf wants to merge 2 commits into
mainfrom
fix/vllm-key-on-every-route
Queued

rominf wants to merge 2 commits into
mainfrom
fix/vllm-key-on-every-route

Conversation

@rominf

@rominf rominf commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm serve protects a vLLM endpoint with an API key when it binds publicly (--allow-public-bind), when --require-api-key is given, and under rocm remote serve. It did this only by setting VLLM_API_KEY. vLLM checks that key only under a fixed set of path prefixes (/v1 and a few others), while the same server also answers routes outside them that run or expose the model, such as /invocations, /tokenize, /detokenize and /metrics. Those routes answered clients that did not present the key, so a key-protected endpoint was only partly protected.

What changes

A keyed vLLM launch now also passes --middleware rocm_cli_vllm_auth.RequireEndpointApiKey.

  • The middleware (engines/vllm/src/rocm_cli_vllm_auth.py) uses only the Python standard library and works at the raw ASGI layer, so it does not depend on the Starlette version a given vLLM release ships. It requires Authorization: Bearer <key>, compared in constant time, on every HTTP and WebSocket route. The only exceptions are GET/HEAD /health (the liveness probe) and real CORS preflights. Because it does not look at path prefixes, it also covers routes a later vLLM release adds.
  • The key is read from the same VLLM_API_KEY vLLM's own check reads. The middleware raises instead of constructing when that key is empty, so a server that loaded it never answers a request unchecked. The key still never reaches argv.
  • Installation. The module is compiled into the engine adapter (include_str!) and written atomically under the engine's data directory at launch. That directory is prepended to the server's PYTHONPATH, and any PYTHONPATH the runtime set or the user inherited is kept after it. The launch command is now built by one function, vllm_server_command, so the keyed and unkeyed launches can be asserted directly.
  • Unkeyed launches are unchanged: no --middleware argument, no module on PYTHONPATH.
  • ASCII contract. The module's # copyright line is added to the xtask ASCII-output exemptions, as install.sh's is, because a Python comment is never printed. The rest of the file is ASCII.

User-visible side effects, documented in the README:

  • /metrics now needs the key too, so rocm dash cannot read engine metrics from a key-protected vLLM server. It leaves that server's KV-cache, queue, throughput and latency figures empty and keeps showing a vllm scrape: ... status 401 Unauthorized warning.
  • rocm bench load sends no API key, so it cannot benchmark a key-protected endpoint. That predates this change; the README now says so.
  • The README states the every-route guarantee for the vLLM engine only, and says that AIM runs the same vLLM server without this check, so a key-protected AIM endpoint still answers routes such as /invocations, /tokenize and /metrics without the key (see Scope).

Upgrade note (not in the docs): a key-protected vLLM service that is already running keeps its old launch, without the middleware, until it is restarted. rocm serve reuses a running service whose recipe and key requirement match, and the middleware is part of neither. Restart key-protected vLLM services after upgrading.

Docs: README, docs/testing.md, docs/manual-testing.md (new section 7, "vLLM Endpoint Key Coverage"; later sections renumbered, and the three places that cited the "Pasting An Error Into rocm diagnose" section by number now cite it by title) and docs/architecture.md (entry for endpoint_auth.rs). The doc comment on ensure_public_bind_engine_supported now names the middleware.

Tests

  • Middleware, run for real (engines/vllm/src/endpoint_auth.rs): the shipped module runs under python3 with a standard-library ASGI harness.
    • Without the key, /invocations, /tokenize, /detokenize, /metrics, /ping, /v1/... and an unknown route all return 401.
    • Wrong, malformed and non-bearer credentials are refused; the correct key opens every route.
    • Only GET/HEAD /health (exact path, after stripping root_path, which is stripped only from a path that starts with it) and real CORS preflights pass without the key. Preflight look-alikes are refused: Origin only, Access-Control-Request-Method only, and both headers on a POST. A WebSocket to /health is closed, and WebSockets without the key are closed before the handshake.
    • Lifespan events pass through, and construction fails without a key.
    • Further tests cover how PYTHONPATH is composed (inherited, removed or set by the command), that the module file is written and replaced atomically, and that the import path matches the shipped class.
  • Launch command (engines/vllm/src/process.rs): a keyed launch gets --middleware, VLLM_API_KEY, the middleware directory ahead of the runtime's own PYTHONPATH entry, and the selected GPU. An unkeyed launch on the same runtime gets none of the key protection: no --middleware, no VLLM_API_KEY, and the runtime's own PYTHONPATH unchanged. --enforce-eager is forwarded either way, and the key never appears in argv.
  • E2E: scenario serve-61 (@id:serve-vllm-key-guards-every-route) in model_serving.feature, tagged @requires-gpu @requires-engine:vllm @nightly. A real vLLM server started with --require-api-key refuses keyless calls to /invocations, /tokenize, /detokenize and /metrics while still answering /health, and accepts all of those calls when the client presents the key. The step passes the key through ROCM_SERVE_API_KEY (a new run_rocm_bounded_with_env harness helper keeps the serve call time-bounded), so the key stays out of the recorded command log.
    • It runs after merge in the nightly runs on Instinct GPUs, where vLLM is the serving engine: the MI300X job in nightly.yml and the shared GPU queue's Instinct lanes (MI350P among them). It is not @merge-queue, because no merge-queue lane runs vLLM (the MI300X lane does not run on merge_group). A failure on the nightly lanes does not block merges.
    • To run it on a branch: gh workflow run e2e-selfhosted.yml --ref <branch> -f platform=app-dev-gpu -f name_filter='^serve-61 '. A name_filter dispatch bypasses the harness's tag gates, so include_nightly is not needed.

Breaking the fix turns the tests red:

  • making the middleware skip non-/v1 paths fails the harness test on every listed route;
  • dropping the --middleware argument, moving protect_endpoint ahead of the runtime environment (so the runtime's PYTHONPATH replaces the middleware directory), or skipping GPU selection fails the keyed launch-command test, and prepending any PYTHONPATH entry to an unkeyed launch fails the unkeyed one;
  • widening the CORS-preflight exemption in any of the three ways above, or stripping root_path from a path that does not start with it, fails the harness.

Not unit-tested: the call in spawn_vllm_server that hands the resolved key to vllm_server_command, because that spawn path probes the host (OpenMPI, libatomic, libnuma) before building the command. Only the GPU scenario covers it.

Verification (Linux, Strix Halo host with no vLLM lane)

  • cargo fmt --all --check and cargo clippy --workspace --all-targets -- -D warnings: clean.
  • cargo test -p rocm-engine-vllm --lib: 183 passed.
  • cargo test --workspace --all-targets: 2336 passed, 1 failed in code this change does not touch: service_control::tests::stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails, because this host's docker is podman's emulation, which ignores the test's DOCKER_HOST.
  • cargo test -p e2e-cucumber --lib --test feature_naming --test skill_reference and cargo xtask check-architecture-doc: pass.
  • cargo xtask smoke: every step passes up to rocmd sandbox examine, which needs a bubblewrap network namespace this host does not permit.

Not verified:

  • The GPU scenario serve-vllm-key-guards-every-route: this host has no vLLM lane.
  • The middleware against the source of each pinned vLLM release (0.26.0, 0.27.x, 0.29.x). It was checked against vLLM's documented --middleware behaviour (the class is mounted with app.add_middleware).

Scope

The AIM engine also runs vLLM's server inside its container and sets only VLLM_API_KEY. This change does not cover it; #1007 applies the same rule to AIM.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows.
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md.
  • If this PR adds, renames or removes a production source file, docs/architecture.md has the matching entry, and the prose of any entry whose file changed role still describes it.
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state — not only the wording — per AGENTS.md §3.

@rominf
rominf force-pushed the fix/vllm-key-on-every-route branch 2 times, most recently from bf532fa to 21de06d Compare November 17, 2026 10:17
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf
rominf force-pushed the fix/vllm-key-on-every-route branch from 21de06d to d35b6cc Compare November 17, 2026 11:01
`rocm serve` protects a vLLM endpoint (public bind, --require-api-key,
`rocm remote serve`) by setting VLLM_API_KEY. vLLM checks that key only
under a fixed set of path prefixes (/v1 and a few others), while the
server also answers routes outside them that run or expose the model,
such as /invocations, /tokenize, /detokenize and /metrics. Those routes
answered clients that did not present the key.

A keyed vLLM launch now also passes
`--middleware rocm_cli_vllm_auth.RequireEndpointApiKey`: a pure-ASGI,
standard-library middleware compiled into the engine adapter, written
atomically under the engine's data directory at launch and prepended to
the server's PYTHONPATH (composed over the inherited, removed or
explicitly set value, so the user's own PYTHONPATH and the runtime's are
kept). It requires the bearer key, compared in constant time, on every
HTTP and WebSocket route except GET/HEAD /health and CORS preflights,
and raises instead of constructing when the key is empty, so a server
that loaded it never answers a request unchecked. Being path-agnostic,
it also covers routes a later vLLM adds. The module's copyright comment
is exempted from the xtask ASCII-output contract, as install.sh's is;
the rest of the file is ASCII.

The launch command is built by one function, vllm_server_command, so a
keyed launch can be asserted to carry --middleware, VLLM_API_KEY, the
middleware directory ahead of the runtime's own PYTHONPATH entry and the
selected GPU, and an unkeyed launch none of the key protection.

Because /metrics is now covered, `rocm dash` cannot read engine metrics
for a key-protected vLLM server and shows a persistent scrape warning,
and `rocm bench load`, which sends no key, cannot benchmark one. The
README says so, and that the every-route guarantee is the vLLM
engine's: AIM runs the same vLLM server without this check, so routes
such as /invocations, /tokenize and /metrics still answer there without
the key. The manual-testing, testing and architecture docs
describe the check; manual-testing sections after the new one are
renumbered, and the three places that cited one of them by number now
cite it by title.

Tested by driving the shipped middleware per route under python3 with a
stdlib ASGI harness (including CORS-preflight look-alikes and a
WebSocket /health), by asserting the launch argv and environment, and
end to end by serve-61 (`@id:serve-vllm-key-guards-every-route`), a
`@nightly` GPU scenario that runs in the nightly runs on Instinct GPUs,
which serve with vLLM (the MI300X job in nightly.yml and the shared GPU
queue's Instinct lanes). A real keyed vLLM server refuses
keyless calls on those routes, accepts keyed ones, and still answers
/health. The e2e step passes the key through the environment, with a
new time-bounded run helper, so it stays out of the recorded command
log.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/vllm-key-on-every-route branch from d35b6cc to 13a9aa6 Compare November 17, 2026 11:33
@rominf
rominf marked this pull request as ready for review November 17, 2026 11:54
@rominf
rominf requested a review from a team as a code owner November 17, 2026 11:54
@rominf
rominf requested a review from siloteemu November 17, 2026 11:54

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read this diff and found nothing blocking. The every-route key check for vLLM matches what the PR body describes, the tests cover the new routes, and the diff is clean of internal references. CI is green apart from a CodeQL job that was still running when I looked. This is a diff-only read; I did not build or run anything.

Two small documentation points are left inline. Beyond those, a few things for you to decide or confirm:

  • Please confirm the middleware behaves as intended on the vLLM releases we pin; I did not check that.
  • /metrics now needs the key, so rocm dash loses engine metrics on keyed vLLM servers. The README says so, which seems fine.
  • /health stays open, and AIM stays unprotected until #1007.
  • The manual-testing renumber leaves two ## 12. headings, inherited from two ## 11. headings.
  • The launch wiring in spawn_vllm_server is covered only by the nightly GPU scenario (serve-61), so it will get no pre-merge coverage until those runners are back.

Comment thread README.md
passing `--api-key`, so the secret stays out of your shell history and the
process table.

With the vLLM engine, an endpoint bound to a public interface

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding that a key-protected vLLM service that is already running keeps its old launch (without the every-route check) until it is restarted. rocm serve reuses it because the middleware is not part of the reuse match, so after upgrading, users should restart keyed vLLM services.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed that it's real: managed_service_reuse_refusal in apps/rocm/src/service_launch/launch_transaction.rs matches on the recipe, requires_api_key and device policy, so a keyed vLLM service started before the upgrade is reused without the middleware until it is restarted. I've left the README unchanged in this PR so the approved diff stays as reviewed; the PR description carries it as an upgrade note ("Restart key-protected vLLM services after upgrading") for the release notes. The latest push (fb7fa46) only merges main and changes no behaviour.

/// primitive only carries path-valued env overrides, so the key never reaches it.
/// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds
/// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter
/// The vllm engine enforces auth wherever it runs (`VLLM_API_KEY` plus the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"wherever it runs" conflicts with docs/testing.md, which now says Linux and WSL. vLLM serving is Linux/WSL-only, so I'd say that here (this check exempts vllm on every platform, and the middleware tests are cfg(linux)).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed. The sentence names the vllm engine, and that engine only runs on Linux and WSL: engines/vllm/src/runtime.rs refuses native Windows, and docs/testing.md says "vLLM (Linux and WSL, ...)" in the same paragraph. "Wherever it runs" is true as written; this function exempts vllm on every platform because there is no platform where it starts without the key check. fb7fa46 (merge of main) leaves it as is.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

come# Review — no blocking findings

Full review of the whole change (base 9e50eac5a03df41a2f23e73dac3a7ec470ea78a7 to head 13a9aa617dd757558866c758d956763a6ae420f2).
No ticket named as fixed/closed by this change; the description references issue #1007 as an explicit out-of-scope follow-up (extending the same protection to the AIM engine), and the diff does not touch engines/aim/, consistent with that scoping.

Blocking

None

Non-blocking

  • Already-running key-protected vLLM services silently keep their pre-upgrade launch (no every-route middleware), and nothing shipped tells an operator to restart them — apps/rocm/src/service_launch/launch_transaction.rs:329, README.md:1316
    rocm serve's reuse check is if require_api_key && !existing.requires_api_key — a bare boolean "did the existing record ask for a key," with no awareness of whether that process has this PR's middleware. README.md, docs/testing.md and docs/manual-testing.md all document the general reuse-matching rule but never mention restarting a keyed vLLM service after upgrading rocm-cli; only the PR description's own "Upgrade note (not in the docs)" states it. Confidence 85 · architectural · Fix: add one sentence near README.md:1316 (or the new "vLLM Endpoint Key Coverage" doc section) telling operators to restart already-running key-protected vLLM services after upgrading.
  • A doc comment says the vLLM engine "enforces auth wherever it runs," overstating against this PR's own Linux/WSL-only scoping elsewhere — apps/rocm/src/service_launch/endpoint_auth.rs:101
    docs/testing.md's new text and the middleware's own Rust test (#[cfg_attr(not(target_os = "linux"), ignore)]) both scope this to Linux/WSL. Not functionally wrong — vLLM is already rejected on native Windows elsewhere (engines/vllm/src/runtime.rs:176) — just inconsistent wording. Confidence 80 · mechanical · Fix: say "wherever it runs (Linux and WSL)".
  • _exempt's scope["type"] != "http" guard has no test proving it necessary — engines/vllm/src/rocm_cli_vllm_auth.py:96
    For a websocket scope, _is_cors_preflight and the UNAUTHENTICATED_ROUTES method check already fail closed (a websocket scope carries no "method" key), so removing this line changes no real ASGI behavior; the Rust test harness's own comment acknowledges this exact gap. Not a bypass, just redundant defense-in-depth no test independently exercises. Confidence 75 · mechanical · Fix: drop the now-provably-redundant line, or comment that it's deliberate belt-and-suspenders.

Decisions for the author

  • Whether rocm serve's reuse check should become middleware-aware (forcing a restart) instead of relying on a manual one — tradeoff. Auto-restarting a live inference server the moment a keyed relaunch is requested is invasive (drops in-flight requests, defeats "reuse" for availability) and arguably riskier than documenting the manual step; the current choice is reasonable, but worth confirming deliberate given the finding above.

Positive signals

  • _digest() hashes both sides to a fixed-length SHA-256 digest before hmac.compare_digest, deliberately removing the timing leak from comparing variable-length secrets directly.
  • The Rust test drives the actual shipped file content (via include_str!) under a real python3 subprocess rather than a reimplemented mock — verified this myself by extracting the file and running an equivalent harness directly; every claimed behavior held (every non-exempt route 401s without the key, the CORS-preflight exemption requires both headers plus OPTIONS, the health exemption is GET/HEAD-only and path-exact, a WebSocket closes before the handshake).
  • vllm_server_command consolidates middleware/env/PYTHONPATH wiring into one function with both a keyed and an unkeyed test asserting presence and absence together — exactly the shape that catches a one-sided mutation (e.g. add --middleware but forget VLLM_API_KEY).
  • The e2e step assertions (assert_other_routes_refused, etc.) panic loudly on a request/connection failure rather than silently miscounting it as a 401, so the nightly scenario cannot pass vacuously if the server never came up.

Deployment notes

  • /metrics now requires the key, so rocm dash cannot read engine metrics from a key-protected vLLM server (shows a persistent vllm scrape: ... 401 warning); documented in README.
  • rocm bench load sends no API key and so cannot benchmark a key-protected endpoint (pre-existing limitation, now documented).
  • A key-protected vLLM service already running before an upgrade keeps answering without the every-route check until restarted (see the first non-blocking finding above).

What this covered

Read completely: engines/vllm/src/rocm_cli_vllm_auth.py and engines/vllm/src/endpoint_auth.rs (by an independent fan-out worker, and separately by me directly, including extracting and running the shipped middleware under python3); engines/vllm/src/process.rs, engines/vllm/src/lib.rs, apps/rocm/src/service_launch/endpoint_auth.rs, xtask/src/ascii_output_contract.rs, all five changed tests/e2e-cucumber files, and README.md/docs/architecture.md/docs/manual-testing.md/docs/testing.md in full, plus surrounding code for impact analysis (apps/rocm/src/service_launch/launch_transaction.rs, spawn.rs, crates/rocm-dash-daemon/src/runner.rs). Commits 9e50eac5a03df41a2f23e73dac3a7ec470ea78a7…13a9aa617dd757558866c758d956763a6ae420f2.

Fan-out: one of four planned independent workers completed (the middleware + Rust-harness group). The other three groups (process.rs/service_launch/xtask; the e2e-cucumber group; the docs group) could not be spawned after two attempts each — concurrent-subagent capacity was exhausted by unrelated concurrent work for the whole run — so I reviewed those three groups myself directly instead of dropping them; they are not independent passes, though every file in them was read in full. Agent-instruction adherence and cross-file consistency were folded into those direct passes (checked against this repo's doc-sync, architecture-doc, and commit-message rules) rather than run as separate dedicated passes; a "prior changes on these files" pass found nothing to check (the touched files are new or have no relevant prior review history). CI was confirmed via the GitHub API: all checks on the current head commit are green (CodeQL's "neutral" conclusion is its normal non-finding state).

Reconciled, in part: after finishing the above, I read the one existing submitted review (APPROVE) and its two inline comments. Both of its points — the undocumented restart-after-upgrade gap, and the "wherever it runs" wording — match two of my findings above, which I had reached independently before reading them, and verified further against the code myself. No other discussion existed to reconcile against.


🤖 Automated by an AI agent

Comment thread README.md
passing `--api-key`, so the secret stays out of your shell history and the
process table.

With the vLLM engine, an endpoint bound to a public interface

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: a key-protected vLLM service already running before an upgrade keeps its old launch (no every-route middleware) until it is restarted — rocm serve's reuse check (apps/rocm/src/service_launch/launch_transaction.rs:329, if require_api_key && !existing.requires_api_key) is a bare boolean "did the existing record ask for a key" test with no awareness of whether that process actually has this middleware. Nothing in README.md, docs/testing.md or docs/manual-testing.md tells an operator to restart a keyed vLLM service after upgrading — only the PR description's own "Upgrade note (not in the docs)" mentions it. Suggest adding one sentence here (or in the new "vLLM Endpoint Key Coverage" manual-testing section) telling operators to restart already-running key-protected vLLM services after upgrading.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same point as the thread above, and the same answer: confirmed against managed_service_reuse_refusal, kept out of this PR's docs so the approved diff is unchanged, and stated as an upgrade note in the PR description. fb7fa46 is a merge of main only.

/// primitive only carries path-valued env overrides, so the key never reaches it.
/// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds
/// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter
/// The vllm engine enforces auth wherever it runs (`VLLM_API_KEY` plus the

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: "enforces auth wherever it runs" overstates against this change's own Linux/WSL-only scoping elsewhere (docs/testing.md's new text, and the #[cfg_attr(not(target_os = "linux"), ignore)] comment on the middleware's Rust test). Not functionally wrong — vLLM is already rejected on native Windows elsewhere (engines/vllm/src/runtime.rs:176), so there's no real gap — but the wording is inconsistent with the rest of the PR. Suggest "wherever it runs (Linux and WSL)" to match.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the thread above: left as is, because the vllm engine runs only on Linux and WSL (engines/vllm/src/runtime.rs rejects native Windows), so "wherever it runs" already means that. No change in fb7fa46, which only merges main.

return b"origin" in names and b"access-control-request-method" in names

def _exempt(self, scope):
if scope["type"] != "http":

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (test-coverage nitpick): this scope["type"] != "http" guard isn't independently exercised by any mutation — for a websocket scope, _is_cors_preflight and the UNAUTHENTICATED_ROUTES method lookup already fail closed (a websocket scope carries no "method" key), so removing this line changes no real ASGI behavior. The test harness's own comment in engines/vllm/src/endpoint_auth.rs acknowledges this exact gap. Not a bypass — just redundant defense-in-depth that no test proves necessary. Either drop it or leave a comment noting it's deliberate belt-and-suspenders rather than load-bearing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, it isn't load-bearing today: a websocket scope has no method, so the /health and CORS-preflight checks already fail closed without it. It's deliberate belt-and-braces, so _exempt stays HTTP-only even if a future ASGI server puts a method on websocket scopes. The harness comment in engines/vllm/src/endpoint_auth.rs records that this guard is equivalent under current behaviour. Not changed in this PR; fb7fa46 only merges main.

@rominf
rominf requested a review from juhovainio November 17, 2026 13:37
Brings the approved branch current with main without rewriting it.
Both sides added a section to docs/testing.md at the same place; both
are kept. Main's new manual-testing section on `rocm fix` steps that do
not finish is renumbered 17 -> 18, after this branch's inserted
section 7 shifted the ones before it.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf added this pull request to the merge queue Nov 17, 2026
Any commits made after this event will not be merged.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (base bfff5216b50ba997e380a90daf3d356dfb44e3f0 to head fb7fa46eb33ab58ec01ab26d16a3fc4b080e6e58).
No ticket closed by this change. #1007 is a draft follow-up pull request (not a plain issue) that depends on this PR's branch and extends the same every-route key middleware to the AIM engine — the description's own claim that AIM is out of scope here, tracked separately, is accurate.

Blocking

None

Non-blocking

  • The manual-testing renumber carries forward a pre-existing duplicate-heading bug instead of fixing it — docs/manual-testing.md:1770, docs/manual-testing.md:1835
    The base branch already has two ## 11. headings ("AIM (Docker) Engine Verification" and "Background Automation Helper"); inserting the new section 7 here shifts both to ## 12., still duplicated. The diff directly edits both exact lines already, so fixing the duplication would cost nothing extra. Confidence 85 · mechanical · Fix: renumber "Background Automation Helper" (and everything after it) to ## 13. onward.

Decisions for the author

  • Three points a reviewer already raised inline, which the author declined to change — tradeoff, confirmed independently rather than re-litigated:
    • An already-running key-protected vLLM service keeps its pre-upgrade launch (no every-route middleware) until restarted, since rocm serve's reuse check (managed_service_reuse_refusal in apps/rocm/src/service_launch/launch_transaction.rs) matches on requires_api_key, not on whether the middleware is installed, and none of README.md/docs/testing.md/docs/manual-testing.md state the restart requirement (only the PR description's "Upgrade note" does). Traced the reuse-check myself and confirm the gap is real; the author's choice to track it as a release note rather than expand this PR's own diff is reasonable.
    • The ensure_public_bind_engine_supported doc comment "the vllm engine enforces auth wherever it runs" reads as a platform-universal claim next to docs/testing.md's explicit "Linux and WSL" wording for the same fact. Confirmed engines/vllm/src/runtime.rs refuses native Windows unconditionally, so the sentence is true as written (vLLM never runs anywhere the check wouldn't apply) even though the phrasing invites misreading.
    • rocm_cli_vllm_auth.py's _exempt has a scope["type"] != "http" guard before the CORS/health checks; traced it myself and confirmed a websocket scope has no "method" key, so _is_cors_preflight and the UNAUTHENTICATED_ROUTES lookup already fail closed without this guard — it is deliberate, provably-redundant defense-in-depth today, not a live gap.

Positive signals

  • _digest() hashes both sides to a fixed-length SHA-256 digest before hmac.compare_digest, removing the timing leak from comparing variable-length secrets directly — traced by hand and confirmed by an independent pass.
  • write_module stages into a NamedTempFile in the target directory and persists it, so a concurrent launch can never observe a half-written middleware module.
  • The Rust test harness drives the actual shipped file content (include_str!) under a real python3 subprocess, not a reimplemented mock, and covers per-route 401s, root_path stripping edge cases, CORS-preflight look-alikes, and WebSocket closure before handshake.
  • vllm_server_command consolidates the middleware/env/PYTHONPATH wiring into one function with both a keyed and an unkeyed test asserting presence and absence of all three protections together — the shape that catches a one-sided mutation.
  • The new e2e step assertions (assert_other_routes_refused, etc.) panic loudly on a request/connection failure rather than silently miscounting it as a 401, so the scenario cannot pass vacuously if the server never came up.

Deployment notes

  • /metrics now requires the key, so rocm dash cannot read engine metrics from a key-protected vLLM server (documented in README).
  • rocm bench load sends no API key and so cannot benchmark a key-protected endpoint (pre-existing limitation, now documented).
  • A key-protected vLLM service already running before an upgrade keeps answering without the every-route check until restarted (see the first "Decisions for the author" item above; not in the shipped docs).

What this covered

Read completely: engines/vllm/src/rocm_cli_vllm_auth.py, engines/vllm/src/endpoint_auth.rs and engines/vllm/src/lib.rs (by an independent fan-out worker, and separately by me directly, tracing the auth/exemption logic by hand given this change's security sensitivity); engines/vllm/src/process.rs, apps/rocm/src/service_launch/endpoint_auth.rs and xtask/src/ascii_output_contract.rs (by a second independent worker, plus apps/rocm/src/service_launch/launch_transaction.rs and engines/vllm/src/runtime.rs by me for impact analysis); all five changed tests/e2e-cucumber files (by a third independent worker); README.md, docs/architecture.md, docs/manual-testing.md and docs/testing.md in full, by me directly. Commits bfff5216…fb7fa46e.

Fan-out: four groups were planned; three ran as independent workers. The fourth (docs) failed to spawn twice on a concurrent-subagent limit, so per the protocol for a worker that never returns, I read and judged that group myself directly instead of dropping it — it is not an independent pass. Agent-instruction adherence was checked against this repo's OSS-contribution AGENTS.md (user-observable behaviour needs a Gherkin scenario; a new test is shown to fail against the code it covers; platform-gated code gets a test on its gate) and is satisfied by the new serve-61 scenario and the Rust harness that runs the shipped module for real. History and prior-changes passes did not run: these files are new or have no prior review history to check.

CI was confirmed via the GitHub API: all 40 checks on the current head are success, neutral (CodeQL) or an intentionally-skipped job.

A process note on blindness: Step 0's mandatory check for a concurrent claim required reading the PR's existing reviews before this pass began, which exposed their text early. Every finding above was independently re-derived and verified against the source (traced by hand or by an independent worker) rather than imported from that text, and the three "Decisions for the author" items were cross-checked against the author's own replies to those threads, which this review's independent tracing confirms are accurate.


🤖 Automated by an AI agent

Comment thread docs/manual-testing.md
back to another engine.

## 11. Background Automation Helper
## 12. Background Automation Helper

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this renumber (inserting the new section 7 and shifting everything after it by one) carries forward a pre-existing duplicate-number bug rather than fixing it. The base branch already had two ## 11. headings ("AIM (Docker) Engine Verification" and "Background Automation Helper"); after this diff they are both ## 12. (line 1770 and this line, 1835). Since the diff directly edits both of these exact headings to renumber them, this would be a one-line fix to take while already touching the lines: bump "Background Automation Helper" (and everything after it) to ## 13. onward.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants