Repository navigation
Conversation
rominf
force-pushed
the
fix/vllm-key-on-every-route
branch
from
November 17, 2026 09:40
fabe4a2 to
bf532fa
Compare
4 tasks done
rominf
force-pushed
the
fix/vllm-key-on-every-route
branch
3 times, most recently
from
November 17, 2026 11:33
d35b6cc to
13a9aa6
Compare
AIM containers bundle vLLM's OpenAI server, and a keyed `rocm serve --engine aim` handed the key to it only as `VLLM_API_KEY`. vLLM checks that key under a fixed set of path prefixes, so routes such as /invocations, /tokenize, /detokenize and /metrics answered without it, the same gap the native vLLM engine had. Move the every-route middleware and the code that writes it into rocm-engine-protocol so both engines use the one copy. A keyed AIM launch now writes it under the engine's data directory, bind-mounts that directory read-only into the container, prepends it to the container's PYTHONPATH and adds "middleware" to AIM_ENGINE_ARGS, all through the env file rather than the docker command line. Values the caller or the image already gave PYTHONPATH and AIM_ENGINE_ARGS are kept; AIM_ENGINE_ARGS that is not a JSON object or already names a middleware is refused. The container's vLLM is not under rocm-cli's control, so once a keyed container is ready the supervisor calls POST /tokenize without the key and, unless that is refused with 401, removes the container and fails the start instead of serving with only /v1 protected. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Moving the middleware writer into rocm-engine-protocol brought its error paths with it, untested, and dropped the crate below its coverage floor. A launch that cannot write the module must fail naming the path rather than start a server without the middleware it asked for, so test that each failure we can provoke says which path it could not create, stage in or write. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A keyed AIM launch writes PYTHONPATH and AIM_ENGINE_ARGS to the container's env file, which replaces the image's own values. When the image's config could not be read, the launch carried on with an empty view of it and silently dropped those values. It now refuses instead, saying why the config could not be read, and treats a `null` config the same way. A refusal over an AIM_ENGINE_ARGS that cannot carry the middleware now names the step that clears it for where the value came from: the launch's own value, one forwarded from this process's environment by a bare --engine-env, or the image's, which can only be overridden whole. Tests now cover the image's env surviving a keyed launch, both refusals and that each named remedy clears them, and a mount path split by `,` or `"` in either the source or the target. Comments and docs that said every AIM start probes the routes now say the `rocm serve` path does, and the protocol module notes the CORS preflight exemption. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-62 is a third @nightly AIM scenario with a 1200s readiness cap, and each AIM container downloads its checkpoint itself, so it adds up to one more 1200s wait to the Strix Halo jobs that run the AIM scenarios. Raise the nightly Strix lane from 120 to 140 minutes and the include_nightly Strix Ubuntu dispatch from 105 to 125, and name serve-62 wherever those budgets and the Docker check list the AIM scenarios. The nightly comment no longer claims serve-27 has an xfail row; it runs expect-pass. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
rominf
force-pushed
the
fix/aim-key-on-every-route
branch
from
November 17, 2026 12:18
dfd1ded to
48dea3f
Compare
Contributor
Security scan found no critical or high findings on lines changed by this pull requestView the security scan run for complete reports. “New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #974 (this PR targets its branch,
fix/vllm-key-on-every-route); the commits to review are the four after it.Why
rocm serve --engine aimprotects an endpoint with an API key when it binds publicly (--allow-public-bind), when--require-api-keyis given, and underrocm remote serve. AIM containers bundle vLLM's OpenAI server, and the key reached that server only asVLLM_API_KEY. vLLM checks that key only under a fixed set of path prefixes (/v1and a few others), so routes such as/invocations,/tokenize,/detokenizeand/metricson a key-protected AIM endpoint answered clients that did not present the key. #974 closes the same gap for the native vLLM engine with an every-route ASGI middleware; this change applies that middleware to AIM containers.What changes
rocm_cli_vllm_auth.py) and the code that writes it move from the vLLM engine intorocm-engine-protocol(endpoint_auth.rs), so both engines import one copy. The native vLLM engine keeps its behaviour: same module, class,--middlewareargument andPYTHONPATHhandling. The module is now written world-readable (file0644, directory0755) so a container user other than its writer can import it; it holds no secret./opt/rocm-cli/python(--mount type=bind,...,readonly; a source or target path containing,or"is refused rather than split into a different mount);VLLM_API_KEY, that directory prepended toPYTHONPATH, and"middleware": "rocm_cli_vllm_auth.RequireEndpointApiKey"added toAIM_ENGINE_ARGS. Nothing new goes on thedockercommand line, and the key never reaches argv.PYTHONPATHandAIM_ENGINE_ARGS, so a keyed launch carries them over behind the added entry; a value the caller passed with--engine-envtakes precedence over the image's.docker image inspectfails or yields no.Config), a keyed launch is refused rather than silently dropping those values.AIM_ENGINE_ARGSthat is not a JSON object, or that already names amiddleware, is refused before anything starts. The message names the step that clears it for where the value came from: change it in--engine-env; for a bare--engine-env AIM_ENGINE_ARGSforwarded from the environment, fix it there or pass an explicit value; for the image's value, override it whole.rocm servechecks the result. The container's vLLM is not under rocm-cli's control, so once a keyed container is readyserve_http(the pathrocm serveandrocmdtake) sendsPOST /tokenizewithout the key. Unless that is refused with401, it removes the container and fails the start, rather than serving with only/v1protected. The engine's lower-levellaunchmethod returns before the container is ready and installs the middleware the same way without this check.@nightlyAIM serve with a 1200s readiness cap, so the Strix Halo nightly job's cap goes from 120 to 140 minutes and theinclude_nightlyStrix Ubuntu dispatch from 105 to 125.docs/testing.md,docs/manual-testing.md(an AIM check in the AIM engine section) anddocs/architecture.mddescribe the shared module and the AIM wiring.User-visible behaviour change:
rocm serve/invocations,/tokenize,/detokenize,/metricswithout the key401/v1/...without the key401401GET/HEAD /health, CORS preflightsAIM_ENGINE_ARGS/v1protectedAIM_ENGINE_ARGSnot a JSON object, or settingmiddlewareUnkeyed (loopback, no
--require-api-key) AIM launches are unchanged: no mount, no extra env, no probe, no new refusal.Tests
rocm-engine-aim): the env built for a keyed launch versus an unkeyed one; keeping the image's and the caller'sPYTHONPATHandAIM_ENGINE_ARGS; eachAIM_ENGINE_ARGSrefusal by source, together with a run showing that the step its message names clears it; the probe verdict; the read-only--mountargument and refusing,or"in either path.docker: a keyedrun_containermounts the directory and writes exactly the mergedAIM_ENGINE_ARGSandPYTHONPATH(image values kept) andVLLM_API_KEYto the env file, with the key absent from argv; an unkeyed run adds no mount and no extra env; a keyed launch whose image config is unreadable ornullstarts no container;serve_httpfails and removes a keyed container whose fake server answers a keyless/tokenize, and serves one that refuses it.rocm-engine-protocol): fix(vllm): require the endpoint key on every server route #974's ASGI-harness tests (per-route401s,root_pathhandling, CORS-preflight look-alikes, WebSocket/health, refusing to construct without a key) move here with the module, plus tests that writing it fails naming the path it could not create, stage in or write, and that it is left readable by other users.nullfilter removed, one mount splitter or field dropped) and went red.@id:serve-aim-key-guards-every-route(serve-62) inmodel_serving.feature, tagged@requires-gpu @requires-engine:aim @serve-timeout:1200 @nightly, with the same steps as fix(vllm): require the endpoint key on every server route #974's serve-61. It serves the AIM canary behind a key and checks that/invocations,/tokenize,/detokenizeand/metricsrefuse a keyless call, that/healthanswers without the key, and that the calls succeed with it. It runs only on the nightly Strix Halo lane, or aninclude_nightlydispatch ofe2e-selfhosted.yml, on a host with Docker — the same Docker caveat as serve-26 and serve-27.aim_serve_scenarios_stay_off_the_per_pr_lanespins its tagging. The new refusals are error paths covered by the subprocess tests above, not by a scenario: reaching them end to end needs a deliberately broken AIM image on a GPU host.Verified locally (Linux, Strix Halo)
cargo fmt --all --check;cargo clippy --workspace --all-targets -- -D warnings;cargo clippy -p e2e-cucumber --test e2e -- -D warnings: clean.cargo test -p rocm-engine-protocol -p rocm-engine-aim -p rocm-engine-vllm: all pass.cargo test --workspace --all-targets --no-fail-fast: 7597 passed, 3 failed in code this change does not touch, each from this host:service_control::tests::stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails(the host'sdockeris podman's emulation, which ignores the test'sDOCKER_HOST) and twobuild_script::tests::git_watch_paths_*tests (their scratchgit commitfails under this host's git configuration).cargo test -p e2e-cucumber --lib --test feature_naming --test skill_reference;cargo xtask check-architecture-doc;cargo xtask check-crate-edges;cargo xtask check-module-size: pass.cargo xtask smoke: every step passes up torocmd sandbox examine, which needs a bubblewrap network namespace this host does not permit.Not verified:
AIM_ENGINE_ARGS{"middleware": ...}into vLLM's--middlewareand passesPYTHONPATHthrough. If it does not, keyedrocm servestarts fail closed through the check above rather than serve unprotected.Design notes
POST /tokenizeis cheap and fails closed on today's vLLM, where/tokenizeis outside vLLM's own key check. A future vLLM that guards/tokenizeitself but not another route would pass it without the middleware; serve-62 checks four routes, but only nightly.401answer fails the start. A transient error right after readiness tears down a healthy container; a retry on transport errors only would keep the guarantee and is left as a follow-up.Possible follow-ups, not done here: carry
docker image inspect's stderr into the unreadable-config refusal; derive the refusal's value source from the same match that resolves the value; check the image config before GPU planning so it is the first error a keyed launch reports.AIM contract sources: aim-build README, aim-build architecture doc.