Skip to content

fix(aim): require the endpoint key on every server route - #1007

Draft
rominf wants to merge 4 commits into
fix/vllm-key-on-every-routefrom
fix/aim-key-on-every-route
Draft

rominf wants to merge 4 commits into
fix/vllm-key-on-every-routefrom
fix/aim-key-on-every-route

Conversation

@rominf

@rominf rominf commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Depends on #974 (this PR targets its branch, fix/vllm-key-on-every-route); the commits to review are the four after it.

Why

rocm serve --engine aim protects an endpoint with an API key when it binds publicly (--allow-public-bind), when --require-api-key is given, and under rocm remote serve. AIM containers bundle vLLM's OpenAI server, and the key reached that server only as VLLM_API_KEY. vLLM checks that key only under a fixed set of path prefixes (/v1 and a few others), so routes such as /invocations, /tokenize, /detokenize and /metrics on a key-protected AIM endpoint answered clients that did not present the key. #974 closes the same gap for the native vLLM engine with an every-route ASGI middleware; this change applies that middleware to AIM containers.

What changes

  • One shared middleware. The middleware module (rocm_cli_vllm_auth.py) and the code that writes it move from the vLLM engine into rocm-engine-protocol (endpoint_auth.rs), so both engines import one copy. The native vLLM engine keeps its behaviour: same module, class, --middleware argument and PYTHONPATH handling. The module is now written world-readable (file 0644, directory 0755) so a container user other than its writer can import it; it holds no secret.
  • How a keyed AIM launch installs it:
    • the module is written under the AIM engine's data directory, and that directory is bind-mounted read-only into the container at /opt/rocm-cli/python (--mount type=bind,...,readonly; a source or target path containing , or " is refused rather than split into a different mount);
    • the container's env file gets VLLM_API_KEY, that directory prepended to PYTHONPATH, and "middleware": "rocm_cli_vllm_auth.RequireEndpointApiKey" added to AIM_ENGINE_ARGS. Nothing new goes on the docker command line, and the key never reaches argv.
  • The image's and the caller's values are kept, or the launch is refused. The env file replaces the image's own PYTHONPATH and AIM_ENGINE_ARGS, so a keyed launch carries them over behind the added entry; a value the caller passed with --engine-env takes precedence over the image's.
    • If the image's config cannot be read (docker image inspect fails or yields no .Config), a keyed launch is refused rather than silently dropping those values.
    • An AIM_ENGINE_ARGS that is not a JSON object, or that already names a middleware, is refused before anything starts. The message names the step that clears it for where the value came from: change it in --engine-env; for a bare --engine-env AIM_ENGINE_ARGS forwarded from the environment, fix it there or pass an explicit value; for the image's value, override it whole.
  • rocm serve checks the result. The container's vLLM is not under rocm-cli's control, so once a keyed container is ready serve_http (the path rocm serve and rocmd take) sends POST /tokenize without the key. Unless that is refused with 401, it removes the container and fails the start, rather than serving with only /v1 protected. The engine's lower-level launch method returns before the container is ready and installs the middleware the same way without this check.
  • Nightly CI budget. The new scenario is a third @nightly AIM serve with a 1200s readiness cap, so the Strix Halo nightly job's cap goes from 120 to 140 minutes and the include_nightly Strix Ubuntu dispatch from 105 to 125.
  • Docs. The README's endpoint-key paragraph states the every-route guarantee for both vLLM and AIM and the refusals above. docs/testing.md, docs/manual-testing.md (an AIM check in the AIM engine section) and docs/architecture.md describe the shared module and the AIM wiring.

User-visible behaviour change:

Keyed AIM endpoint served with rocm serve Before After
/invocations, /tokenize, /detokenize, /metrics without the key answered 401
/v1/... without the key 401 401
GET/HEAD /health, CORS preflights open open
image that ignores AIM_ENGINE_ARGS served with only /v1 protected start fails, container removed
AIM_ENGINE_ARGS not a JSON object, or setting middleware passed through refused before start
image config unreadable started refused before start

Unkeyed (loopback, no --require-api-key) AIM launches are unchanged: no mount, no extra env, no probe, no new refusal.

Tests

  • Unit (rocm-engine-aim): the env built for a keyed launch versus an unkeyed one; keeping the image's and the caller's PYTHONPATH and AIM_ENGINE_ARGS; each AIM_ENGINE_ARGS refusal by source, together with a run showing that the step its message names clears it; the probe verdict; the read-only --mount argument and refusing , or " in either path.
  • Subprocess, against a fake docker: a keyed run_container mounts the directory and writes exactly the merged AIM_ENGINE_ARGS and PYTHONPATH (image values kept) and VLLM_API_KEY to the env file, with the key absent from argv; an unkeyed run adds no mount and no extra env; a keyed launch whose image config is unreadable or null starts no container; serve_http fails and removes a keyed container whose fake server answers a keyless /tokenize, and serves one that refuses it.
  • Middleware (rocm-engine-protocol): fix(vllm): require the endpoint key on every server route #974's ASGI-harness tests (per-route 401s, root_path handling, CORS-preflight look-alikes, WebSocket /health, refusing to construct without a key) move here with the module, plus tests that writing it fails naming the path it could not create, stage in or write, and that it is left readable by other users.
  • Seen to fail: each new unit and subprocess test was run against a mutation of the code it covers (guard removed, image env dropped, value source swapped, null filter removed, one mount splitter or field dropped) and went red.
  • e2e: @id:serve-aim-key-guards-every-route (serve-62) in model_serving.feature, tagged @requires-gpu @requires-engine:aim @serve-timeout:1200 @nightly, with the same steps as fix(vllm): require the endpoint key on every server route #974's serve-61. It serves the AIM canary behind a key and checks that /invocations, /tokenize, /detokenize and /metrics refuse a keyless call, that /health answers without the key, and that the calls succeed with it. It runs only on the nightly Strix Halo lane, or an include_nightly dispatch of e2e-selfhosted.yml, on a host with Docker — the same Docker caveat as serve-26 and serve-27. aim_serve_scenarios_stay_off_the_per_pr_lanes pins its tagging. The new refusals are error paths covered by the subprocess tests above, not by a scenario: reaching them end to end needs a deliberately broken AIM image on a GPU host.

Verified locally (Linux, Strix Halo)

  • cargo fmt --all --check; cargo clippy --workspace --all-targets -- -D warnings; cargo clippy -p e2e-cucumber --test e2e -- -D warnings: clean.
  • cargo test -p rocm-engine-protocol -p rocm-engine-aim -p rocm-engine-vllm: all pass.
  • cargo test --workspace --all-targets --no-fail-fast: 7597 passed, 3 failed in code this change does not touch, each from this host: service_control::tests::stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails (the host's docker is podman's emulation, which ignores the test's DOCKER_HOST) and two build_script::tests::git_watch_paths_* tests (their scratch git commit fails under this host's git configuration).
  • cargo test -p e2e-cucumber --lib --test feature_naming --test skill_reference; cargo xtask check-architecture-doc; cargo xtask check-crate-edges; cargo xtask check-module-size: pass.
  • cargo xtask smoke: every step passes up to rocmd sandbox examine, which needs a bubblewrap network namespace this host does not permit.

Not verified:

  • serve-62 against a real AIM image. No AIM image was run, so it is unconfirmed that the AIM runtime turns AIM_ENGINE_ARGS {"middleware": ...} into vLLM's --middleware and passes PYTHONPATH through. If it does not, keyed rocm serve starts fail closed through the check above rather than serve unprotected.
  • Container user and SELinux. The bind mount was not tried with a non-root container user or under SELinux relabelling.
  • The CI caps are an upper bound from the readiness caps, not a measurement.

Design notes

  • The probe samples one route. A single keyless POST /tokenize is cheap and fails closed on today's vLLM, where /tokenize is outside vLLM's own key check. A future vLLM that guards /tokenize itself but not another route would pass it without the middleware; serve-62 checks four routes, but only nightly.
  • The probe makes one attempt. Any transport error or non-401 answer fails the start. A transient error right after readiness tears down a healthy container; a retry on transport errors only would keep the guarantee and is left as a follow-up.

Possible follow-ups, not done here: carry docker image inspect's stderr into the unreadable-config refusal; derive the refusal's value source from the same match that resolves the value; check the image config before GPU planning so it is the first error a keyed launch reports.

AIM contract sources: aim-build README, aim-build architecture doc.

@rominf
rominf force-pushed the fix/vllm-key-on-every-route branch from fabe4a2 to bf532fa Compare November 17, 2026 09:40
@rominf
rominf force-pushed the fix/vllm-key-on-every-route branch 3 times, most recently from d35b6cc to 13a9aa6 Compare November 17, 2026 11:33
AIM containers bundle vLLM's OpenAI server, and a keyed `rocm serve
--engine aim` handed the key to it only as `VLLM_API_KEY`. vLLM checks
that key under a fixed set of path prefixes, so routes such as
/invocations, /tokenize, /detokenize and /metrics answered without it,
the same gap the native vLLM engine had.

Move the every-route middleware and the code that writes it into
rocm-engine-protocol so both engines use the one copy. A keyed AIM
launch now writes it under the engine's data directory, bind-mounts
that directory read-only into the container, prepends it to the
container's PYTHONPATH and adds "middleware" to AIM_ENGINE_ARGS, all
through the env file rather than the docker command line. Values the
caller or the image already gave PYTHONPATH and AIM_ENGINE_ARGS are
kept; AIM_ENGINE_ARGS that is not a JSON object or already names a
middleware is refused.

The container's vLLM is not under rocm-cli's control, so once a keyed
container is ready the supervisor calls POST /tokenize without the key
and, unless that is refused with 401, removes the container and fails
the start instead of serving with only /v1 protected.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Moving the middleware writer into rocm-engine-protocol brought its error
paths with it, untested, and dropped the crate below its coverage floor.
A launch that cannot write the module must fail naming the path rather
than start a server without the middleware it asked for, so test that
each failure we can provoke says which path it could not create, stage
in or write.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
A keyed AIM launch writes PYTHONPATH and AIM_ENGINE_ARGS to the
container's env file, which replaces the image's own values. When the
image's config could not be read, the launch carried on with an empty
view of it and silently dropped those values. It now refuses instead,
saying why the config could not be read, and treats a `null` config
the same way.

A refusal over an AIM_ENGINE_ARGS that cannot carry the middleware now
names the step that clears it for where the value came from: the
launch's own value, one forwarded from this process's environment by a
bare --engine-env, or the image's, which can only be overridden whole.

Tests now cover the image's env surviving a keyed launch, both
refusals and that each named remedy clears them, and a mount path
split by `,` or `"` in either the source or the target. Comments and
docs that said every AIM start probes the routes now say the
`rocm serve` path does, and the protocol module notes the CORS
preflight exemption.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
serve-62 is a third @nightly AIM scenario with a 1200s readiness cap,
and each AIM container downloads its checkpoint itself, so it adds up
to one more 1200s wait to the Strix Halo jobs that run the AIM
scenarios. Raise the nightly Strix lane from 120 to 140 minutes and the
include_nightly Strix Ubuntu dispatch from 105 to 125, and name
serve-62 wherever those budgets and the Docker check list the AIM
scenarios. The nightly comment no longer claims serve-27 has an xfail
row; it runs expect-pass.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the fix/aim-key-on-every-route branch from dfd1ded to 48dea3f Compare November 17, 2026 12:18
@github-actions

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant