Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions .github/workflows/e2e-selfhosted.yml
Original file line number Diff line number Diff line change
Expand Up @@ -471,10 +471,11 @@ jobs:
# this replaced, and a run took 26m54s against the old 35min cap (run
# 34489547654) versus 18-19m on a warm static host — keep the same headroom
# ratio the 35 was chosen for.
# An include_nightly dispatch also runs the AIM serve scenarios (serve-26/27,
# @nightly, up to 1200s readiness each on a cold pull) and the large-model
# serve, which cannot fit the 45min budget of an ordinary push run.
timeout-minutes: ${{ inputs.include_nightly && 105 || 45 }}
# An include_nightly dispatch also runs the AIM serve scenarios
# (serve-26/27/62, @nightly, up to 1200s readiness each on a cold pull) and
# the large-model serve, which cannot fit the 45min budget of an ordinary
# push run.
timeout-minutes: ${{ inputs.include_nightly && 125 || 45 }}
runs-on: [self-hosted, linux, devlab-dispatch, strix-halo]
needs: [changes]
# See `e2e-gpu`: no build-and-test gate (cross-workflow); strix-ubuntu.
Expand Down
18 changes: 10 additions & 8 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -512,11 +512,13 @@ jobs:
# Strix Ubuntu job.
e2e-gpu-nightly-strix:
name: E2E tests (Strix Halo, incl. nightly-only)
# 120: serve-26/27 (AIM, @nightly) add up to 2 x 1200s readiness waits; on a
# host without Docker they skip and cost nothing. The "Report whether Docker
# is reachable" step logs the answer each run. serve-27's known failure on
# this pool is an xfail row in expectations.toml (rocm-cli#1058).
timeout-minutes: 120
# 140: serve-26/27/62 (AIM, @nightly) add up to 3 x 1200s readiness waits
# (each container downloads its checkpoint afresh: none is cached between
# scenarios); on a host without Docker they skip and cost nothing. The
# "Report whether Docker is reachable" step logs the answer each run.
# serve-27 once failed here (rocm-cli#1058) and now runs expect-pass, with
# no row in expectations.toml.
timeout-minutes: 140
# The DevLab Dispatch pool, identical to the e2e-gpu-strix-ubuntu lane in
# e2e-selfhosted.yml, which runs on push and dispatch but not on pull_request
# or merge_group (pinned by workflow_contract.rs). This lane used to target
Expand Down Expand Up @@ -563,14 +565,14 @@ jobs:
fi
echo "$HOME/.cargo/bin" >> "$GITHUB_PATH"

# serve-26/27 (AIM) need Docker reachable on this host; without it they skip
# serve-26/27/62 (AIM) need Docker reachable on this host; without it they skip
# silently. Say which it is, so a nightly that never ran them is visible.
- name: Report whether Docker is reachable (AIM serve scenarios need it)
run: |
if docker info >/dev/null 2>&1; then
echo "docker reachable: serve-26/27 (AIM) can run on this host"
echo "docker reachable: serve-26/27/62 (AIM) can run on this host"
else
echo "::warning::docker is not reachable on this host: serve-26/27 (AIM) will skip"
echo "::warning::docker is not reachable on this host: serve-26/27/62 (AIM) will skip"
fi

- name: Run E2E tests on Strix Halo (incl. nightly-only)
Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

20 changes: 20 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1313,6 +1313,26 @@ directory that only your user account can read. Set `ROCM_SERVE_API_KEY` rather
passing `--api-key`, so the secret stays out of your shell history and the
process table.

With the vLLM and AIM engines, an endpoint bound to a public interface
(`--allow-public-bind`), or served with `--require-api-key`, answers only
clients that present its API key as an `Authorization: Bearer` header, on every
route the server answers, not just the OpenAI `/v1` API — including vLLM's own
`/invocations`, `/tokenize` and `/metrics` — except `GET`/`HEAD /health` and
CORS preflights: the `/health` liveness check stays open. An AIM container gets
this check through `AIM_ENGINE_ARGS`; one whose image does not apply it fails to
start and its container is removed, rather than serving with only `/v1`
protected. A key-protected AIM launch also refuses to start when it cannot read
the image's config, or when `AIM_ENGINE_ARGS` (from `--engine-env` or the image)
is not a JSON object or already names a `middleware`.

Because `/metrics` is covered, `rocm dash` cannot read engine metrics from a
key-protected vLLM server: it leaves that server's KV-cache, queue, throughput
and latency figures empty and keeps showing a warning such as
`vllm scrape: 1 failed (last: transport: GET http://<host>:<port>/metrics: status 401 Unauthorized)`.

`rocm bench load` sends no API key, so it cannot benchmark a key-protected
endpoint on any engine.

`--temperature` (>= 0.0), `--top-p` (0.0-1.0), and `--max-tokens` (> 0) set
server-wide sampling defaults for the launched engine. They apply only to
`vllm` and `lemonade`; other engines reject them. For vLLM they are folded
Expand Down
9 changes: 6 additions & 3 deletions apps/rocm/src/service_launch/endpoint_auth.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,9 +98,12 @@ pub(crate) fn drop_orphaned_endpoint_key_on_already_running(
/// The one such case today: Windows managed Lemonade. Its server reads the
/// value-typed `LEMONADE_API_KEY` env var, but the Windows detached-spawn
/// primitive only carries path-valued env overrides, so the key never reaches it.
/// vLLM enforces auth on every platform (`VLLM_API_KEY`), and loopback binds
/// (`public_bind == false`) need no key — both pass. `is_windows` is a parameter
/// so both branches are unit-testable off-Windows.
/// The vllm and aim engines enforce auth wherever they run (`VLLM_API_KEY` plus
/// the every-route middleware in `crates/rocm-engine-protocol/src/endpoint_auth.rs`,
/// which an AIM serve through `serve_http` — the path `rocm serve` takes —
/// verifies before serving), and loopback binds (`public_bind == false`) need no
/// key — both pass. `is_windows` is a parameter so both branches are
/// unit-testable off-Windows.
pub(crate) fn ensure_public_bind_engine_supported(
engine: &str,
public_bind: bool,
Expand Down
1 change: 1 addition & 0 deletions crates/rocm-engine-protocol/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ anyhow.workspace = true
rocm-core = { path = "../rocm-core" }
serde.workspace = true
serde_json.workspace = true
tempfile = "3"

[dev-dependencies]
# The leftover a killed owner-only key store leaves, for the tests of the
Expand Down
Loading
Loading