Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 67 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions MANIFEST.md
Original file line number Diff line number Diff line change
Expand Up @@ -538,6 +538,10 @@ repository.
| utf-8 | 0.7.6 | MIT OR Apache-2.0 |
| utf8_iter | 1.0.4 | Apache-2.0 OR MIT |
| utf8parse | 0.2.2 | Apache-2.0 OR MIT |
| utoipa | 6.0.0 | MIT OR Apache-2.0 |
| utoipa-gen | 6.0.1 | MIT OR Apache-2.0 |
| utoipa-swagger-ui | 10.0.2 | MIT OR Apache-2.0 |
| utoipa-swagger-ui-vendored | 0.2.0 | MIT OR Apache-2.0 |
| uuid | 1.23.3 | Apache-2.0 OR MIT |
| valuable | 0.1.1 | MIT |
| version_check | 0.9.5 | MIT/Apache-2.0 |
Expand All @@ -553,6 +557,7 @@ repository.
| wasm-bindgen-macro | 0.2.125 | MIT OR Apache-2.0 |
| wasm-bindgen-macro-support | 0.2.125 | MIT OR Apache-2.0 |
| wasm-bindgen-shared | 0.2.125 | MIT OR Apache-2.0 |
| wasm-streams | 0.4.2 | MIT OR Apache-2.0 |
| wasm-streams | 0.5.0 | MIT OR Apache-2.0 |
| web-sys | 0.3.102 | MIT OR Apache-2.0 |
| web-time | 1.1.0 | MIT OR Apache-2.0 |
Expand Down Expand Up @@ -775,6 +780,14 @@ With the prek hooks installed this runs automatically whenever you commit a
dependency-graph change, so the notices rarely go stale by hand; the hook skips
if cargo-about is absent or not the pinned version.

Assets compiled in that are not crates — today, the Swagger UI bundle behind
`rocm services docs` — are listed in a "Bundled web assets" section maintained
by hand at the end of `about.hbs`. `cargo-about` cannot see them, so their
licenses bypass the `about.toml` check: when a dependency bump changes such an
asset, update that section from the asset's own upstream notices. A test in
`rocmd` fails when the served Swagger UI version is not the one the notices
name.

CI verifies the committed file is current with `cargo xtask tpn --check`; a new
dependency under a license not listed in `about.toml` fails the check until the
license is reviewed and added (or the dependency dropped). CI pins the
Expand Down
46 changes: 44 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1251,8 +1251,9 @@ the running service, or stop it first to serve on the new port.

By default the server runs in the background under ROCm CLI's supervision and
prints a deployment summary: a progress indicator while it starts, then a table
with the status, the full inference endpoint, the API-qualified model name, and a
quick smoke test (time to first token and approximate tokens per second). Control
with the status, the engine, the API-qualified model name, the full inference
endpoint, the command that serves the server's API docs (see API docs below),
and a quick smoke test (time to first token and approximate tokens per second). Control

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (mechanical, confidence 85): this line and two others touched by this PR break the surrounding paragraphs' ~75-80 column wrap convention: this line runs to 87 columns, line 1585 ("...vLLM also documents its full API at") to 100, and line 1590 ("...work offline. They use port 11430, or a free port when 11430 is") to 107. Purely cosmetic — worth re-wrapping to match the rest of the file.

returns to your shell with the server still running; manage it later with `rocm
services` (below).

Expand Down Expand Up @@ -1477,6 +1478,7 @@ Manage background servers started with `--managed`:

```
rocm services list [--all] [--json]
rocm services docs [<service-id>] [--port <port>]
rocm services logs <service-id> [--follow]
rocm services stop <service-id> --yes
rocm services restart <service-id> --yes
Expand Down Expand Up @@ -1549,6 +1551,46 @@ started while a `prune` is scanning waits for the prune.
`--json` prints the service records verbatim, for scripting and for the remote
orchestration below.

#### API docs

`rocm services docs` serves interactive API docs for the running model servers,
in the foreground, until you press Ctrl+C. `rocm serve` names the command next
to the endpoint, and `rocm services list` repeats it for each running server
(`--json` prints the records alone, without it):

```
endpoint http://127.0.0.1:11435/v1/chat/completions
docs rocm services docs <service-id>
```

Run it to get the links:

```
$ rocm services docs <service-id>
API docs for running model servers: http://127.0.0.1:11430/
<service-id> (<model>): http://127.0.0.1:11430/services/<service-id>/docs/
Press Ctrl+C to stop serving the docs; the model servers keep running.
```

Without a service id it links every running server. It refuses, and serves
nothing, when the named server is not running or when no server is.

The page lists the routes the server answers (the model list, chat and text
completions, and the engine's health check), with request examples already
filled in with the served model. "Try it out" sends the request through the
docs page to the server, so it works for every engine without enabling
cross-origin access on the server; for a server started with an endpoint key,
enter the key under "Authorize" (it is kept only until the page is closed).
The root page lists every running server, judged as `rocm services list` judges
it; a server that stops or exits drops off about a second later. vLLM also documents its full API at
its own `/docs` page: `rocm serve` and `rocm services list` print it as
`engine docs`, and the docs page links to it.

The docs listen on 127.0.0.1 only, answer browser requests only from their own
pages, cannot be framed by other sites, and work offline. They use port 11430, or a free port when 11430 is
taken; `--port <port>` picks one and fails if it is taken. Nothing keeps
running after Ctrl+C, and the model servers are never touched.

### Remote machines (preview)

Run a model on a different GPU machine and reach it from your own. Both machines
Expand Down
Loading
Loading