Skip to content

feat: serve API docs for running model servers with rocm services docs (ROCMAI-671) - #723

Open
rominf wants to merge 2 commits into
mainfrom
feat/service-api-docs
Open

rominf wants to merge 2 commits into
mainfrom
feat/service-api-docs

Conversation

@rominf

@rominf rominf commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Why

rocm serve hands the user an endpoint URL and nothing that says what it accepts. Of the servers ROCm CLI runs, only vLLM generates API docs; Lemonade's router and the llama.cpp server behind it serve none, so on most hosts there is no reference to read and no way to try a request without writing one by hand.

What changes

  • New rocm services docs [<id>] [--port <port>] serves Swagger UI for the running model servers on 127.0.0.1, in the foreground, until Ctrl+C. / lists the running servers; /services/<id>/docs/ is one server's page; /services/<id>/openapi.json its description. Without an id it links every running server (the docs of every running server stay reachable on that port either way). It refuses, before serving anything, when the named server is not running, naming rocm services restart <id> --yes, or when none is. It uses port 11430, or a free port when 11430 is in use and says so on stderr; --port never falls back, and no other startup failure falls back either. After Ctrl+C nothing is left serving and the model servers are not touched.
  • Which servers are running is judged by the loader rocm services list uses (process and endpoint checks), passed in to the docs server, so a server that stops or crashes drops off about a second later. Requests that arrive together share one read of the servers.
  • rocm serve (summary, plain and attached output) and rocm services list print docs: rocm services docs <id> for each running server, and engine docs: <url>/docs for vLLM. rocm services list --help describes both lines. --json is unchanged.
  • Descriptions are built with utoipa from one route table in rocm-engine-protocol (OPENAI_CHAT_ROUTES plus each engine's entry in ENGINE_HEALTH_ROUTES). The three engines report the same table as supported_routes in their endpoint answer. Examples, the page title and the index label use the name the server answers to (served_model_name), which for AIM is not the image reference recorded as the canonical id; a server started with an endpoint key gets an "Authorize" box, and the key is not persisted in the browser.
  • "Try it out" goes through a same-origin forwarder (/services/<id>/api/...) that streams the answer back, so it works on every engine whatever its cross-origin settings. It passes on only Content-Type, Accept and Authorization, keeps the path percent-encoded, accepts bodies up to 64 MiB, does not follow redirects, gives up after 5 s on a server that does not accept the connection, and bypasses any configured HTTP proxy. A server bound to 0.0.0.0 is reached on 127.0.0.1, one bound to :: on ::1; that mapping is one helper in rocm-core (reachable_host), which the Anthropic proxy now uses as well.
  • Browser requests are admitted only from the docs server's own pages: by Origin, or by Sec-Fetch-Site (same-origin/none) when there is no Origin. The Host guard refuses non-loopback names (DNS rebinding). Every answer carries frame-ancestors 'none', X-Frame-Options: DENY and nosniff; forwarded answers are also CSP-sandboxed. Ids are percent-encoded in links and escaped in pages.
  • ManagedServiceRecord gains local_base_url and engine_docs_url.
  • rocm services docs is added to the command surface the e2e coverage report counts (crates/e2e-report).
  • Swagger UI 5.32.6 comes from utoipa-swagger-ui with vendored, so the build downloads nothing and the page works offline. It is not a crate, so cargo-about cannot see it: its Apache-2.0 NOTICE and the notices its bundle retains are in a hand-maintained "Bundled web assets" section of about.hbs, documented in MANIFEST.md. A test fails if the served version stops matching the notices.
  • New dependencies in rocmd (and so rocm): utoipa, utoipa-swagger-ui and reqwest (no TLS, stream).

Verification

  • cargo test --workspace --all-targets: all pass except three tests that fail on the test host for reasons unrelated to this change (stop_of_a_container_engine_is_unconfirmed_when_the_engine_stop_fails, where the host's podman docker shim ignores DOCKER_HOST, and two build_script::tests::git_watch_paths_*, where git commit fails in the test's temp repo); on that heavily loaded host the *_never_reuses_another_writers_temp_file timing probes also report themselves inconclusive at times. cargo fmt --check, cargo clippy --workspace --all-targets -D warnings, check-architecture-doc, check-module-size and check-crate-edges pass, and cargo xtask e2e -- -n "service-docs-" passes 11/11. cargo xtask smoke was not run on that host.
  • Unit tests run a real docs server against a stand-in model server that echoes what reached it. Each of these was run against a deliberately broken target and went red:
    • forwarded headers: dropping Content-Type or Accept from the allowlist;
    • forwarding the decoded path instead of the raw one;
    • removing the 64 MiB body limit (a 3 MiB body);
    • following redirects;
    • removing the connect timeout (a listener that never accepts; Linux only);
    • reaching :: on 127.0.0.1 instead of ::1;
    • not escaping the id reflected into the "not running" page, or the model name on the index;
    • not percent-encoding ids in links;
    • admitting any Sec-Fetch-Site, and dropping either anti-framing header;
    • ignoring the loader passed in (the_docs_list_what_the_loader_reports), and giving the docs a loader that reads records as written instead of rocm services list's (a_server_whose_process_died_has_no_docs);
    • an engine's endpoint answer reporting another engine's routes (documented_routes_are_the_routes_each_engine_reports, which asks each engine for that answer the way the CLI does);
    • the docs using the canonical id instead of the served name for the example, title or index label (fixtures with an AIM-style image reference as the canonical id);
    • falling back to a free port on a failure other than the port being in use;
    • reading the servers on every request, or letting concurrent requests each start a read (requests_within_the_ttl_share_one_read_of_the_servers: five requests at once against a slow loader read once, and once more after the TTL).
  • e2e (service_docs.feature, cargo xtask e2e -- -n "service-docs-"), each on a port of its own:
    • service-docs-01: the command rocm services list names opens docs listing the chat routes with the server's model.
    • service-docs-02: try-it reaches the model server.
    • service-docs-03: a stopped server's docs say it is not running and it gets no docs: line (Linux only, it stops a process).
    • service-docs-04/05: an unknown server, or no server, is refused while another program holds the port given, so a command that served before refusing would fail on the port instead. Checked red with serving moved before the refusal.
    • service-docs-06: Ctrl+C prints API docs stopped., closes the port, and the model server still answers and is still listed (Linux only, it sends SIGINT). Checked red with the message removed and with Ctrl+C not ending the command.
    • service-docs-07: with port 11430 held, the command says so and its link opens on a free port (@serial), and the note is on stderr, not among the links on stdout. Checked red with the note removed and with it printed on stdout.
    • service-docs-08: the engine docs link rocm services list prints opens the model server's own /docs (the stand-in answers it as vLLM would). Checked red with the URL changed.
    • service-docs-09: a stopped server's refusal names rocm services restart e2e-mock --yes and rocm services list --all; the scenario runs the list it names and finds the server stopped with that same restart command. Like 04/05 it holds the port it passes, so a command that served instead of refusing fails rather than hanging. Checked red with the old advice. Running the restart through needs an engine to start, so no lane does.
    • service-docs-10/11 (Linux, vLLM record): rocm serve of a model that is already running, piped and with --verbose, prints the docs and engine docs rows, and the printed docs command opens the docs. No GPU needed, since the server is reused. Checked red with the rows removed from each path.
  • The GPU serve canaries serve-05 (vLLM), serve-07 (Lemonade) and nightly serve-26 (AIM) run the docs: command rocm serve printed and try every GET route the docs list against the real engine. serve-07 is an expected failure on Linux Lemonade lanes for an unrelated reason, so there the check is effective only on Windows Lemonade lanes.
  • Not covered by a scenario, and listed in docs/manual-testing.md 4e: the summary's docs row (only shown on a terminal; the harness pipes output; its placement is unit-tested in serve_summary), and the rows of an attached fresh launch (rocm serve --verbose of a model not yet running) and its "detached - server still running" block, which need an engine to start. All print the rows of the same launch_docs_rows, which is unit-tested. The Ctrl+C handler is installed before anything is started; a Ctrl+C landing in that first instant ends the command before anything is served, and no test can aim at that window.
  • Not verified: a real engine behind the docs on this change's own lanes (GPU lanes only).

Scope

  • No --json field for the docs command: rocm services list --json prints the records verbatim, as before.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (Not a bug fix.)

  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (apps/rocm/src/services_docs.rs, apps/rocmd/src/api_docs.rs.)

  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, not only the wording, per AGENTS.md §3.

Comment thread apps/rocm/src/serve_summary.rs Fixed
Comment thread apps/rocm/src/serve_summary.rs Fixed
@rominf
rominf force-pushed the feat/service-api-docs branch from 54625d3 to 9ac33b1 Compare October 26, 2026 11:29
@rominf
rominf marked this pull request as ready for review October 26, 2026 12:11
@rominf
rominf requested a review from a team as a code owner October 26, 2026 12:11
@rominf
rominf requested a review from johnl-amd October 26, 2026 12:11
@rominf rominf changed the title feat: serve API docs for every running model server feat: serve API docs for every running model server (ROCMAI-671) Oct 26, 2026

@tomastola tomastola left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Blocking

  • docs_redirect can panic on a crafted service id — apps/rocmd/src/api_docs.rs:287-289
    Path<String> is percent-decoded, so /services/%0A/docs yields a newline, and Redirect::permanent(&format!("/services/{service_id}/docs/")) then builds an invalid Location header value, which panics (read from the code, not run). The loopback Host guard does not stop it: a web page the user visits can make their browser request http://127.0.0.1:11430/services/%0A/docs, and that Host passes. Only that connection is lost, but the handler also never checks that the service exists and answers a cacheable 308. No test covers the redirect.
    Confidence 70 · logic · Fix: percent-encode the id, or check find_running_service first and return 404.

  • The docs: output has no scenario on the summary and attached paths, and docs/testing.md overstates the coverage. — apps/rocm/src/serve_cmd.rs:937, apps/rocm/src/service_launch.rs:634-637,922-933,957-968,983-985, docs/testing.md
    The e2e harness pipes stdout, so summary_mode is false and serve-05/serve-07 only reach the plain docs: line. Nothing runs the TTY summary wiring (docs_url: docs_link), the --verbose attached output, or the already-running docs: line; the serve_summary unit tests build DeploymentSummary by hand, so they stay green if the serve_cmd wiring breaks. AGENTS.md §3 requires a scenario for user-observable output, and docs/testing.md says the link rocm serve prints is "checked by serve-05/07", which holds only for non-TTY plain output.
    Confidence 85 · architectural · Fix: add a scenario or PTY check for the summary and attached paths, or narrow the docs/testing.md claim and name the manual check (docs/manual-testing.md 4b covers only the TTY summary).

  • rocm services list --help does not mention the new docs: line. — apps/rocm/src/main.rs:1046-1050
    serve and daemon help were updated; the ServicesCommand::List doc comment was not. AGENTS.md requires the --help doc comment to change with observable behaviour.
    Confidence 85 · mechanical · Fix: add a clause about the docs link.

  • Tests that stay green when one branch of the change is broken.

    • service_docs_links's /health short-circuit (apps/rocm/src/api_docs_link.rs:~88): no_link_when_nothing_listens_on_the_docs_port asserts only <2s, so deleting the probe leaves it green, although the doc comment claims it costs one refused connection.
    • The .filter(managed_service_is_live) at apps/rocm/src/main.rs:6666: service-docs-03 stays green without it, because the docs server answers 404 for a stopped record.
    • no_link_when_the_docs_port_is_zero stays green with served_port()? removed, because connecting to port 0 fails anyway.
    • service-docs-04 passes with the whole feature reverted (a reverted build prints no docs: line); it does go red if the probe is removed.
      Confidence 70 · mechanical · Fix: have the stand-in server count connections and assert exactly 1 for /health; assert the stopped-record case against a server that answers 200 for that id; make the zero-port test use a port that would answer.

Non-blocking

  • Record read errors are shown as "not running" and logged nowhere — api_docs.rs:~158, running_services. load_managed_services(..).unwrap_or_default() turns an unreadable records directory into "No model server is running" and a 404. Fix: log on Err, and ideally return 5xx on the per-service routes. (Confidence 75)
  • Daemon shutdown can hang on an active try-it stream — api_docs.rs:62-65,~96, apps/rocmd/src/service.rs:~558. Graceful shutdown waits for in-flight responses, the forwarder has no overall timeout, and stop() then joins the thread synchronously. Fix: bound the serve future with a timeout after the shutdown signal. (Confidence 70)
  • The liveness predicate is duplicated — apps/rocmd/src/api_docs.rs:217 (service_is_running) and apps/rocm/src/main.rs:8189 (managed_service_is_live). Same four statuses today; drift would give no link or a link to a 404. A shared helper in rocm-core would fix it. (Confidence 70)
  • Attached mode waits silently for up to 5 s — apps/rocm/src/service_launch.rs:957-962. service_docs_link(..., &mut || {}) passes no progress callback, so after endpoint: there is a silent stall when a just-spawned helper's docs never answer. (Confidence 70)
  • Misplaced comment — api_docs.rs:~286. The comment about the endpoint key lasting "until the page is closed" sits above .try_it_out_enabled(true), which does not control that; it relies on the Swagger UI default and nothing asserts it. Fix: move the comment or set persist_authorization(false) explicitly. (Confidence 70)

Decisions for the author

  • Docs are served only by the persistent helper (automations_enabled) — tradeoff. An on-demand rocmd run returns before start_api_docs. The CLI probes before printing a link, so no dead link appears; the cost is that no docs exist in that mode. Confirm this is intended.
  • The Origin check admits any loopback origin on any port — tradeoff. Another local web app (e.g. a dev server on localhost:3000) can POST through the forwarder to any running model server. That is deliberate for a same-origin docs page and still blocks cross-site pages and DNS rebinding. Confirm this is the intended boundary.
  • Forwarded responses keep only Content-Type — non-blocking improvement. A text/html reply from a model server (e.g. llama.cpp's UI) would run same-origin with the docs. Consider X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox on forwarded responses.
  • The change is large for one PR — about 2.7k lines across a new HTTP subsystem, CLI wiring, e2e scenarios and licence plumbing; the licence work could ship separately.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9ac33b1

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (29 files, +2709/-42), split into five independent file-coupling groups (forwarder/docs server, protocol/config contract surface + engine consumers, CLI link-printing, e2e tests, docs/license/lockfile), plus a direct read of CI results and git history for the touched files. Implements ROCMAI-671 ("Serve browsable API docs for every running model server") in full — the ticket's description is the PR description verbatim, backfilled from GitHub.

Blocking

  • The scenario the PR's own description names as the real-hardware verification for this feature fails on CI, on the exact commit under review — tests/e2e-cucumber/features/model_serving.feature:73, tests/e2e-cucumber/tests/e2e/service_docs_steps.rs:288-289
    This PR adds a Then step, "the API docs link printed when it started opens the server's API docs," to both GPU serve canaries (model_serving.feature:46 for vLLM/serve-05, :73 for Lemonade/serve-07). The PR description states explicitly: "serve-07 is an expected failure on Linux Lemonade lanes for an unrelated reason, so there the check is effective only on Windows Lemonade lanes." The E2E tests (Strix Halo, Windows, current) check on this PR's head commit (9ac33b1) failed: Reconciliation: ... 1 unexpected failure(s). FAIL: 'serve-lemonade-inference' was expected to pass on this host but FAILED — a regression. The captured output shows the model actually served and answered chat completions correctly (the two preceding assertions on the real chat response passed); only the new step panicked: `rocm serve` printed no API docs link, i.e. docs_link_in_serve_output(stdout) found nothing in the captured rocm serve stdout within whatever window the implementation waits. expectations.toml:96 confirms Windows is not in the xfail scope for this scenario (when = { effective_engine = "lemonade", os = "linux" }), so this is a genuine, newly-introduced regression on exactly the lane the PR's own text claims validates the feature — not a known flake and not something the PR's xfail bookkeeping accounts for.
    Confidence 100 · architectural · Fix: investigate why the Windows/Lemonade lane's rocm serve stdout doesn't carry the docs link within the up-to-5s wait this PR implements (slow backend install before serve, as seen in the log, may be eating into or racing that window) and get this lane green, or narrow the description's "effective on Windows Lemonade lanes" claim if it cannot be fixed before merge.

Non-blocking

  • The "shared route catalogue" this PR adds doesn't actually feed the docs server's rendered routes; two independent implementations are kept in sync only by a unit test — crates/rocm-engine-protocol/src/lib.rs (OPENAI_CHAT_ROUTES/chat_supported_routes, new), apps/rocmd/src/api_docs.rs:430-435 (routes_for's health-path dispatch), engines/vllm/src/state.rs, engines/lemonade/src/lib.rs.
    A workspace-wide grep for .supported_routes turns up zero production read sites — the new catalogue only feeds EndpointResponse.supported_routes, a field nothing outside engine unit tests reads. The docs renderer that this PR's own description calls the consumer of "the route list now lives once in rocm-engine-protocol" in fact derives its per-engine health path independently: let health_path = if record.engine == "vllm" { "/health" } else { "/v1/health" }; — a raw string match with an implicit "everything else is Lemonade" fallback, never reading the catalogue. The two are kept consistent today only by documented_routes_are_the_routes_the_engines_report (api_docs.rs:671-686); a third engine with a different health convention would get silently mis-documented unless someone remembers to touch this unrelated match and the test happens to catch it.
    Confidence 70 · architectural · Fix: have routes_for read the health path from the engine's own reported route data (or a rocm_engine_protocol helper keyed by engine id) instead of re-deriving it from record.engine, so the "single source of truth" the description claims is actually one.

Decisions for the author

  • Cross-site GET requests with no Origin header bypass the same-origin forwarder guard — tradeoff. apps/rocmd/src/api_docs.rs:171-173 treats an absent Origin header as allowed (headers.get(header::ORIGIN).is_none_or(...)), which a non-fetch/XHR cross-origin load (e.g. an <img src="http://127.0.0.1:11430/services/<id>/api/v1/models"> on any other open tab) satisfies. Impact is low today — every forwarded route is read-only GET, and the gap is explicitly named in a comment at api_docs.rs:159-164 as a deliberate scope choice ("requests that are not from a browser carry no Origin and are not affected"). Worth confirming this remains acceptable if a state-changing route is ever added behind this forwarder.
  • Security-relevant behavior (Host/Origin/DNS-rebinding guards, the Authorize bearer-key flow) is covered only by crate-level HTTP integration tests, not a Gherkin scenario — non-blocking-improvement. AGENTS.md §3 requires a Gherkin scenario for user-observable behavior changes ("a unit test asserting the internal helper does NOT discharge this"). The existing tests in apps/rocmd/src/api_docs.rs are real HTTP requests against a really-bound router (not internal-helper unit tests), and the PR description is accurate about this being unit-level coverage rather than overclaiming e2e coverage — so this isn't a description mismatch. But it is a judgment call whether security behavior answered by a browser request counts as the kind of "user-observable behavior" §3 means; worth the author's explicit confirmation either way.
  • The bundled Swagger UI third-party-notices section is guarded by a version-string match, not a content match — non-blocking-improvement. the_bundled_swagger_ui_is_the_version_named_in_the_notices (api_docs.rs:717-739) only asserts the version string appears in THIRD_PARTY_NOTICES.txt; it does not check the hand-copied sub-license list (classnames, buffer, DOMPurify, js-yaml, etc.) still matches that version's actual bundle. Nothing is stale today (independently verified: 5.32.6 is indeed what utoipa-swagger-ui-vendored 0.2.0 vendors), but a future version bump that changes the bundle's own third-party set would pass this test with stale notices. Consider noting the gap in MANIFEST.md's maintenance section, or parsing the bundle's own license comments for comparison.
  • No end-to-end scenario for api_docs.enabled: false — non-blocking-improvement. docs/manual-testing.md lists it as a manual check and api_docs_link.rs unit-tests it, but none of the four new service_docs.feature scenarios exercise the disabled-config path through a real daemon. Low-risk (thin wrapper around an already unit-tested config read).
  • IPv6 loopback ([::1]) passes the Host/Origin validators but the server only ever binds IPv4 127.0.0.1 — non-blocking-improvement, apps/rocmd/src/api_docs.rs. Harmless today (nothing is listening on ::1 to reach), just a minor dual-stack completeness gap that could affect browsers that try ::1 first when resolving localhost.

Positive signals

  • The Host/Origin/DNS-rebinding defenses are adversarially tested against the named bypass patterns, not just the happy path: a test exercises the classic 127.0.0.1.evil.example rebinding string and a cross-site POST rejection, both against a real forwarded request (api_docs.rs, tests around lines 699-715, 939-964).
  • The forwarder is proven immune to the proxy-bypass it claims: a dedicated test sets HTTP_PROXY/ALL_PROXY to a dead endpoint and confirms the real request still succeeds via .no_proxy() (api_docs.rs:992-1047), rather than just asserting the client builder option is set.
  • rocm serve's printed docs link is tied to the server actually having answered for that service, not printed unconditionally — the invariant is stated in the module doc, backed by the control flow, and covered by both positive and negative unit tests plus an e2e scenario that fetches the real swagger-initializer.js rather than a stand-in (apps/rocm/src/api_docs_link.rs, tests/e2e-cucumber/features/service_docs.feature).
  • Every failure boundary (docs disabled, port 0, taken port, no helper running, another program on the port, config load failure) degrades to "no link" rather than a crash or a broken URL, each with its own test.
  • Crate-layering is intact in both directions: apps/rocm/Cargo.toml is untouched by this PR, and apps/rocmd/Cargo.toml's new dependencies (reqwest, utoipa, utoipa-swagger-ui) are crates.io-only — no new first-party edge for cargo xtask check-crate-edges to catch.
  • The ASCII-output-contract convention (xtask/src/ascii_output_contract.rs, from the unrelated but highly detailed #683 history on these same files) is respected: every non-ASCII character introduced by this PR is inside a comment or doc-comment, none inside a user-facing string literal.

Deployment notes

  • A new default-on local listener is introduced: api_docs.enabled: true by default, binding 127.0.0.1:11430. Loopback-only, but operators scanning for new listening sockets after upgrade should expect it.
  • New third-party bundled content ships in the binary (Swagger UI 5.32.6, Apache-2.0, vendored via utoipa-swagger-ui's vendored feature) — about.hbs/THIRD_PARTY_NOTICES.txt/MANIFEST.md updated accordingly; see the notices-maintenance note above for what isn't automatically guarded on a future bump.
  • Three new crates.io dependencies land in apps/rocmd (reqwest, utoipa, utoipa-swagger-ui); Cargo.lock's diff was checked and is narrowly scoped to them (no incidental version churn elsewhere).

What this covered

Read completely, against 1c33248c318c72a9e837627ff71476c8583f1930...9ac33b1a39589178aa8eacdd34dab3fe496eff80: every changed file in full at the PR head plus the pre-change version of each, via five independent workers grouped by coupling (apps/rocmd/src/api_docs.rs + its service/lib/Cargo.toml wiring; rocm-engine-protocol + rocm-core config + the two engine crates; apps/rocm's link-printing/serve-summary/service-launch cluster; the full e2e feature/step/naming-check set; README/MANIFEST/architecture docs/about.hbs/THIRD_PARTY_NOTICES/Cargo.lock), each also reading the relevant surrounding/caller code (e.g. webhook.rs for HTTP-server convention comparison, crate_edges.rs's ALLOWLIST, xtask/src/ascii_output_contract.rs's history). I additionally read AGENTS.md in full, checked git log/git blame on the touched pre-existing files for relevant prior conventions (this surfaced the ASCII-output-contract and ticket-ID history used above), read all CI check results for the head commit via live gh/GitHub-API queries including the actual failing job's log, and read the PR description and its named Jira ticket (ROCMAI-671) only after all findings above were reached independently from the code and CI.

Did not run: no Rust toolchain was available to any review worker in this sandbox, so no cargo build/check/clippy/test/xtask check-crate-edges was executed locally — all code-correctness findings are from static reading (quoted and line-cited) plus the one piece of dynamic evidence available, which is the project's own CI run on this exact commit. Analyze (rust), clippy, Test (affected crates), windows-build-and-test, and build-and-test all show passing on this commit's checks, which is taken at face value rather than re-derived. E2E tests (MI300X, current) and E2E tests (W7900, current) were still pending at review time and are not reflected above. Not reconciled against any PR discussion/comments — per the method, none were read.

@nowycondro

Copy link
Copy Markdown
Collaborator

🤖 Automated review (Claude Code) · 9ac33b1

I read the full diff (29 files, ~2.7k additions) with focus on the new security-sensitive surface: apps/rocmd/src/api_docs.rs (the loopback docs/forwarder server), apps/rocm/src/api_docs_link.rs (link probing/printing), apps/rocmd/src/service.rs (daemon wiring) and crates/rocm-core/src/config.rs (new api_docs config). The loopback binding, Host/Origin guard (DNS-rebinding and cross-site protection), no_proxy() forwarding client, and the header allowlist (Content-Type/Accept/Authorization only, cookies excluded) are all implemented carefully and are backed by solid unit tests that exercise exactly those guards (including a proxy-env test and a cross-site/rebinding test).

Non-blocking: docs_redirect (apps/rocmd/src/api_docs.rs:287-289) builds Redirect::permanent(&format!("/services/{service_id}/docs/")) directly from the percent-decoded Path<String> service_id, without checking find_running_service first (unlike docs_index/docs_file, which do check and HTML-escape on the not-running path) and without re-encoding. A request like /services/%0Aevil/docs decodes to a literal newline in service_id; axum's Redirect::with_status_code builds the Location header via HeaderValue::try_from(uri).expect(...), which panics on control characters. Impact is limited (request must already pass the loopback Host/same-origin Origin checks, and a panicking axum handler only drops that one connection/task rather than the whole process), so this is low severity, but a find_running_service check plus percent-encoding the id before interpolating would make it consistent with the other /services/{id}/... handlers. Confidence: medium (based on my recollection of axum 0.8's Redirect implementation; I did not build the crate to confirm the exact panic path).

Could not verify: I did not have a Rust toolchain/network access set up to run cargo test --workspace or cargo clippy in this sandbox, so I relied on manual reading of the new code and its existing tests rather than executing them; the PR description's own test coverage (unit, CLI, and e2e service_docs.feature) looks to exercise the route catalogue, forwarder header/cookie handling, the Host/Origin guards, and taken-port fallback, which matches what I found in the code.

🤖 Generated with Claude Code

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · a170585

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (32 files, +3125/-50) at HEAD a1705854a248dc6aa3568502610b53597564f36d. Implements ROCMAI-671 ("Serve browsable API docs (Swagger UI) for every running model server") in full; the ticket is still "In Progress," consistent with this PR being open.

Blocking

  • e2e-cucumber's test target does not compile under the project's own CI gate, on a function this PR adds — tests/e2e-cucumber/tests/e2e/service_docs_steps.rs:112,140
    async fn missing_docs_link_evidence(world: &E2eWorld, stdout: &str) -> String holds &E2eWorld across client.get(&url).send().await at line 140. E2eWorld carries a field that is Send but not Sync (a portable_pty::MasterPty trait object per the diagnostic), so the future is !Send, and clippy::future_not_send fires as a hard error under -D warnings. Verified directly from the live CI job logs for this exact commit: the clippy job and the Windows build and test (debug) job both fail identically — error: could not compile e2e-cucumber (test "e2e") due to 3 previous errors — quoting this same function/line. This also cascades to the windows-build-and-test gate job (fails because its windows-debug input is failure).
    Confidence 100 · mechanical · Fix: stop holding &E2eWorld across the .await — extract the few fields missing_docs_link_evidence actually needs (service id already comes from stdout; paths/config/event-log reads can happen before the .await or via owned copies) so the future no longer captures a non-Send reference.

  • The new GPU-canary docs-link assertion fails on live CI at this exact commit, on the lane the PR's own text calls the only one that still exercises it — root-caused to the e2e harness's docs-port randomization silently no-op'ing — tests/e2e-cucumber/tests/e2e/service_docs_steps.rs:75-94 (give_api_docs_a_port_of_their_own), tests/e2e-cucumber/tests/e2e/serving_steps.rs:179,580 (its call sites), crates/rocm-core/src/config.rs:93-97 (RocmCliConfig.api_docs: ApiDocsConfig under plain #[serde(default)]), docs/testing.md ("effective on the Windows Lemonade lanes" claim), PR description (same claim, "serve-07 ... effective only on Windows Lemonade lanes").
    give_api_docs_a_port_of_their_own only randomizes a port when config["api_docs"]["port"] is absent from config.json; its own doc comment frames the early-return as "a serve retried within one scenario reuses the helper the first attempt started." But #[serde(default)] on RocmCliConfig::api_docs only fills a missing field on deserialize — it does not suppress the field on serialize — so the very first time anything in the scenario's isolated tree loads-then-saves config.json (which an ordinary rocm invocation does), the file gains an explicit, literal api_docs.port equal to the stock default (DEFAULT_API_DOCS_PORT = 11_430), indistinguishable from a port this function chose itself. From that point on the "already chosen" short-circuit returns 11430 instead of ever calling free_port(). This matches live evidence precisely: the E2E tests (Strix Halo, Windows, current) job on this commit shows serve-lemonade-inference (serve-07) failing, with the helper's own logged event "action":"api_docs_unavailable","message":"...failed to bind API docs server to 127.0.0.1:11430: ...Only one usage of each socket address...(os error 10048)" and the scenario's own config.json dump reading "api_docs":{"enabled":true,"port":11430} — the un-randomized stock default, colliding with whatever else on the shared runner already held it. docs/testing.md and the PR description both assert this exact lane ("Windows Lemonade") is the one place this check is "effective" — at this commit it is red for exactly the mechanism above, so that claim does not hold at HEAD.
    Confidence 85 · architectural · Fix: make give_api_docs_a_port_of_their_own (or its call sites) distinguish "already randomized by me this scenario" from "config.json merely contains the serialized default" — e.g. track the chosen port in E2eWorld instead of re-reading it back out of config.json, or always rewrite a non-zero sentinel/random port rather than trusting whatever key already exists.

  • The docs renderer never reads the "single source of truth" route catalogue it documents itself as using, and the mismatch is reachable today through the already-shipping "aim" engine, untested — apps/rocmd/src/api_docs.rs:500-542 (routes_for), apps/rocmd/src/api_docs.rs:561-575 (engine_docs_note), crates/rocm-engine-protocol/src/lib.rs:631-639 (OPENAI_CHAT_ROUTES/chat_supported_routes), engines/aim/src/state.rs:66-72 (EndpointResponse.supported_routes reports health at "/health"), apps/rocm/src/main.rs:166 (SUPPORTED_ENGINES = ["lemonade", "vllm", "aim"]).
    routes_for's doc comment says the routes come from "its health route plus rocm_engine_protocol::OPENAI_CHAT_ROUTES... The single place the docs learn what a server serves," but the function body never calls into that crate — it hardcodes the three OpenAI-style paths as string literals and picks the health path with if record.engine == "vllm" { "/health" } else { "/v1/health" }. engine_docs_note repeats the same binary split, printing Lemonade/llama.cpp-specific prose for anything that isn't literally "vllm". Meanwhile lemonade and vllm were correctly updated to report their routes through the shared chat_supported_routes helper (confirmed in their diffs) — but a third engine, aim, is already a real, currently-shipping, selectable engine (rocm serve --engine aim) whose own EndpointResponse.supported_routes names its health route "/health", not "/v1/health". Because routes_for's match only special-cases "vllm", an AIM-served model's generated docs page documents (and lets "Try it out" hit) the wrong health path, and describes the server with Lemonade/llama.cpp prose that is false for AIM's Docker-based backend. No test in api_docs.rs's suite exercises "aim": chat_server_docs_list_its_routes_with_the_served_model, documented_routes_are_the_routes_the_engines_report, and engine_note_links_vllm_docs_and_llama_cpp_routes all iterate only ["vllm", "lemonade"]. This also means the PR description's claim that "the route list now lives once in rocm-engine-protocol... used by both engines and tested against the docs" is accurate only for the engines' side; the docs side duplicates it independently and is kept consistent with the engines only by a test that itself never covers the third engine.
    Confidence 85 · architectural · Fix: have routes_for/engine_docs_note read the health path (and backend description) from the engine's own reported supported_routes/a rocm_engine_protocol helper keyed by engine id, instead of re-deriving it from a two-way record.engine match; add an "aim" case to the three tests above.

Non-blocking

None

Decisions for the author

  • The forwarder accepts any HTTP method and treats an absent Origin as allowed — tradeoff. apps/rocmd/src/api_docs.rs:173-176 (require_loopback_host) and the router's .route("/services/{id}/api/{*path}", any(forward)...): this is not limited to read-only GET — it forwards POST /v1/chat/completions (model-invoking, cost-bearing) and any other method. A same-origin "Try it out" page or curl carries no Origin and is intentionally admitted; the in-code comment frames this as deliberate for non-browser tooling. Worth the author's explicit confirmation that the scope is still right now that the forwarder's surface is confirmed to be more than GET.
  • A taken docs port leaves docs permanently off for that daemon's lifetime rather than retrying another port — tradeoff, directly relevant to the finding above: predictable, bookmarkable addressing (127.0.0.1:11430) vs. resilience to the exact kind of port collision seen in CI. Confirm the fixed-address choice is still preferred.
  • The bundled-Swagger-UI notices are guarded by a version-string match only — non-blocking-improvement. the_bundled_swagger_ui_is_the_version_named_in_the_notices (apps/rocmd/src/api_docs.rs:792-813) asserts the served version string appears in THIRD_PARTY_NOTICES.txt, not that the hand-copied sub-license list (classnames, buffer, DOMPurify, js-yaml, React, …) still matches that version's actual bundle. Nothing is stale today; a future version bump that changes the bundle's own third-party set would pass this test while leaving stale sub-license entries. Consider noting the gap in MANIFEST.md's maintenance section.
  • IPv6 loopback ([::1]) passes the Host/Origin validators but the server only binds IPv4 127.0.0.1 — non-blocking-improvement, apps/rocmd/src/api_docs.rs. Harmless today (nothing listens on ::1 to reach); a minor dual-stack completeness gap.

Positive signals

  • Every blocking/non-blocking item a stale round of review raised against an earlier commit of this same branch has been independently re-verified as fixed here, each now behind its own regression test: the docs_redirect panic on a crafted id (docs_redirect_only_names_a_running_server, testing literal %0A/%0D%0A injection), unreadable service records surfacing as 503 instead of silently "no servers" (unreadable_records_are_reported_not_shown_as_no_servers), daemon shutdown no longer blocking on an open try-it stream (stopping_does_not_wait_for_an_open_try_it_stream, bounded by SHUTDOWN_GRACE), the live/liveness predicate unified into one ManagedServiceRecord::is_live() used by both apps/rocm and apps/rocmd (crates/rocm-core/src/service_record.rs), and --help text updated for rocm serve/rocm daemon/rocm services list.
  • The Host/Origin/DNS-rebinding guard is adversarially tested against concrete bypass strings (127.0.0.1.evil.example, a crafted Host via percent-encoded newline, a cross-site POST, an absent-vs-same-origin Origin) against a really-bound router, not just unit-level string matching.
  • The proxy-bypass claim is backed by a test that actually fails when .no_proxy() is removed (502), with careful env-var save/restore to avoid cross-test interference, rather than only asserting the builder option was set.
  • apps/rocm/Cargo.toml is untouched; the new HTTP/OpenAPI dependencies land only in apps/rocmd, preserving the rocmd-must-never-depend-on-rocm layering invariant without needing a cargo xtask check-crate-edges allowlist change.

Deployment notes

  • A new default-on loopback listener is introduced (127.0.0.1:11430 by default, api_docs in config.json, can be disabled).
  • New bundled third-party web assets ship in the rocmd binary (Swagger UI 5.32.6, Apache-2.0, vendored via utoipa-swagger-ui's vendored feature); about.hbs/THIRD_PARTY_NOTICES.txt/MANIFEST.md updated accordingly.
  • Three new crates.io dependencies land in apps/rocmd only (reqwest, utoipa, utoipa-swagger-ui); the Cargo.lock diff is scoped to that dependency tree (two coexisting versions of base64/wasm-streams are normal cargo-about/lockfile output for differing transitive requirements, not unrelated churn).

What this covered

DEGRADED REVIEW (partial fan-out failure) — every Agent spawn attempt (two batches, nine workers total, across roughly 15 minutes) failed immediately with "Concurrent subagent limit reached," confirmed via repeated ListAgents calls showing this sandbox's 20-concurrent ceiling permanently saturated by unrelated sessions sharing the same pool, never freeing a slot across the whole review. No fan-out worker of mine ever started, so this is entirely a single-context (driver-only) review rather than the coupled-file-group fan-out the method calls for. I compensated by reading, myself, every one of the 32 changed files in full (the five new files — apps/rocmd/src/api_docs.rs all 1342 lines, apps/rocm/src/api_docs_link.rs, tests/e2e-cucumber/tests/e2e/service_docs_steps.rs, tests/e2e-cucumber/features/service_docs.feature, and the about.hbs/THIRD_PARTY_NOTICES.txt/MANIFEST.md additions) or the complete diff plus surrounding/caller context (every changed file), against 9875940ebc7d1dbf9db0706e01d29842e8ec1d27...a1705854a248dc6aa3568502610b53597564f36d; cross-file consumer searches (git grep at the PR head) for chat_supported_routes/OPENAI_CHAT_ROUTES, is_live, and the "aim" engine's selectability; the root AGENTS.md in full; and live CI job logs for this exact commit read directly (not just the status-check rollup) for clippy, Windows build and test (debug), windows-build-and-test, E2E tests (Strix Halo, Windows, current), and E2E tests (Strix Halo, Ubuntu, nightly, current) (the last of these failed on "No space left on device," an infrastructure flake unrelated to this diff, not raised as a finding). Recommend re-running the fan-out passes once subagent capacity is available, particularly a dedicated pass over apps/rocm/src/main.rs/serve_cmd.rs (read here only via diff-plus-context, not in full, since both files are very large) and a git log -p/git blame history pass per file, neither of which I completed to the method's normal standard in degraded mode.

One unavoidable step exposed prior-round text: listing existing reviews with perPage=100 (required to find the current HEAD's state and avoid double-posting) returned the full bodies of two earlier reviews on this PR, both at an older commit (9ac33b1a) — one flagging a real failing e2e lane among other things. All findings above were independently re-derived from the code and from live CI at the current head; in several cases my own reading reached the same conclusion that review had already reached (re-verified fresh, not assumed), in several cases I confirmed something it flagged is now fixed (see Positive signals), and the three Blocking findings above are either new (the aim-engine mismatch, never raised before) or reach the same CI lane by a different, independently-traced root cause (a concrete port-randomization bug rather than a timing/wait-window guess). Not reconciled against any PR discussion or review comments beyond that one mechanical, disclosed exception.


🤖 Generated with Claude Code

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 981ca91

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (32 files, +3137/-50) against base 95e63388, head 981ca91e. This is round 3 on this PR (prior rounds at 9ac33b1a and a1705854a); several earlier findings are now fixed, but three blocking issues raised in earlier rounds are still present at this exact head, plus one new one. Implements ROCMAI-671 ("serve browsable API docs for every running model server") in full for the vLLM/Lemonade case; see the first Blocking item below for where the "single source of truth" claim doesn't hold for the third shipping engine.

Blocking

  • routes_for/engine_docs_note mis-document the already-shipping "aim" engine — standing from round 2, still unfixed — apps/rocmd/src/api_docs.rs:505-509,562-575
    Both functions special-case only record.engine == "vllm" and treat every other engine as Lemonade. aim is a real, selectable engine (apps/rocm/src/main.rs:166, SUPPORTED_ENGINES = ["lemonade", "vllm", "aim"]) whose own EndpointResponse (engines/aim/src/state.rs:66-72) reports its health route at /health, not /v1/health — so an AIM-served model's generated docs page documents (and lets "Try it out" hit) the wrong health path, and engine_docs_note prints false prose claiming the server "runs with the llama.cpp server," which is meaningless for AIM's Docker-based backend. The PR's own description frames rocm_engine_protocol::chat_supported_routes as "the single place the docs learn what a server serves... used by both engines and tested against the docs" — true for vllm/lemonade (both were migrated onto the shared helper), false for aim, whose own EndpointResponse keeps a hand-written duplicate list never routed through routes_for. None of the three relevant tests (chat_server_docs_list_its_routes_with_the_served_model, documented_routes_are_the_routes_the_engines_report, engine_note_links_vllm_docs_and_llama_cpp_routes) include an "aim" case — adding one would fail today, confirming this is live, not hypothetical.
    Confidence 88 · architectural · Fix: have routes_for/engine_docs_note read the health path from the engine's own reported supported_routes (or a rocm_engine_protocol helper keyed by engine id) instead of a two-way record.engine match; add an "aim" case to the three tests above.

  • The e2e harness's docs-port randomization silently no-ops the first time any scenario saves config.json — standing from round 2, still unfixed — tests/e2e-cucumber/tests/e2e/service_docs_steps.rs:75-94 (give_api_docs_a_port_of_their_own), crates/rocm-core/src/config.rs (ApiDocsConfig under plain #[serde(default)])
    The helper only randomizes api_docs.port when the key is absent from config.json; it treats presence as proof "this scenario already chose a port." But RocmCliConfig::save() always serializes the whole struct, so the very first rocm invocation that loads-then-saves a fresh config (e.g. "a managed runtime is active"'s rocm runtimes activate, which every GPU canary scenario runs first) writes the stock default (DEFAULT_API_DOCS_PORT = 11430) to disk — indistinguishable from a chosen port. From then on, serve_gpu_model/serve_and_wait's call to this helper returns 11430 instead of calling free_port(), defeating the isolation this function exists to provide, for exactly the two scenarios (serve-05, serve-07) this PR adds docs assertions to, and for every other GPU scenario that happens to run after an activate/install step. This was the traced root cause of a Windows-lane CI failure in the previous review round; no CI has yet run on the current head to show whether it still reproduces (see next finding), but the code path itself is unchanged.
    Confidence 80 · architectural · Fix: track "did this harness already choose a port for this scenario" in E2eWorld instead of re-deriving it from whatever happens to already be on disk.

  • A probe-removal mutation leaves a test green — standing from round 1, still unfixed — apps/rocm/src/api_docs_link.rs:240-253 (no_link_when_nothing_listens_on_the_docs_port)
    The test only asserts the return value is None and that it returned in under 2s; it cannot distinguish "probed and got refused" from "never probed at all." If service_docs_link/service_docs_links's actual TCP connect were deleted and the functions made to unconditionally return None/empty, this test would still pass unchanged. Contrast with the file's own a_docs_server_that_is_down_is_asked_once_for_the_whole_list, which does instrument a connection counter and genuinely verifies the probe ran.
    Confidence 85 · test-quality · Fix: have the stand-in server count connections and assert exactly 1, as the sibling test already does.

  • No CI check has run for this exact head commit — 981ca91ea84e8074c63fa8d70ca0383384fbfb6f
    get_check_runs returns zero check runs for this SHA. None of build-and-test, clippy, the Windows lane, or any E2E lane (including the two GPU canaries this PR's own description names as its real-hardware verification) have reported one way or the other for the commit under review — including whether the port-randomization bug above still reproduces as it did at the prior head.
    Confidence 100 · Fix: push a no-op or otherwise re-trigger CI for this head and confirm green (in particular the two GPU canaries and the e2e consolidated report) before merge.

Non-blocking

  • The new docs-link wait (up to 5s) is not gated on terminal interactivity, unlike every other progress indicator in the same call paths — apps/rocm/src/serve_cmd.rs:1225-1230, apps/rocm/src/service_launch.rs:1097-1111 (run_attached_service)
    Both call service_docs_link(...) unconditionally, including in the plain/piped branch that the surrounding code's own comments describe as "CI, the chat assistant." cli_progress::Spinner is hard-gated on stderr().is_terminal(), so in a piped session the wait still runs (and blocks for up to 5s on a cold helper start) with zero visible progress output. No test exercises "helper cold-starting + non-interactive/attached serve" (every service_docs.feature scenario starts with Given the background helper is running).
    Confidence 75 · logic · Fix: gate the wait behind the same interactivity check the rest of the function already uses, or shorten/skip it for non-TTY output.

Decisions for the author

  • The forwarder accepts any HTTP method and treats an absent Origin as allowed — tradeoff, apps/rocmd/src/api_docs.rs:171-176. Forwards POST /v1/chat/completions (model-invoking) and any other method, not just read-only GET; a same-origin page or curl carries no Origin and is intentionally admitted, per the code's own comment. Worth re-confirming this scope now that the surface is more than GET.
  • A taken docs port leaves docs off for the daemon's whole lifetime rather than retrying another port — tradeoff. Predictable, bookmarkable addressing vs. resilience to exactly the collision class the port-randomization bug above can produce.
  • rocmd now bundles two separate HTTP-client stacks (reqwest for the new try-it forwarder, pre-existing ureq elsewhere) — non-blocking-improvement. The Cargo.toml comment justifies this (the stream feature ureq lacks), reasonable but worth noting as accepted dependency growth.
  • IPv6 loopback ([::1]) passes the Host/Origin validators but the server only ever binds IPv4 127.0.0.1 — non-blocking-improvement, harmless today since nothing listens on ::1 to reach, but a dual-stack completeness gap for browsers that try ::1 first when resolving localhost.

Positive signals

  • Two defects from earlier review rounds are now genuinely fixed and independently re-verified this round: the docs_redirect panic on a crafted service id (now covered by docs_redirect_only_names_a_running_server, testing literal %0A/%0D%0A injection), and the clippy::future_not_send failure in missing_docs_link_evidence (confirmed by checking every async fn and every &E2eWorld/&mut E2eWorld-taking function in the file — the fix is complete and nothing else in the file reintroduces the pattern).
  • ManagedServiceRecord::is_live() is a clean, correctly-reused consolidation of a liveness predicate that was previously duplicated between apps/rocm and apps/rocmd.
  • The Host/Origin/DNS-rebinding guard is adversarially tested against concrete bypass strings (127.0.0.1.evil.example, a cross-site POST, an absent-vs-same-origin Origin) against a really-bound router, and the proxy-bypass claim is backed by a test that actually fails when .no_proxy() is removed (502), not just an assertion the builder option was set.
  • Crate layering is intact both ways: apps/rocm/Cargo.toml is untouched, and the three new crates.io dependencies (reqwest, utoipa, utoipa-swagger-ui) land only in apps/rocmd.
  • Cargo.lock's diff is narrowly scoped to the new dependency tree, MANIFEST.md's new dependency-table rows match it exactly, and the new README/docs sections (API docs defaults, forwarder guarantees, persistAuthorization: false) were spot-checked against the actual test suite and found accurate with no overclaiming.
  • Every failure boundary (docs disabled, taken/zero port, no helper, config load failure) degrades to "no link" rather than a crash, each backed by its own test.

Deployment notes

  • New default-on loopback listener: api_docs.enabled: true by default, binding 127.0.0.1:11430 (api_docs in config.json, can be disabled).
  • New bundled third-party web assets ship in the rocmd binary (Swagger UI 5.32.6, Apache-2.0, vendored via utoipa-swagger-ui's vendored feature); about.hbs/THIRD_PARTY_NOTICES.txt/MANIFEST.md updated accordingly.
  • No CI has run on this exact head commit — see Blocking.

What this covered

Every file in the diff read completely at head 981ca91e against base 95e63388, via five independent fan-out workers grouped by coupling: (A) apps/rocmd/src/api_docs.rs + its daemon/lib/Cargo.toml wiring; (B) rocm-engine-protocol + rocm-core config + the engine crates (lemonade/vllm/aim); (C) apps/rocm's link-printing/serve-summary/service-launch cluster; (D) the full e2e feature/step files; (E) README/MANIFEST/docs/about.hbs/THIRD_PARTY_NOTICES/Cargo.lock. Each worker also grepped the whole tree for consumers of changed symbols outside its own group; all candidates above were independently re-verified against the source before inclusion, and cross-checked against the two prior review rounds on this PR only to confirm which of their findings are now fixed (not to anchor new findings on their text).

Did not run: cargo build/test/clippy — no Rust toolchain available in this environment; all findings are from direct source reading (the aim-engine mismatch and the e2e port bug were both traced by hand through the actual call chains, not assumed). CI was read via the Checks API (zero runs for this head — see Blocking) rather than assumed. Not reconciled against this PR's own review discussion beyond the minimum needed to confirm which earlier findings are now fixed.

🤖 Generated with Claude Code

`rocm services docs [<id>]` serves browsable API docs (Swagger UI) for the
running model servers on 127.0.0.1, in the foreground, until Ctrl+C. Each
server's page lists the routes it answers, with request examples filled in
with the served model, and its "Try it out" forwards requests through the
page to the server, so it works for every engine whatever the engine's own
cross-origin settings. Examples, titles and labels use the name the server
answers to, which for AIM is not the image reference. The command refuses,
before serving anything, when the named server is not running (naming
`rocm services restart <id> --yes`) or when no server is. It uses port 11430, or a
free port when 11430 is in use and says so on stderr; `--port` picks one and
never falls back. Nothing runs in the background: after Ctrl+C nothing is left serving and
the model servers are not touched.

The docs judge which servers are running with the loader
`rocm services list` uses, which checks each server's process and endpoint,
so a server that stops or crashes while the docs are open drops off them
about a second after the read that finds it gone. Requests that arrive
together share one read.

`rocm serve` (summary, plain and attached output) and `rocm services list`
print a `docs:` line naming the command for every running server, and an
`engine docs:` line with vLLM's own /docs page.

The routes the docs describe come from one table in rocm-engine-protocol:
the OpenAI chat routes plus each engine's health route. The engines report
the same table as `supported_routes` in their `endpoint` answer, which a test
in `rocm` asks each engine for and compares with the docs.

The forwarder passes on only Content-Type, Accept and Authorization, keeps
the request path percent-encoded, accepts bodies up to 64 MiB, does not
follow redirects, gives up on a server that does not accept the connection
within 5 s, and bypasses any configured proxy. It reaches a server bound to
`0.0.0.0` on 127.0.0.1 and one bound to `::` on ::1, so an IPv6-only socket
is reached too; that mapping is now one helper in rocm-core
(`reachable_host`), which the Anthropic proxy uses as well.

Browser requests are admitted only from the docs server's own pages: by
`Origin` when the browser sends one, and by `Sec-Fetch-Site` when it does
not. The `Host` guard keeps DNS rebinding out. Every answer carries
`frame-ancestors 'none'`, `X-Frame-Options: DENY` and nosniff, and forwarded
answers are also CSP-sandboxed. Server ids are percent-encoded in links and
escaped in pages. Swagger UI is vendored (no CDN); its licence is in the
notices.

Scenarios service-docs-01 to -11 cover the listed command opening the docs,
try-it, a server stopped while its docs are open, refusals made before any
port is used, Ctrl+C stopping the docs while the model server keeps
answering, the fallback from a taken port 11430, the `engine docs` link, the
advice given for a stopped server, and the rows `rocm serve` prints when the
model is already running, piped and attached.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the feat/service-api-docs branch from 981ca91 to 588a3c6 Compare November 16, 2026 11:35
CI's per-crate clippy run over the e2e harness (`cargo clippy -p
e2e-cucumber --test e2e`) rejects `.map(..).unwrap_or_else(..)` on an
Option; use `map_or_else` instead. No behaviour change.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Comment thread apps/rocm/src/serve_summary.rs Dismissed
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

3 medium, low, or unrated finding(s) are not listed; the scan run has them.

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@rominf rominf changed the title feat: serve API docs for every running model server (ROCMAI-671) feat: serve API docs for running model servers with rocm services docs (ROCMAI-671) Nov 16, 2026
@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

@tomastola thanks for the review. The branch is rebuilt as one commit, and the reason is your first decision below, so the shape has changed since 9ac33b1a. Answers to the decisions and the points that went with them:

"Docs are served only by the persistent helper (automations_enabled) ... Confirm this is intended."
Not intended, and since #838 the premise is gone: the helper now starts only while an automation watcher is enabled, which is off by default, so most hosts would have had no docs at all. The docs no longer run in the helper. rocm services docs [<id>] serves them in the foreground, in the user's own process, until Ctrl+C, and leaves nothing running afterwards. rocm serve and rocm services list print docs: rocm services docs <id> for each running server (plus vLLM's own /docs as engine docs:), so there is no probe, no 5 s wait and no link that may or may not answer. The helper's lifecycle from #838 is untouched. Which servers count as running is decided by the same loader rocm services list uses, so a server whose process dies drops off the docs within a second.

"The Origin check admits any loopback origin on any port ... Confirm this is the intended boundary."
Narrowed. Browser requests are admitted only from the docs server's own origin, http://127.0.0.1:<port> and http://localhost:<port>; a page from another local port (http://localhost:3000) gets 403. A browser request that carries no Origin (a link, an image, a navigation) is judged by Sec-Fetch-Site instead, which must be same-origin or none; requests from programs such as curl carry neither header and pass. The Host guard against DNS rebinding stays. Covered by only_the_docs_servers_own_pages_may_send_browser_requests, origin_less_browser_requests_must_come_from_the_docs_pages and the live-router checks in docs_server_describes_and_forwards_to_a_running_server.

Forwarded responses: they carry X-Content-Type-Options: nosniff and Content-Security-Policy: sandbox; frame-ancestors 'none' (forwarded_answers_cannot_run_as_pages_of_the_docs_origin), and every page the docs serve carries frame-ancestors 'none' and X-Frame-Options: DENY (no_other_site_can_frame_the_docs).

Size / licence split: the licence work stays here, since the notices only describe the Swagger UI and crates this change adds. The helper wiring, the link probe and the api_docs config section are gone.

Your blocking findings from the first round, as they stand now:

  • docs_redirect with a crafted id: redirects only a running server's id, percent-encoded, and answers 404 otherwise; docs_redirect_only_names_a_running_server sends %0A and %0D%0A.
  • Scenario coverage of the docs output: service-docs-01 runs the exact command rocm services list prints and opens the docs, and service-docs-08 opens the engine docs link it prints. service-docs-10/11 run rocm serve of a model that is already running, piped and with --verbose, check the docs/engine docs rows and open the docs through the printed command. service-docs-04/05 check the command refuses before serving, 09 that a stopped server's refusal names the restart command rocm services list --all shows, 06 that Ctrl+C stops the docs while the model server keeps answering (Linux only, it sends SIGINT), and 07 the fallback when port 11430 is taken. The GPU canaries serve-05, serve-07 and nightly serve-26 (AIM) run the command a fresh rocm serve printed against the real engine. The summary row (terminal only) and the rows of an attached fresh launch and its "detached" block print rows from the same launch_docs_rows and are listed for manual testing in docs/testing.md/docs/manual-testing.md rather than claimed as scenario-covered.
  • rocm services list --help describes the docs: and engine docs: lines.
  • The tests that stayed green when broken were about the link probe, which no longer exists. The new tests listed in the PR body were each run against a deliberately broken target and go red.

@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Replies to the automated review at 981ca91e, point by point. The branch has since been rebuilt as one commit: the docs moved out of the background helper (which since #838 runs only while an automation watcher is enabled) into a foreground rocm services docs [<id>] command.

Blocking

  1. routes_for/engine_docs_note treat aim as Lemonade: fixed. Each engine's health route now comes from one table, ENGINE_HEALTH_ROUTES in rocm-engine-protocol (engine_health_route, engine_supported_routes). Each engine reports it as supported_routes in its endpoint answer, and the docs build their route list from the same table. An engine the table does not know gets no health route rather than another engine's. engine_docs_note has explicit Lemonade and AIM arms, and the vLLM link comes from ManagedServiceRecord::engine_docs_url. Tests: each_engine_reports_its_own_health_route_before_the_chat_routes (protocol), every_supported_engine_has_a_health_route_and_no_other_engine_does (iterates SUPPORTED_ENGINES), documented_routes_are_the_routes_each_engine_reports (in rocm, which links the engines: it asks each engine for its endpoint answer and compares its supported_routes with the docs' routes), and the aim/unknown cases in chat_server_docs_list_its_routes_with_the_served_model and engine_note_names_each_engines_own_docs. At e2e level the nightly AIM canary serve-26 tries every GET route the docs list, health included, against the real container.
  2. give_api_docs_a_port_of_their_own reuses a saved port: gone with the config section it worked around. Scenarios pass --port <free_port()> to rocm services docs and write nothing to config.json. The one scenario about the default port (service-docs-07) runs @serial.
  3. no_link_when_nothing_listens_on_the_docs_port is mutation-weak: removed with the link probe it tested; nothing is probed any more. The new tests listed in the PR body were each run against a deliberately broken target and go red.
  4. No CI on the head: CI has run on the rebuilt head (9d4e2957) and every check passes, including windows-build-and-test and E2E consolidated report, except CodeQL, which flags three alerts in test code (an assertion message and two e2e panic! messages), answered separately.

Non-blocking

  • The 5 s docs wait in piped and attached rocm serve: gone. rocm serve prints the command line without contacting anything.

Decisions

  • Any HTTP method, and an absent Origin, through the forwarder: any method is kept, since "Try it out" needs POST. A browser request without an Origin is now judged by Sec-Fetch-Site, which must be same-origin or none; only requests carrying neither header (curl and other programs) pass unchecked. Browser requests with an Origin are admitted only from the docs server's own origin, so another local web app can no longer use the forwarder.
  • A taken docs port: the command falls back to a free port when 11430 is taken and says so; an explicit --port that is taken fails instead.
  • Two HTTP clients in rocmd (reqwest for the streaming forwarder, ureq elsewhere): unchanged, for the reason in Cargo.toml.
  • IPv6 loopback: the docs server binds 127.0.0.1 only, and the Origin check admits only 127.0.0.1 and localhost on the docs port. A model server bound to :: is now reached on ::1, so an IPv6-only socket is reached too.

@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Replies to the earlier automated reports on this PR (the comment at 9ac33b1a and the reviews at 9ac33b1a and a1705854), for the points the replies to the later review and to the change request do not already cover. The branch has since been rebuilt as one commit: the docs no longer run in the background helper but in a foreground rocm services docs [<id>] command.

docs_redirect with a crafted id (comment at 9ac33b1a): it redirects only the id of a running server, percent-encoded, and answers 404 otherwise; docs_redirect_only_names_a_running_server sends %0A and %0D%0A.

Windows Lemonade canary printing no docs link (9ac33b1a) and the e2e docs port reused from config.json (a1705854): both came from the helper-hosted docs server, its api_docs config section and the probe rocm serve waited on, and all three are gone. rocm serve now prints docs: rocm services docs <id> without contacting anything, and the canaries serve-05, serve-07 and nightly serve-26 run that printed command on a free port passed with --port and try every GET route the docs list against the real engine.

missing_docs_link_evidence not Send (a1705854): the function is gone with the probe; cargo clippy --workspace --all-targets -D warnings passes on the new head.

Route catalogue not feeding the docs, and aim documented as Lemonade (both rounds): the docs and the engines' supported_routes now take each engine's health route from one table, ENGINE_HEALTH_ROUTES in rocm-engine-protocol, with tests on both sides and an aim case; details in the reply to the review at 981ca91e.

Decisions

  • Security behaviour without a Gherkin scenario: the Host/Origin/Sec-Fetch-Site guards are tested against a really bound router in apps/rocmd/src/api_docs.rs; what a scenario can observe from the CLI (the command opening the docs, try-it, refusals, Ctrl+C) is service-docs-01 to -11. A scenario cannot send a browser's cross-site headers more truthfully than those tests do, so the guards stay at that level.
  • Bundled Swagger UI notices guarded by version only: still the case. MANIFEST.md says what to update from the asset's own upstream notices when a bump changes it; the test only catches a version that no longer matches.
  • No scenario for api_docs.enabled: false: the setting is gone; nothing serves docs unless rocm services docs runs.
  • IPv6 loopback: the docs bind 127.0.0.1 only and admit only their own origin; a model server bound to :: is reached on ::1.

@rominf
rominf requested a review from tomastola November 16, 2026 12:40

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (40 files, +4467/-47), base ee605065 (merge-base with main), head 9d4e2957. Implements ROCMAI-671 ("Serve browsable API docs (Swagger UI) for every running model server") for the scope the PR describes; the ticket is "In Progress," consistent with this PR being open.

Blocking

None

Non-blocking

  • Two unreachable defensive branches — apps/rocmd/src/api_docs.rs:498-505 (docs_redirect's HeaderValue::try_from Err arm) and apps/rocmd/src/api_docs.rs:571,628-640 (raw_forwarded_path's None arm). Both look unreachable given the current call sites (service_path only emits valid header bytes; axum's route pattern guarantees the 5-way split always matches). Harmless dead code, not functional bugs — see inline comments.
  • chat_routes() hardcodes its three OpenAI route literals instead of reading rocm_engine_protocol::OPENAI_CHAT_ROUTES — apps/rocmd/src/api_docs.rs:677-700. The sibling health-route lookup just above it (routes_for, line 666) correctly reads from the new shared table; the chat routes don't. Kept consistent today only by tests, not by construction — see inline comment.
  • A second, pre-existing engine→health-path mapping doesn't read from the new shared table — apps/rocm/src/engine_runtime_select.rs:395-399 (service_http_readiness_paths), used for serve-readiness polling. This file has no diff against the merge-base (not touched by this PR) and the extra probe paths it tries for non-Lemonade engines are currently inert, so there's no live behavior bug — but it means the route list doesn't literally "live once in rocm-engine-protocol" repo-wide, only for the two things this PR touches (engine supported_routes and the docs server's health route).
  • No per-engine unit test directly asserts supported_routes() content in engines/aim, engines/lemonade, or engines/vllm themselves — that invariant is verified only by rocm-engine-protocol's own table test and by a cross-crate integration test (apps/rocm/src/services_docs.rs) that drives each engine's real stdio protocol. The cross-crate test does catch a regression end-to-end, so this is a coverage-layering gap rather than an untested behavior.
  • README.md line-wrap convention broken at 3 spots introduced/touched by this PR (lines 1256, 1585, 1590 — 87/100/107 columns vs. the surrounding ~75-80). Cosmetic only.

Decisions for the author

  • utoipa-swagger-ui-vendored appears in MANIFEST.md but not in THIRD_PARTY_NOTICES.txt's generated section — tradeoff/unconfirmed. Plausibly correct if it's a build-dependency of utoipa-swagger-ui (about.toml's ignore-build-dependencies = true would then explain the asymmetry), but this could not be confirmed without registry/network access to the crate's real manifest. Worth a quick sanity check before merge.
  • The "try it out" forwarder accepts any HTTP method and any path under /api/..., not just the documented GET routes — tradeoff, apps/rocmd/src/api_docs.rs. Low incremental risk since the model server is already directly loopback-reachable by the same local user, but worth confirming the scope is still intended now that it's more than read-only GET.
  • Forwarded requests have only a 5s connect timeout, no overall request timeout — tradeoff, needed to support streaming chat completions; documented and tested, just worth the author reconfirming it's the intended tradeoff.
  • An absent Origin and absent Sec-Fetch-Site together are treated as same-origin ("curl") and admitted — tradeoff, narrow gap for pre-Fetch-Metadata browsers on a cross-site GET navigation; low blast radius since only GET routes are reachable that way, and it's a deliberate, commented choice.

Positive signals

  • Deep, adversarial security test coverage: DNS-rebinding (127.0.0.1.evil.example-style Host headers), cross-site Origin rejection, the full Sec-Fetch-Site admission matrix, and CSP/X-Frame-Options/nosniff asserted on every route including error pages — not just the happy path.
  • The proxy-bypass claim (.no_proxy()) is backed by a test that actually fails when the option is removed, not just an assertion the builder flag was set.
  • reachable_host (crates/rocm-core/src/net.rs) is a clean de-duplication: it replaces logic that was previously inline-duplicated in apps/rocmd/src/anthropic_proxy.rs, confirmed to be the only two call sites repo-wide.
  • The engine health-route centralization (ENGINE_HEALTH_ROUTES in rocm-engine-protocol) is a correct, behavior-preserving extraction for all three engines, including aim — it is not special-cased or defaulted, and is guarded by a bidirectional test (every_supported_engine_has_a_health_route_and_no_other_engine_does) plus an integration test that drives each engine's real stdio protocol and diffs the result against the docs server's computed routes.
  • The new GPU-canary e2e step (serve-05/serve-07/serve-26) extracts every documented GET route from the live OpenAPI spec and round-trips each through the real engine via the docs proxy, rather than only checking that a page renders — it would catch a docs page that renders but lists stale or wrong routes.
  • The Swagger UI version-consistency test extracts the real vendored bundle's version string at test time and compares it against THIRD_PARTY_NOTICES.txt, rather than relying on a coincidental literal match.
  • Crate layering is respected both ways: the new reqwest dependency is default-features = false (no TLS pulled in), and no new dependency edge from rocmd onto rocm is introduced.
  • docs/architecture.md entries for both new production files (apps/rocmd/src/api_docs.rs, apps/rocm/src/services_docs.rs) are present and accurate, as are the other touched entries.

Deployment notes

  • A new default-on loopback HTTP listener is introduced: binds 127.0.0.1:11430 by default, falling back to a free port only when no --port was given and 11430 is taken; an explicit --port never falls back and fails instead.
  • Three new crates.io dependencies land in apps/rocmd only: reqwest (no TLS, stream feature), utoipa, utoipa-swagger-ui (vendored, so no network fetch at build time). The Cargo.lock diff is narrowly scoped to their transitive tree.
  • New bundled third-party web assets ship in the rocmd binary (Swagger UI, Apache-2.0); THIRD_PARTY_NOTICES.txt/about.hbs/MANIFEST.md updated accordingly (one item above for the author to double-check).

What this covered

Every file in the diff read completely at head 9d4e2957 against the merge-base ee605065, via 5 independent fan-out workers grouped by coupling: (A) apps/rocmd/src/api_docs.rs + its daemon/lib/Cargo.toml wiring + rocm-core's net.rs/service_record.rs/lib.rs; (B) rocm-engine-protocol + the three engine crates (aim/lemonade/vllm); (C) apps/rocm's CLI/link-printing/serve-summary/service-launch/services_docs cluster; (D) the full e2e-cucumber feature/step/naming files; (E) README.md/MANIFEST.md/docs/about.hbs/THIRD_PARTY_NOTICES.txt/Cargo.lock. Each worker read its files completely (one pre-existing, untouched file's unchanged test module — anthropic_proxy.rs — was sampled rather than read in full) and grepped the repo for cross-file consumers of changed symbols; every candidate finding was independently re-verified against the source before inclusion here.

CI was checked live via the GitHub API for this exact head commit: every reported check (build-and-test, clippy, Windows build/test, E2E tests, E2E consolidated report, security scans, commit-signatures, etc.) is green, none failing.

Did not run: no Rust toolchain was invoked by me or any worker in this environment, so no cargo build/test/clippy was executed locally — all code-correctness findings come from static reading of the source (quoted/line-cited) plus the project's own live CI results on this commit, taken at face value rather than re-derived.

One procedural note for transparency: listing this PR's existing reviews (required to confirm this account hadn't already reviewed it, to avoid double-posting) surfaced the text of earlier review rounds at prior commits on this branch. All findings above were independently reached by the 5 fan-out workers from fresh reading of the current code and live CI, not from that text. Where this review's own findings happen to overlap with something flagged in an earlier round (e.g. the engine health-route centralization, previously inconsistent for a third engine, now confirmed correct for all three), that is noted as independently re-verified, not assumed. Not otherwise reconciled against this PR's review-comment discussion.


🤖 Automated by an AI agent

)
.into_response(),
Err(_) => not_running_page(&service_id),
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (mechanical, confidence 75): the Err branch of HeaderValue::try_from(...) here looks unreachable given the current service_path implementation — it only ever emits ASCII alnum/-._~//%XX bytes, all valid HeaderValue content, so nothing can currently drive this into not_running_page. No test exercises this arm. Not a functional bug, just dead defensive code — consider a short comment noting why it's kept (e.g. "defensive, should be unreachable") or dropping it.

}

/// The part of `/services/<id>/api/<path>` after `api/`, still percent-encoded.
fn raw_forwarded_path(request_path: &str) -> Option<&str> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (mechanical, confidence 70): the None arm this function can return (consumed at line 571 as a 404) looks unreachable in practice — axum's route pattern /services/{id}/api/{*path} guarantees the 5-way split here always matches (Some(""), Some("services"), Some(_), Some("api"), Some(path)). Harmless dead code, not a functional bug; a short comment noting the fallback is defensive would save the next reader the trace.

health.into_iter().chain(chat_routes()).collect()
}

fn chat_routes() -> [RouteDoc; 3] {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (architectural, confidence 80): chat_routes() hardcodes the three OpenAI route literals (/v1/models, /v1/chat/completions, /v1/completions) instead of mapping over rocm_engine_protocol::OPENAI_CHAT_ROUTES, even though routes_for just above (line 666) correctly reads the health route from the shared engine_health_route helper. The protocol crate's own doc comment on OPENAI_CHAT_ROUTES says the API docs describe the same list — today that's true only because tests (documented_routes_are_the_routes_each_engine_reports and similar) keep the two in sync, not because this function reads from the shared constant. Worth deriving this [RouteDoc; 3] from OPENAI_CHAT_ROUTES directly so the "single source of truth" the PR describes is structural rather than test-enforced.

Comment thread README.md
quick smoke test (time to first token and approximate tokens per second). Control
with the status, the engine, the API-qualified model name, the full inference
endpoint, the command that serves the server's API docs (see API docs below),
and a quick smoke test (time to first token and approximate tokens per second). Control

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking (mechanical, confidence 85): this line and two others touched by this PR break the surrounding paragraphs' ~75-80 column wrap convention: this line runs to 87 columns, line 1585 ("...vLLM also documents its full API at") to 100, and line 1590 ("...work offline. They use port 11430, or a free port when 11430 is") to 107. Purely cosmetic — worth re-wrapping to match the rest of the file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants