Repository navigation
feat(examine,serve): report real GPU/backend support instead of a static ROCm claim #678
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
e12ed39
c63fa43
67cd482
61bb3c3
9f78b66
8ada511
18666d4
1b85b9e
a9ed8fa
7fba9aa
e1daa50
a3f27af
6d79b3d
f4e02f7
11819e9
68bea03
f72bfb3
ea43070
73ec5ae
c65224a
34845bf
c806ee3
8ffeca6
5de5b43
b653cc8
a9d03d1
130747e
6a92843
129b043
b6c37c6
61ae34a
d3b3e34
654c2de
3754e92
67c27ea
c03972e
545da1f
db00642
2ce0e14
9b5788e
3631908
c618148
c2446ba
f050eb1
e6c0df9
64dc074
cc9e90a
39bec51
de1e3f3
81a5f5b
557aa34
4608e96
074becf
d99fd16
414bf50
4a651b5
aca4f36
cd895c7
132394e
017a109
4e435f7
b8a064b
5ba7eb2
9e0f3c1
f04357f
7bfee25
06ece66
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -682,6 +682,12 @@ family is substituted for it. `rocm examine` reports the same conclusion on a | |
|
|
||
| These GPUs still work with the Vulkan backend, which needs no ROCm SDK. | ||
|
|
||
| `rocm examine` prints that `rocm_support` line on **every** host, not only this | ||
| case, at one of three levels: ROCm/HIP-supported, Vulkan-only (the case above), | ||
| or unknown. Unknown covers both no GPU detected and a target this build's | ||
| catalog does not recognise — detection gaps rather than a verdict about the | ||
| hardware, so neither is reported as "no ROCm build". | ||
|
|
||
| #### Compiler toolchain (--devel) | ||
|
|
||
| Pass `--devel` to also install the compiler and headers needed to build GPU | ||
|
|
@@ -1390,18 +1396,33 @@ the running service, or stop it first to serve on the new port. | |
|
|
||
| By default the server runs in the background under ROCm CLI's supervision and | ||
| prints a deployment summary: a progress indicator while it starts, then a table | ||
| with the status, the full inference endpoint, the API-qualified model name, and a | ||
| quick smoke test (time to first token and approximate tokens per second). On an | ||
| interactive terminal, if the smoke test cannot complete, the summary is headed | ||
| `(smoke test failed)`, shows the status as `started, smoke test failed` instead | ||
| of `ready`, and a note gives the reason and points at | ||
| with the status, the engine and the backend device the engine reports (for | ||
| example `rocm gpu 0 (AMD Instinct MI300X)`, or `vulkan (AMD Radeon Graphics)` | ||
| where the engine pinned no GPU ordinal). It is read back from the running | ||
| engine, so it is that engine's own account of the backend it was launched on | ||
| rather than an independent measurement of the hardware. The summary also carries | ||
| the full inference endpoint, the API-qualified model name, and a quick smoke | ||
| test (time to first token and approximate tokens per second). `rocm services` | ||
| reports the same `device` for each running service. | ||
|
|
||
| On an interactive terminal, if the smoke test cannot complete, the summary is | ||
| headed `(smoke test failed)`, shows the status as `started, smoke test failed` | ||
| instead of `ready`, and a note gives the reason and points at | ||
| `rocm logs --service <id>`. ROCm CLI does not stop the server, and the command | ||
| still exits 0. Which failures are detected, and which still read as `ready`, is | ||
| listed under | ||
| [Serve Smoke Test Failure](https://github.com/ROCm/rocm-cli/blob/main/docs/testing.md#serve-smoke-test-failure) | ||
| in the testing guide. Control | ||
| returns to your shell with the server still running; manage it later with `rocm | ||
| services` (below). | ||
| in the testing guide. | ||
|
|
||
| The device line is omitted when the engine cannot say yet. A server is reported | ||
| ready as soon as it answers, which can be before it has finished loading the | ||
| model, and until then the engine does not know which device it ended up on; | ||
| `rocm serve` waits a few seconds for it rather than holding your terminal, and | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Waits a few seconds" understates it: the code is a 6s start gate plus one in-flight probe (about 14s per the |
||
| `rocm services` shows it once it is known. The GPU name is omitted on a machine | ||
| with more than one AMD GPU, where naming one of them could name the wrong one. | ||
|
|
||
| Control returns to your shell with the server still running; manage it later | ||
| with `rocm services` (below). | ||
|
|
||
| `--verbose` (or `--foreground`) instead attaches to the server in the current | ||
| terminal and streams every engine log line. Use it to debug a startup problem. | ||
|
|
@@ -1646,6 +1667,15 @@ It does not suggest running the stop again: when only a process under the | |
| recorded one survived, a second stop can report the server stopped while that | ||
| process still runs. | ||
|
|
||
| `list` asks each running server which backend and GPU it is actually on, so it | ||
| can take a few seconds rather than returning instantly. The query is capped in | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "Capped in total ... however many servers" isn't true for external plugin engines: the code doc on |
||
| total — about 18 seconds worst case, however many servers are listed — and a | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "About 18 seconds worst case, however many servers are listed" only holds for built-in engines.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "About 18 seconds worst case, however many servers are listed" only holds for built-in engines: |
||
| server that does not answer in time simply has no `device` line rather than | ||
| holding up the rest of the list. The same applies to asking about servers in | ||
| `rocm chat`, which prints the same list. `list --json` is unaffected: it emits | ||
| the stored service records, which have no `device` field, asks no server | ||
| anything, and returns as fast as it always did. | ||
|
|
||
| `restart` starts a server again only once its stop has confirmed that every | ||
| recorded process is gone; if the stop cannot confirm that, it says so, starts | ||
| nothing, and leaves the stop request recorded. | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -32,15 +32,24 @@ use crate::managed_service_status::load_managed_services; | |
|
|
||
| pub(crate) fn build_codex_bridge_snapshot(paths: &AppPaths) -> Result<CodexBridgeSnapshot> { | ||
| let config = RocmCliConfig::load(paths).unwrap_or_default(); | ||
| // Hoisted out of the struct literal so the engine rows can be told what this | ||
| // host actually supports. The probe is already paid for: `gpu_support_level` | ||
| // takes exactly the two fields this summary already carries, so the engine | ||
| // list costs no detection the snapshot was not doing anyway. | ||
| let examine = ExamineSummary::gather()?; | ||
| let gpu_support_level = rocm_core::gpu_support_level( | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This positional
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This positional |
||
| examine.detected_gfx_target.as_deref(), | ||
| examine.compatible_therock_family.as_deref(), | ||
| ); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Blocking (test coverage) — this
Confidence 72. Suggested fix: add a test that plants an |
||
| Ok(CodexBridgeSnapshot { | ||
| protocol: "rocmd-codex-bridge-v0".to_owned(), | ||
| generated_at_unix_ms: rocm_core::unix_time_millis(), | ||
| examine: ExamineSummary::gather()?, | ||
| examine, | ||
| gpu: build_codex_bridge_gpu_snapshot(&config), | ||
| config, | ||
| automation_runtime: AutomationRuntimeState::load(paths)?, | ||
| recent_automation_events: load_recent_automation_events(paths, 32)?, | ||
| engines: builtin_codex_bridge_engine_inventory(), | ||
| engines: builtin_codex_bridge_engine_inventory(gpu_support_level), | ||
| services: load_managed_services(paths)?, | ||
| }) | ||
| } | ||
|
|
@@ -63,7 +72,22 @@ pub(crate) fn build_codex_bridge_gpu_snapshot(config: &RocmCliConfig) -> CodexBr | |
| } | ||
| } | ||
|
|
||
| pub(crate) fn builtin_codex_bridge_engine_inventory() -> Vec<CodexBridgeEngine> { | ||
| /// The engine catalogue in the codex-bridge snapshot and the bridge's `engines` | ||
| /// tool, built from [`engine_inventory`] with lemonade's note replaced | ||
| /// by `lemonade_engine_note` for the given support level. | ||
| /// | ||
| /// It takes the level rather than reading the static row because the stale | ||
| /// "with ROCm llama.cpp backend" claim is exactly what ROCMAI-550 exists to | ||
| /// remove, and this is a user-facing surface like the other two: a bridge | ||
| /// client renders `summary` to the operator. | ||
| /// | ||
| /// Only `summary` moves. The field set, its names and `installed_binary: true` | ||
| /// beside it are untouched, so the bridge's JSON shape is the same; what | ||
| /// changes is that one string stops asserting ROCm support on a host that has | ||
| /// none. | ||
| pub(crate) fn builtin_codex_bridge_engine_inventory( | ||
| gpu_support_level: rocm_core::GpuSupportLevel, | ||
| ) -> Vec<CodexBridgeEngine> { | ||
| let default_engine = default_engine_for_platform(); | ||
| let binary_path = daemon_binary_path() | ||
| .ok() | ||
|
|
@@ -72,7 +96,7 @@ pub(crate) fn builtin_codex_bridge_engine_inventory() -> Vec<CodexBridgeEngine> | |
| .iter() | ||
| .map(|(id, summary)| CodexBridgeEngine { | ||
| id: (*id).to_owned(), | ||
| summary: (*summary).to_owned(), | ||
| summary: crate::engine_inventory_note(id, summary, gpu_support_level).to_owned(), | ||
| default_for_platform: *id == default_engine, | ||
| installed_binary: true, | ||
| binary_path: binary_path.clone(), | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Waits a few seconds" understates it: the code is a 6s start gate plus one in-flight probe, about 14s per the
engine_ipc.rscomment, on a launch that ends without a row. Please align this withdocs/testing.mdand the code.