Repository navigation
fix(lemonade): resolve registered GGUF models and discover serving models in dash - #631
Conversation
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · ace6ca1
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
First automated round on this PR: every finding below is new.
Review — needs work
Full review of the whole change.
No ticket named. The description claims registered-GGUF resolution, read-only dash discovery, separate install and availability status, and doc updates. Findings 1, 3 and 4 below conflict with parts of those claims.
Blocking
-
A registered
llamacppcatalog entry the resolver doesn't fully understand now failsrocm serveon Linux, or is served without its extra files —engines/lemonade/src/direct_llama.rs:257-291,engines/lemonade/src/lib.rs:514-519
resolve_direct_llama_checkpointrequires a plain stringcheckpointof the formowner/repo:variantand returns an error otherwise.serve_httppasses that error up with?on Linux for every model reference. Before this change these names went to Lemonade's router.
I ran a probe test against the head code:- an entry with a
checkpointsmap returnsErr(registered modelVision-GGUFhas no checkpoint); - an entry whose checkpoint has no variant returns
Err(... needs an explicit quantization variant ...); - an entry with
checkpointplus a top-levelmmprojreturnsOk(direct), anddirect_llama_server_commandnever passes--mmproj, so the model is served without its vision projector and nothing reports it.
Lemonade documents thecheckpointsmap (main/mmproj/draft) as the current format and shows built-in models using it (custom models, API). I could not confirm the pinned 11.5.1 catalog itself, because the repo has no fixture of it. Aserver_models.jsonthat can't be read or parsed also hard-fails, including for names the file wouldn't contain.
Confidence 85 · logic · Fix: serve directly only shapes the resolver fully understands (a singlecheckpointwith a variant, nommproj/checkpoints/extra checkpoint types). ReturnOk(None)for everything else and for catalog read or parse errors, so the router handles them. Add a test for each shape.
- an entry with a
-
The new serve routing decisions have no test that fails when they break —
engines/lemonade/src/lib.rs:453-458,:514-519,:546-553;direct_llama.rs:297
I mutated each of these on its own and rancargo test -p rocm-engine-lemonade --lib; all 129 tests still passed every time:- disabled the bail for an HF checkpoint when no selected backend binary exists;
- inverted the Linux check in
launch_service; - forced
direct_modeltoNone, so registered names never take the direct path; - removed the
backend_recipe != llamacppguard indirect_serve_binary.
The GPU scenarios serve-07, serve-08 and dash-25 cover only the happy path.
Confidence 100 · logic · Fix: pull the direct / router / bail decision into a pure function (inputs: platform, parsed reference, catalog result, binary present) and unit-test each outcome, including the bail message.
-
launch_servicestops normalizing the model reference on Linux, with no stated reason, and engine state can then disagree with the served name —engines/lemonade/src/lib.rs:453-458,engines/lemonade/src/state.rs:152-153
rocm servealready passes the canonical name (serve_cmd.rs→resolve_model_response), so for that command this branch changes nothing.
On the engine's ownlaunchpath, the state file now records the raw alias (for exampleqwen, or text with spaces around it) asmodel_refandcanonical_model_id. The router path, however, servesresolve_lemonade_model_ref(...).healthcheck_servicereads the state value back, and its inference probe then uses the raw alias.
WSL also counts as Linux here. With this branch, model-name normalization now happens in three separate places.
Confidence 70 · logic · Fix: normalize once inlaunch_serviceas before. If the requested text must be kept, store it separately and keepcanonical_model_idas the name actually served. -
External models found by discovery can be selected for "stop"/"restart" in the Services overlay, and the approval text calls them managed —
crates/rocm-dash-tui/src/ui/services_manager.rs:93-110,:169-200
service_rowslists every daemon instance. Pressingsorron alemonade-<host>-<port>-model-<hex>row stagesrocm services stop <id> --yes, with the text "This runs the CLIservices stopfor this managed service."
The stop itself is harmless: the record lookup fails for an id that isn't in the registry, so nothing is killed. But the text is false for these rows, and the action can only fail.
The code was already there, but this change makes discovery default-on inrocm dash, so these rows now appear on every host running Lemonade.docs/manual-testing.mdworks around it ("Do not use managed stop or restart actions…") instead of the code preventing it. AGENTS.md §3 says a promise that something will not happen must be tested together with the behavior.
Confidence 85 · logic · Fix: give discovered rows an external/read-only marker. Hide or refuse lifecycle actions on them, and add a test thatson such a row stages nothing. -
A contract test's step claims something it cannot detect —
crates/rocm-dash-daemon/tests/telemetry_contract.rs(step "adding another model clears previously attributed rate immediately")
I deletedgen_trackers.remove(&row.id)inrunner.rs:897, so nothing clears the old rate.lemonade_multiple_models_keep_identity_and_do_not_share_last_request_ratestill passes; it took 7.1s instead of 1.2s because it waits for the held value to expire on its own. So the step does not prove "immediately".
For contrast, the "last access is not last request" step is a real check: breakingstats_model's single-model rule fails it at line 192.
Confidence 100 · logic · Fix: put a time bound on that wait that is shorter than the hold window, or check the first snapshot after the health change. -
Code left with no caller after the runner rewrite —
crates/rocm-dash-collectors/src/lemonade.rs:169(parse_health_model),:313(fetch_health_model),:334(discover),:117-121
The runner now calls onlydiscover_modelsandfetch_health. The other three have no non-test callers in the workspace (it ispublish = false), yet their doc comments say they exist "for compatibility/legacy callers".
Instats_model, the trailingif loaded.len() == 1 { … } else { None }can't be reached with any other length, because of thelen() != 1early return above it.
Confidence 85 · mechanical · Fix: delete them, or make them test-only and move their tests todiscover_models/fetch_health. Collapse the dead branch.
Non-blocking
- The new bail message has no remediation, and it is wrong for some cases —
engines/lemonade/src/lib.rs:546-553. The old message said "runrocm engines install lemonade". The new "no selected GPU llama-server backendXis installed" also fires whenbackend_recipeisn'tllamacppor the backend iscpu, where "not installed" is not the problem. Confidence 70 · mechanical · Fix: handle the recipe and cpu cases separately, and name a real command (rocm engines install lemonade --reinstallis already used inbackend_alignment.rs). - Comments made stale by this change — Confidence 85 · mechanical · Fix: update each:
tests/e2e-cucumber/features/model_serving.feature:73-77andtests/e2e-cucumber/tests/e2e/serving_steps.rs:613cite the removedserve_hf_checkpointand the removed "managed-lemonade fallback".crates/rocm-dash-daemon/src/runner.rs:67-69and:311-312still say discovery is "Opt-in … off by default", butrocm dashnow turns it on.- The
registry.rs:26-31module doc saysrecord.hostis honored only for the co-located daemon, but managed Lemonade is now scraped atrecord.host.
- Lemonade HTTP calls run one after another inside the daemon tick loop —
runner.rs:~678-686,~833-925. Each external or managed endpoint gets a health request and possibly a stats request, each with a 1.5s timeout (doubled by the 404 fallback), and a new reqwest client is built per managed endpoint on every tick. A hung endpoint delays every snapshot. The vLLM path next to it usesparallel_scrape. Confidence 70 · architectural · Fix: run the endpoint futures concurrently and reuse the collectors. - One failed health poll removes a discovered model —
lemonade.rs:323-330(unwrap_or_default),runner.rs:~697-709. A timeout or 5xx while Lemonade is busy emitsInstanceGone, and the next good poll emitsInstanceDiscoveredagain, so the row flaps. Before, this applied to one instance and discovery was opt-in; now it applies to every loaded model by default. Health errors are logged attrace!only. Confidence 70 · logic · Fix: require several consecutive misses before removal, and report health errors throughscrape_warns. - dash-27 ends on a step that checks nothing —
tests/e2e-cucumber/features/dash.feature:372-373. After "selects engine status in the Serving view", the final Then only waits for the heading text "Serving actions". Nothing checks that "Lemonade — externally running" is gone once the endpoint is unreachable. Confidence 85 · logic · Fix: replace it with an absence check on the engine pane. - "External" is recognized only by string matching —
crates/rocm-dash-tui/src/ui/tabs/pane.rs:393-398. The TUI matchescontainer_name == "lemonade"plus alemonade-id prefix against how the collector happens to build ids. Removing thestarts_withhalf fails none of the fourpane::tests, as I confirmed by running them. Confidence 70 · architectural · Fix: share a single predicate, or add an origin field toInstance; the Services-overlay fix above needs the same marker. - One commit holds several separate changes — AGENTS.md §11. It bundles the engine serving rewrite, a new dashboard discovery feature with config schema, a default flip, the engine-pane status, and a protocol fixture, about 1,900 inserted lines, with an empty commit body. Consider splitting the serving fix from the dash discovery work. Confidence 75 · architectural.
Decisions for the author
- Direct llama-server is now tried first for every registered
llamacppname on Linux, and the router's load error no longer falls back to direct — tradeoff
For: router load errors are no longer hidden, and the binary is chosen by the selected backend. Against: router-only per-model wiring (vision projector, draft and other checkpoint types, labels) is skipped whenever direct applies. Finding 1 is the correctness half of this; the question is whether bypassing the router for all registered names is intended. - The direct-serve binary lookup is now limited to
manifest.backend_name— tradeoff
For: a stale ROCm or Vulkan install can no longer be picked. Against: a host whose manifest saysrocmbut has onlyvulkanon disk (or the reverse) used to serve HF checkpoints and now fails. rocm dashnow probes127.0.0.1:13305by default — tradeoff
It's documented. The cost is a poll on every discovery tick on hosts with no Lemonade, and any local listener on that port can put rows into the dashboard.
Positive signals
- The
/api→/v1fallback happens only on a 404, never on an auth failure or 5xx, andhealth_http_failure_never_falls_back_to_successful_routelocks that in with.expect(0)plusverify(). - Throughput from Lemonade's server-wide "last request" stats is shown only when exactly one model is loaded, and breaking that rule fails both the unit test and the contract test, as I checked by mutation.
- Catalog metadata is treated as untrusted: checkpoints that look like path traversal or local paths are rejected and tested, and CPU or unknown backend names are refused.
- dash-26 checks the external fixture is still alive after the dashboard quits and that no service record was created, so the behavior is asserted, not just the wording.
Deployment notes
rocm dashnow reads the newdashboard.daemon.enable_lemonadesetting (defaulttrue) and probeslemonade_host:lemonade_port(default127.0.0.1:13305). Existing config files pick up these defaults. A standalone daemon built fromRunnerOptions::default()stays opt-in.- On Linux,
rocm serve <registered llamacpp name> --engine lemonadenow runs the packaged llama-server directly on the selected backend whenever the name resolves in the catalog, rather than going through Lemonade's router.
What this covered
Every file in the change at 34fbdfe9…ace6ca15, plus callers and siblings. Workers looked at:
services_manager.rsand theservices stoppath;serve_cmd.rsandmain.rsreadiness, and the engine'sstate.rs,install.rsandbackend_alignment.rs;expectations.tomland the serve step definitions.
Four coupled code passes, three design questions, and the four source-reading passes (agent-instruction adherence, history, prior PRs #80/#94/#104/#242/#295/#404, code comments) ran as separate workers.
What I ran myself:
- The unit and contract tests for the lemonade engine, the dash collectors and daemon, and the TUI pane, plus seven mutations and one probe test. Each run's test names and counts were checked.
- The leak scan was clean.
- At review time the Strix Halo Ubuntu (release) GPU lane had passed serve-07, serve-08 and dash-25 through dash-27.
Did not run or only partly ran:
- The e2e suite was not run locally. The dash-27 finding is from reading the step definitions.
- The TUI worker read
event_loop.rs,app/mod.rsandinstances.rsonly as their one-line diffs plus call sites, not completely; it also did not read the rubric reference. - MI300X, Strix Halo Windows and WSL2 had not reported; Ubuntu nightly was still running.
- The Lemonade catalog shape comes from Lemonade's public docs, not from the pinned 11.5.1 file.
Test verdicts:
- The engine
serving_*tests and therocm-core/dash.rsconfig tests would fail if the production change were reverted. The routing branches inside the change are not covered (finding 2). - In the contract test, the "immediately clears" step passes with the clearing removed (finding 5); its other steps are real checks.
- The pane tests do not cover the id-prefix half of the "external" check.
This report is not reconciled against the PR discussion.
ace6ca1 to
5f26173
Compare
Addressed in 5f26173: preserve router wiring for complex catalog models, persist the served identity, restrict lifecycle controls to managed services, and strengthen routing and telemetry regressions. Rebased onto current main; local workspace and dashboard protocol checks passed. Updated CI is running.
ac02905 to
93324b5
Compare
|
🔴 Automated review · pr-review-watcher · ac02905 This automation never files a GitHub approval, so no approving review will Informational — describes Review — needs workFull review of the whole change. Blocking
Non-blocking
Decisions for the author
Positive signals
Deployment notes
What this covered
Did not run:
Real Lemonade Not reconciled against any PR discussion. |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 93324b5
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Round 3. ac029055, the commit the previous round described, was force-pushed away, so this is a full review of 93324b5e. [new] marks findings this round raised for the first time. [standing] marks objections from round 2 (the informational comment on ac029055) that still hold. The last four Blocking items were not re-found by this round's blind review. They were settled afterwards by putting each one, in its original wording, to the reviewer that had just read this head. That check is anchored, meaning it was told what to look for, so it is weaker evidence than a blind finding; each item carries its verdict. The description finding would not gate on its own, because the squash commit is built from commit messages; the other blockers do gate.
Review — needs work
Full review of the whole change.
No ticket named. The description covers the registered-GGUF resolution and the dash discovery work. It leaves out several behaviour changes from the second commit (see Blocking).
Blocking
-
[standing, in a new form: round 2's "dash-25 hangs both Strix Halo Ubuntu lanes"] The required GPU lanes never ran on this head, so the hardware-only scenarios are unverified. Locations:
tests/e2e-cucumber/features/dash.feature:332(dash-25),tests/e2e-cucumber/features/model_serving.feature:326(serve-25),tests/e2e-cucumber/expectations.toml:96-101.- The self-hosted e2e run
38359301083for93324b5ehas satpendingwith no jobs since 06:53Z. No GPU lane appears instatusCheckRollup. - That means three things never ran: dash-25 (per-PR,
@requires-gpu), serve-08, and the nightly serve-25. serve-25 is the only end-to-end check that complex catalog entries stay on the Lemonade router. - dash-25 reuses the step "a GGUF model is being served on lemonade".
serve-lemonade-inferenceuses the same step and has a Linux/lemonadeflakyxfail row (EAI-7423). dash-25 has no row, so it may fail hard on the Strix Ubuntu lane. - Confidence 100 · architectural · Fix: get the self-hosted run scheduled (it is stuck, not failed). Then either add a matching expectations row for dash-25 or show on hardware that EAI-7423 is fixed. Dispatch
include_nightlyonce for serve-25 and cite that run.
- The self-hosted e2e run
-
[standing: round 2's "router fallback removal has no test", now with new mutations] Several lemonade engine branches this change introduces are not exercised by any test. Locations:
engines/lemonade/src/lib.rs:546,:569,:578,:585,:588-611;engines/lemonade/src/direct_llama.rs:152-159;engines/lemonade/src/process.rs:814-856.- Each mutation below left the whole lemonade lib suite green (143 tests):
- persisting
request.model_refinstead ofrouter_model_refat:546; - passing
request.model_refto the load, loaded-model and chat-smoke calls; - pulling the alias instead of
checkpoint.model_ref(); - swallowing the pull error with
.ok().
- persisting
- The removed router-failure fallback has no test either.
serving_gpu_model_load_process_error_is_returnedcallsrun_lemonade_model_load, whose body this diff does not change, so it would still pass with the fallback restored. - That same test is also not hermetic. With
XDG_RUNTIME_DIRunset it tries to create the real~/.rocm/data/lemonade. On a clean tree here it failed withassertion failed: error.to_string().contains("Lemonade model load failed")(env -u XDG_RUNTIME_DIR cargo test -p rocm-engine-lemonade --lib serving_gpu_model_load_process_error_is_returned: 1 run, 1 failed). - Confidence 100 · logic · Fix:
- Make the router branch's state persistence and failure handling testable, for example with a seam around spawn and load, and assert the id that gets persisted.
- Assert the
pull owner/repo:variantargv and that a non-zero exit is propagated, using the existing fake-executable pattern. - Give the load test an explicit
ParentRuntimeEnvironmentor a tempdir runtime dir.
- Each mutation below left the whole lemonade lib suite green (143 tests):
-
[new] Several dashboard branches this change introduces are not exercised by any test. Locations:
crates/rocm-dash-daemon/src/runner.rs~641-656, ~741-757, ~858-868;crates/rocm-dash-tui/src/ui/services_manager.rs:142;crates/rocm-dash-tui/src/ui/tabs/pane.rs(mark(...)).- Each of these mutations left the relevant suite green:
- Model/host-change telemetry reset: turning it into a no-op left all 7 lemonade contract tests green.
- Scrape-only cadence: forcing
discover_external: trueleft everything green, because every new contract test setsdiscovery_tick == instance_tick. That leaves the "hold rows on a scrape-tick health failure" path unexercised. - Selection clamp: removing it left all 25 TUI tests green.
- Pane glyph: forcing
mark(false)left all 25 TUI tests green.
- Confidence 100 · logic · Fix:
- Add a contract test that rewrites a managed record's model while the clock is held, and asserts the rate resets.
- Add one test with
discovery_tick= 3 ×instance_tickthat covers the scrape-tick hold. - Add a TUI test that shrinks the rows below
selectedand then pressess. - Assert the ✓/· glyph.
- Each of these mutations left the relevant suite green:
-
[standing in part: round 2's "external server remains alive after quit check cannot fail"; the other checks are new] Some new checks claim coverage they cannot fail on. Locations:
engines/lemonade/src/direct_llama.rs:584-587,:461and:875-876;tests/e2e-cucumber/tests/e2e/dash_steps.rsexternal_survives_quit(dash.feature:356,361) andreal_lemonade_visible.assert_eq!(requested.model_ref, "qwen-smoke", "direct aliases are not normalized")checks a valueserving_routeonly borrows, so it is always true.- The test comment says the
is_file()filter incollect_gguf_filesis covered. The fixture is a directory that the earlieris_dir()branch handles first, so removingis_file()leaves everything green. - The e2e step "the protocol fixture remains alive" checks an in-process fixture thread that nothing the dashboard does can stop.
- "Real managed Lemonade model is displayed" only waits for "Qwen3-0.6B" anywhere on screen. Managed rows were already shown before this change.
- A check that names a branch it cannot fail on is worse than no check, because it stops the next reader from looking.
- Confidence 85 · mechanical · Fix:
- Assert the
--aliasproduced bydirect_llama_server_commandalong thelaunch_servicepath. - Add a dangling-symlink fixture (
#[cfg(unix)]) and correct the comment. - Run the fixture as a child process and check its PID, or rename the step to what it really asserts ("no managed records").
- Anchor the managed-row check to that row's port or status.
- Assert the
-
[standing in part: round 2 flagged the "remaining alive" overclaim; the omissions are new] The description misstates the change's scope.
- It does not mention four things the diff does:
- It removes the Linux fallback to direct llama-server after a router load or verify failure (
lib.rs:588-611). - The Services overlay now lists only
managedinstances, so Docker-discovered containers that used to appear are gone (services_manager.rs:98). - Complex catalog entries now go to the Lemonade router.
- It adds the nightly scenario
@id:serve-lemonade-complex-registered-inference.
- It removes the Linux fallback to direct llama-server after a router load or verify failure (
- It claims "the external endpoint remaining alive after dashboard quit" is covered "with resulting-state assertions". That assertion cannot fail (previous finding).
- Confidence 85 · mechanical · Fix: update the description to list these changes. Either narrow the "remaining alive" claim or make the assertion real.
- It does not mention four things the diff does:
-
[standing, not re-found blind, settled by objection check: HOLDS]
discover_modelshas no production caller; its three tests check a copy of the logic the daemon runs —crates/rocm-dash-collectors/src/lemonade.rs:292-300,crates/rocm-dash-daemon/src/runner.rs:800-814- Only tests in the same file call
discover_models, and a workspace grep finds no production caller. The runner repeats the health →loaded_models→lemonade_servicemapping inline. Its error handling differs: it propagates thefetch_health()error with?, whilediscover_modelsusesunwrap_or_default(). - Partly mitigated: the shared helpers (
fetch_health/get_text,parse_health,loaded_models,lemonade_service) are production code. Only the way they are put together, and the error-to-empty behaviour, are tested on a copy. - Fix: extract one pure function both paths use, or delete
discover_modelsand point the tests at the runner's path.
- Only tests in the same file call
-
[standing, not re-found blind, settled by objection check: HOLDS, confirmed by a run] No test covers dropping external rows when a managed record takes over the endpoint —
crates/rocm-dash-daemon/src/runner.rs:726-745- With the
for gone in &lemonade_ids { … }loop andlemonade_ids.clear()deleted,RUSTC_WRAPPER= cargo test -p rocm-dash-daemonstayed fully green: lib 45, a second binary 9,telemetry_contract16. All 7lemonade_*contract tests ran by name. lemonade_external_endpoint_deduplicates_managed_owner_then_reappears_after_record_removalwrites the managed record before the runner starts (telemetry_contract.rs:328-329). So the transition from external rows to a managed record never happens.- Fix: add a contract case where external rows exist first and a matching registry record then appears.
- With the
-
[standing, not re-found blind, settled by objection check: HOLDS, read from code] Cache-root priority is lost when the same GGUF exists in several roots —
engines/lemonade/src/direct_llama.rs:428-446,:477,:115- The base served
hf_cache_roots(..).into_iter().find_map(..), so the first root in priority order won (prw-basedirect_llama.rs:327-329). - Now every root is flattened,
filter_ggufs_by_variantsorts by full path, andmatches.into_iter().next()is served. Nothing canonicalizes afterwards, so the alphabetically first path wins. It matters when copies differ, for example a stale~/.cachesnapshot against the configuredHUGGINGFACE_HUB_CACHE. - The only multi-root test (
:865-893) assertsfile_name()alone, so it cannot see which root won. - Fix: filter per root in priority order, and add a two-root test that asserts the full path.
- The base served
-
[standing, not re-found blind, settled by objection check: HOLDS, read from code]
lemonade_managed_installedhas no test that tells "installed and managed" apart from "installed" —apps/rocm/src/dash.rs:252-261- The only unit test in
dash.rscovers config. The TUI tests set theOptiondirectly. - Hosted dash-26/27 reach only a Detect that succeeds with
installed=false. So neither dropping themanaged_envconjunct nor mapping a Detect error toSome(false)is caught there. - Only dash-25 exercises
Some(true), and dash-25 did not run on this head (first finding). The check could not confirm round 2's claim that dash-25 hangs. - Fix: inject the Detect result and unit-test the three outcomes.
- The only unit test in
Non-blocking
- [standing] Stale "opt-in / off by default" comments —
crates/rocm-dash-daemon/src/runner.rs:88-90,:345-347.rocm dashnow enables discovery by default throughDashboardDaemonConfig.enable_lemonade(crates/rocm-core/src/lib.rs:2413), whichapps/rocm/src/dash.rs:61passes through. Confidence 85 · mechanical · Fix: say discovery is on by default inrocm dashvia config, and that only the bareRunnerOptions::default()is off. - [new]
rocm dash --helppromises discovery on every platform —apps/rocm/src/main.rs:598-602. The embedded daemon returnsNoneon Windows (apps/rocm/src/dash.rs:975-978), so nothing is discovered there. README already scopes the claim to Linux/WSL. Confidence 85 · mechanical · Fix: prefix the help text with "On Linux/WSL,". - [new] A failed router serve leaves engine state at
running, and this now happens on Linux too —engines/lemonade/src/lib.rs:588-611. Both error arms kill the child and return withoutmark_json_status(.., "failed"), unlike the direct path (process.rs:536-540). Before this change, Linux took the fallback instead. The rocmd record does flip to failed. Confidence 85 · logic · Fix: mark the state failed in both arms. - [new] Remediation hints dropped —
engines/lemonade/src/direct_llama.rs:56-75,:98-112. The missing-backend error no longer says to runrocm engines install lemonade(that command does fix it), and the variant errors lost their{model}:Q4_K_Mexamples. Confidence 85 · mechanical · Fix: restore both. - [new] "open to check" points the user at an overlay that never shows install status —
crates/rocm-dash-tui/src/ui/tabs/pane.rs(Lemonade and managed-runtime lines).draw_engine_manager(engine_manager.rs:252-300) shows no Detect status. The wording copies the existing vLLM line. Confidence 85 · logic · Fix: say "unknown — runrocm engines list", or show status in the overlay. - [new] Install status is captured once per session —
apps/rocm/src/dash.rs:314,crates/rocm-dash-tui/src/app/event_loop.rs:805. Installing Lemonade from the Engines overlay leaves "managed runtime not installed" on screen until restart. Confidence 85 · logic · Fix: re-run Detect when an engines job completes. - [standing] Wrong URL in errors after the
/v1fallback —crates/rocm-dash-collectors/src/lemonade.rsget_text~309-325. The message reports the original/api/v1URL with the fallback's status, and the transport error drops the host. Confidence 85 · mechanical · Fix: format errors with the URL that was actually requested. - [new] Leftover parameter from the fallback design —
engines/lemonade/src/process.rs:451,lib.rs:528.reason: &anyhow::Erroris now always a fixed string and only gets logged. Confidence 85 · mechanical · Fix: make it&stror drop it. - [new] Stale doc and comment wording — three places:
docs/testing.md:852still calls direct serving "the packaged llama-server fallback".direct_llama.rs:240-244: theDIRECT_LLAMA_SERVER_BACKENDSdoc describes lookup ordering it no longer drives.direct_llama.rs:1098-1111: test names and comments refer to the removedfind_llama_server_binary.- Confidence 85 · mechanical.
- [new] A managed service bound to
::can show up twice —crates/rocm-dash-daemon/src/registry.rs~257-261 andendpoints_match. A dual-stack::bind on a non-default port 13305 is also reached by the external probe, giving a false "externally running" row and double-countedgen_tps. Confidence 70 · logic · Fix: match a::record against both loopback families, and add a test case. - [new] A twin construction drops ownership —
apps/rocm/src/dash.rs:621-628.launcher_serving_instancesbuildsInstancefrom registry records withoutmanaged: svc.managed, whilemerge_instancecarries it. Harmless today, because the launcher does not read the flag. Confidence 70 · mechanical. - [new] A stale approval swallows one keypress —
services_manager.rs:132-139. The modal is hidden, but the next key only clears it. Confidence 70 · logic · Fix: fall through and handle the key normally. - [new] Managed direct llama-server rows may carry a standing "lemonade health" warning —
registry.rs:257-270,runner.rs:862-868. Every Lemonade-engine record is scraped at/api/v1/health, which a direct llama-server likely does not serve the way the collector expects. Inferred from the code, not run. Confidence 70 · logic. - [new] Match arms the current writers never reach —
direct_llama.rs:258-259(rocm-stable/rocm-nightly). Manifest writers only ever storerocmorvulkan; deleting both arms left all tests green. Confidence 70 · mechanical. - [new] A dead branch in
lemonade_service—lemonade.rs~247-266. Every caller passes a trimmed, non-empty name, so the empty-name path is unreachable and untested. Confidence 85 · mechanical. - [new] The manual-testing sequence cannot run as written —
docs/manual-testing.md:219-224.rocm serve qwen-smoke … --port 11435without--managedattaches to the log, so the nextrocm services stopline is unreachable. Confidence 70 · mechanical. - [new] README never says where
config.jsonlives —README.md:196-199. Editing the file is the only way to setdashboard.daemon.*, but the path and theROCM_CLI_CONFIG_DIRoverride are not documented. Confidence 70 · mechanical.
Decisions for the author
- Removing the router-failure → direct llama-server fallback on Linux — tradeoff. Pro: a real load error is no longer hidden behind a cached-model direct serve. Con: the narrow set of users whose router load failed while a direct server was available now gets an error on first serve. Confirm this is intended, and say so in the description.
- The default model's Linux route now depends on catalog labels — non-blocking-improvement.
resolve_direct_llama_checkpointserves directly only when every label is in {reasoning, tool-calling, tools}. If the pinned 11.5.1server_models.jsonlabelsQwen3-4B-Instruct-2507-GGUFwith anything else (Lemonade's generated docs showhot), defaultrocm servesilently moves to the router path, which is the one the EAI-7423 xfail rows describe. A unit fixture with the pinned entry would lock this down. - Provenance as
managed: boolpluscontainer_name == "lemonade"— non-blocking-improvement. A Docker vLLM container namedlemonadeis labelled "Lemonade — externally running". The lifecycle gate usesmanagedalone, so the impact is cosmetic. An explicitenum Provenance { Managed, Docker, ExternalLemonade { host, port } }would make the confusion unrepresentable and carry the endpoint the dedup needs. - The external probe is awaited inside the daemon tick loop — non-blocking-improvement. With an unreachable or slow
lemonade_host, each discovery tick can stall GPU sampling and the vLLM scrape for 1.5–3 s. This follows the vLLM collector's existing pattern. A separate task feeding a channel would isolate it. - Scope — non-blocking-improvement. 33 files across two separable changes: lemonade serving-route resolution, and dash external discovery/ownership. Splitting them would make each easier to review and to revert.
Positive signals
serving_routedecides direct vs router before any download or spawn, and re-checks the GPU-required policy there for both platforms, so no CPU path is introduced (direct_llama.rs:44).- Direct-server readiness now requires an exact alias match (
process.rs:650). Mutating it back to fuzzy matching fails the tests. - Ownership is assigned in exactly one place (
registry.rs:169), and legacy snapshots default tomanaged: false. An old snapshot can never unlock stop/restart. stats_modelrefuses to attribute a last-request rate unless exactly one model is loaded. The dash-26 fixture puts the rate's model first so that naive attribution fails.- The
sr+ single-Esccheck inexternal_lifecycle_keys_do_nothingis a sound black-box probe: any staged approval or job console would keep the overlay open.
Deployment notes
rocm dashnow probes127.0.0.1:13305for Lemonade on every discovery tick by default (Linux/WSL embedded daemon). It adds the config keysdashboard.daemon.enable_lemonade,lemonade_hostandlemonade_port. They have serde defaults, so existing configs keep loading.- The Services overlay no longer lists unmanaged instances, Docker-discovered containers included.
- On Linux, a router load or verify failure in
rocm servewith the lemonade engine is now an error instead of a fallback to direct llama-server.
What this covered
- What was read. Every file in
prw-base…93324b5e(a02a859a…93324b5e, two commits), split across seven read-only workers grouped by coupling:- lemonade
direct_llama/state; - lemonade
lib/process; - dash daemon, collectors and core;
- dash TUI plus
apps/rocm/src/dash.rs; - e2e features and steps;
- docs plus AGENTS.md adherence;
- history, prior changes and code comments.
- lemonade
- Design questions. Two more workers took the design questions in Step 7. The callers of changed symbols were traced.
- Prior changes. The pass consulted the review comments of #80, #94, #242, #404 and #457.
- Tests run. Mutation verdicts came from real test runs, mostly in per-worker copies. The non-hermetic test failure was reproduced on a clean tree.
- Did not run / limits:
- The TUI worker read only the diff hunks of
crates/rocm-dash-tui/src/app/mod.rsandapp/event_loop.rs(4- and 1-line changes), not those files in full. - Workers shared this checkout and briefly mutated tracked files in it. The local e2e run of dash-26/27 (2 scenarios, both passed, and also green in hosted CI) may have been built with another worker's
pane.rsglyph mutation in place. - The
apps/rocmtestlemonade_discovery_defaults_and_overrides_reach_daemonwas not run because of the time box. - GPU scenarios (dash-25, serve-08, serve-25) could not run: there is no GPU here, and CI never scheduled them.
- The pinned Lemonade 11.5.1 catalog was not available offline, so the default-model label question is unresolved.
- The
ai-instruction-qualitycheck was not needed, since no agent-instruction files changed. - The leak scan of the diff was clean, and no prompt-injection content was found.
- The TUI worker read only the diff hunks of
- Not reconciled. This report was not reconciled against the PR's discussion.
cd8558a to
7778139
Compare
|
Thanks for the round 3 review. I pushed Blocking
Non-blocking
Decisions
Verification: the full workspace suite passed serially (33 targets, 3,775 tests), and so did both clippy commands, fmt, crate edges, smoke, and the external protocol scenarios. I could not run GPU scenarios, the Windows build, or |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 7778139
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
No ticket named; the description's claims were checked against the code (see the first and third Blocking findings).
Blocking
-
[new] Several serve error paths never mark the engine state
failed, while the docs and description say every error after the spawn does —engines/lemonade/src/router_serve.rs:49-56,:95-111,:112;engines/lemonade/src/lib.rs:509-529;docs/architecture.md:161
serve_routeronly stops the server and callsmark_json_status(.., "failed")for errors raised inside theoutcomeclosure. Three?exits outside it skip both: thewrite_running_state_for_modelright after the spawn, themerge_json_stateonce the model is ready, andwait_server().LemondChildhas noDrop, so on those pathslemondis left running and the state staysrunningorready.
The direct route has the same gap. Errors fromserving_routeandensure_direct_llama_model_available(missing binary, invalid checkpoint, ambiguous variant, a failed pull) leave the state at thestartingthatserve_httpwrote. This change fixed the router copy of this pattern but not the direct one.
docs/architecture.md:161says the module will "mark engine state failed on any error after the spawn". The code is what governs here, since it is what actually runs, so the doc claim is false.
Confidence 85 · logic · Fix: one guard inserve_httpthat marks the state failed (and stops any spawned child) on everyErrafterstartingis written. Otherwise narrow the doc line to what the code does. -
[new] The only end-to-end scenario for the managed engine pane's "installed" state can never go red —
tests/e2e-cucumber/expectations.toml:106-111,tests/e2e-cucumber/features/dash.feature:332
dash-25 is tagged@requires-os:linux @requires-gpu @requires-engine:lemonade. Its new row matches{ effective_engine = "lemonade", os = "linux" }withflaky = true, and that covers every lane the scenario can run on.flakyaccepts both pass and fail.
So the "Lemonade — managed runtime installed" assertions have no scenario that can fail, which AGENTS §3 requires for user-observable behaviour. The dash assertions are hidden behind an unrelated serve bug (EAI-7423). The description says the row exists, but not that it leaves this behaviour without a gate.
Confidence 85 · logic · Fix: assert the managed pane from a scenario that doesn't need the flaky real serve (for example a planted registry record plus a real Detect), or narrow the row to the lanes where the serve bug actually occurs. -
[new] Text about the CLI's own behaviour is pinned with no pointer to a test of that behaviour, despite the description's checkbox —
apps/rocm/src/main.rs(dash_help_scopes_lemonade_discovery_to_linux_and_wsl);crates/rocm-dash-tui/src/ui/tabs/pane.rs:432-438andstartup_snapshot_note_shown_only_with_a_verdict;engines/lemonade/src/direct_llama.rs:65-73and its test at:660-672
AGENTS §3: "Where the text genuinely has to be pinned on its own, the assertion carries a comment naming the test that proves the behaviour". None of these three does:- "quitting leaves external servers running" (help text);
- "Install status is read at startup. Restart the dashboard to refresh." (engine pane);
- the advice to run
rocm engines install lemonade --reinstall, which §3 says must be shown to actually fix the condition.
The description ticks "Discovery and ownership messages are asserted together with the resulting state."
Confidence 85 · mechanical · Fix: add the pointer comments (dash-26 /@id:dash-lemonade-external-protocol-discoveryexists for the first), and add a test that restores the backend through the reinstall path, or name one that does.
-
[new] A tracked file is cited as a bare backtick path in new
docs/text —docs/testing.md(the dash-25 bullet: "serve_timeout_secsinexpectations.toml")
AGENTS §5 requires repo files cited fromdocs/to be relative links. The same section already links../tests/e2e-cucumber/expectations.tomla few lines later.
Confidence 85 · mechanical · Fix: make it[expectations.toml](../tests/e2e-cucumber/expectations.toml). -
[standing in part since 93324b5 — not re-found blind; settled by putting it to the reviewer after its report was frozen (anchored verdict): HOLDS IN PART] The GPU lanes have not run on this head, so the hardware-only scenarios are unverified. The part about dash-25 lacking an expectations row no longer holds — that row now exists (and is the second new finding above).
- The only self-hosted run for 7778139, "E2E self-hosted" run 38386320833, has been pending with zero jobs since 10:52Z. No GPU, Strix or WSL check run appears for this head, so dash-25, serve-08 and serve-25 have no hardware result.
- No
include_nightlydispatch exists for this head. serve-25 is still the only end-to-end check that complex catalog entries stay on the Lemonade router. - Fix: get the self-hosted run scheduled (it is stuck, not failed) and dispatch
include_nightlyonce for serve-25; cite both runs.
-
[standing in part since 93324b5 — not re-found blind; settled by putting it to the reviewer (anchored verdict, read-only, no mutations run): HOLDS IN PART] Some lemonade engine connections this change introduces are still exercised only through fakes. The new
router_serve.rstests, the pull-argv test, the pull-error tests and the now-hermetic load test cover most of the original mutations. What remains:- The removed router-to-direct fallback is covered only structurally; nothing fails if one is re-added in
serve_http(the author agrees). - The real
LemondRouterSteps::load_model,model_loadedandchat_smoke(router_serve.rs:176-205) could ignore the name they receive and useself.request.model_ref;router_name_not_alias_reaches_load_loaded_and_smoke_checksdrivesFakeStepsonly. - The real pull closure in
ensure_direct_llama_model_available(direct_llama.rs:137-169) could pullrequest.model_refinstead of the checkpoint ref and stay green. - Fix: one test through the real
LemondRouterStepsagainst a fake HTTP server asserting the name on the wire, and one through the real pull closure with the existing fake-executable pattern.
- The removed router-to-direct fallback is covered only structurally; nothing fails if one is re-added in
-
[standing in part since 93324b5 — not re-found blind; settled by putting it to the reviewer (anchored verdict): HOLDS IN PART] Two checks still claim coverage they cannot fail on. The
is_file()dangling-symlink test and the renamed stray-request step now fail as they should.- The alias test (
direct_llama.rs:605-628) builds the command from its ownrequestedvalue, so thelaunch_service/serve_httppath todirect_llama_server_commandis never exercised. That is also why thelaunchnormalization change listed under Non-blocking is uncaught. - "Real managed Lemonade model is displayed":
observe_row_cells(dash_steps.rs:302-307) needs a cell starting with "Qwen3-0.6B" followed by 7 more cells, with no port or status check. Managed rows already appeared before this change, so the step passes with it reverted — and dash-25 sits under a flaky row anyway. - Fix: drive
launch_serviceorserve_httpand assert the served--alias; anchor the managed-row check to that row's port or status.
- The alias test (
Standing objections from 93324b5 that no longer hold
Each was not re-found by this round's blind review, then put to the reviewer one at a time with the author's reply (anchored verdict, from reading; no mutations could run). All were judged no longer to hold:
- Dashboard branches (telemetry reset, scrape-tick hold, selection clamp, glyph): each now has a named test that would plausibly fail under the original mutation.
discover_models: removed; the collector tests call the runner'sservices_from_health.- External rows dropped on managed takeover:
lemonade_managed_record_taking_over_endpoint_drops_existing_external_rowsexercises the transition. - Cache-root priority: per-root filtering in priority order;
checkpoint_matches_prefer_the_highest_priority_cache_rootorders roots so alphabetical and priority order disagree, and asserts the full path. lemonade_managed_installed: the puremanaged_installed_from_detectis unit-tested on all outcomes.- Description scope: the description now lists the removed fallback, the managed-only Services overlay, complex-entry routing and serve-25. The commit messages still do not mention the removed fallback or the Services-overlay change; that is an omission, not a false claim, and does not gate.
Non-blocking
-
[new] A stale cached collector can take over a new managed endpoint and dial the wrong IP family —
crates/rocm-dash-daemon/src/runner.rs:770-797,crates/rocm-dash-daemon/src/registry.rs:251-272
Grouping chainsmanaged_lemonade_collectors.keys()intoendpoints_match, which is deliberately non-transitive:::matches127.0.0.1. Say a::record is removed and a127.0.0.1service takes the same port. It keeps the cached collector, which dialsscrape_host("::") = "::1"against an IPv4-only listener, and the warning never clears. This contradicts the in-code comment "without … conflating IPv4 and IPv6 listeners". No test exercises the cached-key reuse branch.
Confidence 70 · logic · Fix: reuse a cached collector only whenscrape_host(cached) == scrape_host(target.host). -
[new] Managed Lemonade rows now show the last request's throughput as a fresh live rate indefinitely —
crates/rocm-dash-daemon/src/runner.rs:925-935,crates/rocm-dash-collectors/src/lemonade.rs:121
/api/v1/statsreports the most recent request (module doc, lines 8-10). Every scrape callsobserve_direct, which refresheslast_observed_at, so an idle server keeps showing, say, 42 tok/s as Fresh. External rows already behaved this way; this change extends it to every managed Lemonade row. The doc comment at :121 calls the value "instantaneous".
Confidence 70 · logic · Fix: fingerprint the stats tuple and stop refreshing freshness (or report 0) while it is unchanged. -
[new] The engine's
launchmethod no longer normalizesmodel_ref, and the description doesn't mention it —engines/lemonade/src/lib.rs:451vs:409
Through Launch,qwen-smokeis now served with--alias qwen-smoke, and direct readiness requires that exact name.resolvestill reportsQwen3-0.6B-GGUF.rocm serveandrocmdare unaffected, because they passcanonical_model_idtoserve-http. This still changes therocm-engine-protocolcontract (AGENTS §6).
Confidence 70 · architectural · Fix: keep normalizing inlaunch_service, or pin the new identity with a protocol test and mention it. -
[new] The new lib.rs test reads with
unwrapfrom sockets that are non-blocking on Windows —engines/lemonade/src/lib.rs:694-726(serving_regression_persisted_identity_drives_health_chat_model)
listener.set_nonblocking(true), thenstream.read(..).unwrap()on the accepted streams. On Winsock, accepted sockets inherit non-blocking mode, so an early read returnsWouldBlockand panics. The Windows lane runscargo test; it passed once on this head.
Confidence 70 · mechanical · Fix: callstream.set_nonblocking(false)afteraccept. -
[new]
run_rocm_boundedlimits the whole CLI serve to the harness's readiness window —tests/e2e-cucumber/tests/e2e/serving_steps.rs:150-151,tests/e2e-cucumber/tests/e2e.rs(run_rocm_bounded)
For@serve-timeout:2400the harness clock starts at spawn, but the CLI's own wait (ROCM_CLI_VLLM_READY_TIMEOUT_SECS) starts after its pre-launch work. The harness therefore kills the CLI first, and the CLI's own diagnostic is lost in the slow cold-load case. The 90 s rows now also have to fit any cold backend or weights fetch into that window.
Confidence 70 · logic · Fix: give the CLI calltimeout + margin, or a separate launch budget. -
[new] serve-25 runs
rocm serve --managedthrough the unboundedrun_rocmand has no EAI-7423 row —tests/e2e-cucumber/tests/e2e/serving_steps.rs:676-689,tests/e2e-cucumber/features/model_serving.feature:326
This change addsrun_rocm_boundedprecisely because the unbounded call "stalls the whole lane with no report". serve-25 is the same Linux managed Lemonade serve as the rows that carry EAI-7423, but has no row.
Confidence 70 · logic · Fix: userun_rocm_bounded, and add a row or say why the router path isn't affected. -
[new]
write_dashboard_configoverwritesconfig.jsonand drops the runtime that the previous step activated —tests/e2e-cucumber/tests/e2e/dash_steps.rs:92-101
This is harmless today only because Lemonade skips the active-runtime check (serve_cmd.rs:722-730).
Confidence 70 · mechanical · Fix: read the file, set onlydashboard.daemon.*, and write it back. -
[new] No gating scenario asserts the new Linux direct route for simple registered names —
engines/lemonade/src/lib.rs:512-530;tests/e2e-cucumber/features/model_serving.feature:61-70
The only scenario servingQwen3-0.6B-GGUFis covered by a flaky row on Linux and checks inference only. If the route were reverted to the router, every scenario would stay green.
Confidence 70 · logic · Fix: assertload_response.methodon a gating lane, or name the unit test covering the route in the PR text. -
[new] The README says unrecognized catalog shapes use the router, but on Linux a malformed registered checkpoint fails the serve —
README.md(~681);engines/lemonade/src/direct_llama.rs:46-47,:359-385
resolve_direct_llama_checkpointruns beforedirect_serve_binaryand only whenis_linux || explicit_checkpoint. On Linux the malformed entry fails even if no direct binary exists, while on Windows the same entry goes to the router. The comment "must not escape validation via routing" holds only on Linux. The code governs, because it is what runs.
Confidence 70 · logic · Fix: send a malformed registered entry to the router unless the user gave an explicitowner/reporeference, or document the error. -
[new] A sharded GGUF variant in a registered entry goes direct, downloads every shard, then fails as ambiguous —
engines/lemonade/src/direct_llama.rs:97-127,:508-540
collect_gguf_filesrecurses into the shard folder. The substring filter then matches every-0000N-of-0000Mfile and bails "matches multiple files". This contradicts the new docs line that multi-file entries stay with the router. It was not a working path before either. Whether the pinned catalog actually contains such an entry was not checked.
Confidence 70 · logic · Fix: when every match shares one shard stem, pick shard 00001, or route split matches to the router before downloading. -
[new] A Lemonade router serving on Vulkan never reaches the CLI's readiness check, and more Linux serves now take the router —
apps/rocm/src/main.rs:19055,:19113-19120
The CLI accepts a router model only when its backend starts withrocm, while the engine side acceptsvulkan(router_serve.rstests recordbackend_requested: "vulkan"). The check existed before this change. What is new is that complex entries on Linux are now routed through the router, so a Vulkan host serving one waits forever.
Confidence 70 · logic · Fix: accept the backend the engine recorded, and add a Vulkan router case to the readiness tests. -
[new] No GPU-lane check ran for this head — CI rollup
Every hosted check passes. No self-hosted GPU-lane checks appear, so dash-25, serve-08 and nightly serve-25 have no result for 7778139.
Confidence 70 · mechanical · Fix: run the required GPU lanes before merging.
Decisions for the author
- This PR makes two logical changes — non-blocking-improvement
The Lemonade serve-route rework (registered-GGUF resolution, router extraction, removal of the direct fallback) and the dash discovery and ownership rework share almost no code. Together they come to about 5,300 lines. AGENTS §11 asks for one logical change per PR; splitting would let each be gated and reverted on its own. - The Linux router-to-direct fallback was removed — tradeoff
The description already discloses this. Router failures now surface as errors instead of being masked, at the cost of a first-serve failure on hosts where a direct server would have worked. Confirm this is the intended choice for Vulkan and WSL hosts, given the readiness gap above.
Positive signals
- Moving the router path into
router_serve.rsbehind aRouterStepstrait makes the failure paths testable without a real server.router_failures_return_error_stop_server_and_mark_state_failedwould fail on the old code. - Lifecycle ownership comes from a positive
managedflag from the registry rather than id or engine string markers. The approval tests cover a stale target on both the key and draw paths. - The e2e protocol fixture records every request that is not a health or stats GET, which turns "never touches external servers" into something the scenarios actually assert.
Deployment notes
rocm dashnow probes local Lemonade (default port 13305) by default on Linux and WSL; it can be turned off in config.- The Services overlay now lists only registry-managed services. Docker-discovered containers and external Lemonade servers no longer appear there.
- A newer TUI attached to an older daemon gets no
managedfield, so its Services overlay is empty (by design: missing ownership is treated as unmanaged).
What this covered
Every file in the change was read completely (37 files), plus callers and callees in apps/rocm, apps/rocmd, rocm-core, the vLLM engine and the e2e harness. Diff prw-base…77781396 (merge base 2844d79).
What did not run, and why:
- No build or test ran. The host's free disk fell below the build floor during the review, so builds were stopped. Every finding is from reading, none is scored 100, and all mutation checks — including the ones behind the standing-objection verdicts above — are unverified.
telemetry_contract.rslines outside the diff were not re-read.- The pinned Lemonade 11.5.1
server_models.jsonis not on this host, so the sharded-catalog question is unverified. - The prior-changes pass was limited to inline comments on about seven earlier merged PRs.
- GPU and Windows execution is not possible here.
- No prompt-injection content was found.
The review ran fanned out across six independent workers: two for the engine, one for dash collectors/daemon, one for the TUI, one for e2e and docs, and one for the instruction, history, prior-change and comment passes. The findings were reached without the PR's discussion; the standing objections above were then settled against the author's reply of 10:53Z.
7778139 to
11b6452
Compare
|
Thanks for round 4. I pushed Blocking
Non-blocking
Not changed: the provenance enum, the probe in the tick loop, and the PR split. Verification on the rebased tree: the full workspace suite passed serially (33 targets, 3,897 tests), and so did both clippy commands, fmt, crate edges, smoke, and the external protocol scenarios. GPU scenarios, the Windows build, and |
Addressed in 11b6452. Details are in the reply comment.
788a1a3 to
5e0a33d
Compare
|
🔴 Automated review · pr-review-watcher · 788a1a3 This automation never files a GitHub approval, so no approving review will
Review — needs workFull review of the whole change (two commits, 40 files, +6629/−741). Blocking
Standing objections from 7778139 this round did not re-findThis round's blind review did not re-find these, and they were not settled this round. The next round settles them:
Non-blocking
Decisions for the author
Positive signals
Deployment notes
What this coveredEvery file in
What did not run, and why:
Not reconciled against any PR discussion here. The driver read the author's reply of 09:19Z. The standing objections that reply addresses and this round did not re-find are listed above as unsettled. |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 5e0a33d
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
No ticket named. The description matches the change closely. It discloses the removed Linux router-to-direct fallback, that the Services overlay now hides Docker containers, and the scenario gaps it knows about.
Round 6. This round reviewed 5e0a33d in full and blind, because the previous review point, 788a1a3, is no longer in the branch. Each finding is marked [new] or [standing]. Standing objections the blind pass did not re-find were each put to the reviewer afterwards. That check is anchored: the reviewer had been told the objection. Each is listed below with its outcome.
Blocking
- [standing since 788a1a3, re-found blind] An idle Lemonade's last-request rate stays "Fresh" forever, against the contract the PR's own test file states —
crates/rocm-dash-daemon/src/runner.rs:932-941,crates/rocm-dash-daemon/tests/telemetry_contract.rs:910-912,crates/rocm-dash-collectors/src/lemonade.rs:117-122/api/v1/statstokens_per_secondis the rate of the most recent request. The PR's own doc onparse_statssays it "stays the same until another request completes".- The runner calls
observe_direct(rate, cycle_at)on every stats scrape, andobserve_directre-stampslast_observed_atwithout checking whether the body changed. So an idle server's old rate shows as Fresh on every cycle and never reaches the validity-window expiry. - The doc comment added in 5e0a33d says the opposite: "An unchanged stats body is the same request, so it must not keep the rate fresh: once the validity window passes the rate becomes unavailable". No code does this. The test the comment sits on (
lemonade_managed_direct_llama_server_row_has_no_standing_warning) tests something else. - The comment governs: it is the only statement of intent, and it agrees with
parse_stats' own description of the value. - The base branch had the same weakness for external discovery, but discovery was off by default there. This PR turns it on by default and scrapes managed Lemonade services for the first time, which widens the blast radius.
- dash-27's
external_single_model_ratestep asserts "38" tok/s from a fixture that serves no requests. - Confidence 85 · logic · Fix: call
observe_directonly when the stats body changes (output tokens or decode times differ), and add the test that comment describes. Or remove the comment and label the value "last request".
- [standing since 7778139, re-found blind] The new remediation advice is pinned as wording only, and its own comment says no test proves it —
engines/lemonade/src/direct_llama.rs:67-75,:712-733- The error tells the user to run
rocm engines install lemonade --reinstallto restore a missing llama-server binary. - The test comment says: "This pins the wording only … NO test proves that the llama-server binary is back afterwards: that needs a real backend download, which no scenario asserts."
- AGENTS.md §3 requires remediation advice to be asserted together with the command actually clearing the condition. A wording-only pin is allowed only when it names a test that proves the behaviour.
- The comment's reason does not hold.
@id:runtime-lemonade-backend-alignment-reported(runtime-16,@nightly @requires-gpu) already runs that exact reinstall with a real backend download. One addedThenthere would prove the binary is present afterwards. - Confidence 80 · mechanical · Fix: add a step to runtime-16 asserting that the selected backend's llama-server exists after the reinstall (or that a direct serve then starts). Point the pin's comment at that step.
- The error tells the user to run
- [standing since 7778139, not re-found blind; confirmed in full by the follow-up check] The direct llama-server route leaves a running server behind when a step after spawn fails, while a doc comment says it doesn't —
engines/lemonade/src/process.rs(serve_direct_llama_server, from L504),engines/lemonade/src/lib.rs:547-548- After
command.spawn(), three?returns exit without stopping the child:write_running_state(..., Some(child.id()), "running")?,merge_json_state(&request.state_path, …"ready"…)?andchild.wait().context(…)?. Only the readiness/smoke closure's error arm callsterminate_pidandchild.wait(). std::process::Childdoes not kill the process when dropped.serve_http_resolvingthen marks the statefailed, so neitherstopnorhealthcan reach the server that is still running.- The doc comment on
serve_http_resolvingsays "Servers spawned insideserve_with_contextare stopped by the code that spawned them". That is false for these three sites. The PR description says the same for "the router and the direct route". The author's reply on 7778139 conceded the gap, and the code has not changed since. - The router twin got this fix (
router_serve.rsStateWrite/ReadyMerge/WaitServer tests). No test makes a post-spawn step fail on the direct route. - Confidence 85 · logic · Fix: put one terminate-and-reap guard around every post-spawn step, as
serve_routerdoes. Ifwait()itself fails, still attemptterminate_pid. Add a failure-matrix test for the direct route.
- After
- [standing since 788a1a3, not re-found blind; confirmed by a run in the follow-up check] The suggested fix in the "matches multiple files" error does not resolve for split GGUF sets —
engines/lemonade/src/direct_llama.rs(ambiguity error ~L117-135,first_shard_of_complete_set,is_incomplete_shard_set)- Setup: the existing
split_gguf_variant_resolves_to_its_first_shardfixture, which has a complete two-shard set plusother-Q4_K_M.gguf. The error suggests…:model-Q4_K_M-00001-of-00002.gguf. Fed back throughdirect_llama_model_path, that suggestion returnsOk(None). - The reason: an exact-filename variant matches only that one shard, so the set reads as incomplete.
- The serve would then pull and fail with "did not download". That part was read from the code, not run.
serving_variant_errors_carry_concrete_examplesonly covers non-split files. AGENTS.md §3 requires that remediation advice actually clears the condition that printed it.- Confidence 100 · logic · Fix: when the matches include a split set, suggest its quantization stem instead. Or let an exact shard filename resolve to its whole set. Add a split case to the test.
- Setup: the existing
- [standing since 788a1a3, not re-found blind; confirmed by runs in the follow-up check] A shard set can be assembled across cache roots or snapshot revisions —
engines/lemonade/src/direct_llama.rs(dedup ~L484-497,collect_hf_checkpoint_ggufs~L459-468,first_shard_of_complete_set~L520)- Matches are deduplicated by file name only. Shard 1 in root
h1plus shard 2 in rooth2counted as complete and returned theh1shard. The same happened with shard 1 undersnapshots/revAand shard 2 undersnapshots/revB. - llama-server looks for the remaining shards next to shard 1, so the launch fails after spawn. On the direct route it then also hits the cleanup gap above.
- Deleting the
(1..=total)index-range check leaves the engine suite green (166 passed), so the{00001, 00003}-of-00002case has no test. - Confidence 85 · logic · Fix: count a set as complete only when every shard shares the first shard's parent directory. Add the cross-root and out-of-range cases as tests.
- Matches are deduplicated by file name only. Shard 1 in root
- [standing since 788a1a3, not re-found blind; preview half confirmed, overlay half dropped] The Serving preview says nothing is running while external models are serving, and a new test pins that —
crates/rocm-dash-tui/src/ui/tabs/pane.rs:463-477, testservices_preview_ignores_external_only_instances(L706-713)- The preview now counts only managed, serving instances (
is_listed_service(i) && i.status.is_serving(); base counted every serving instance). The label was not changed and still reads"Running now — {n}"/"none running". - With two external models Ready, the Serving tab says "Running now — 0 / none running", while the Engines pane in the same tab says "Lemonade — externally running". The new test asserts exactly that text. AGENTS.md §3: "a pin over a false claim holds the claim in place."
- The overlay's "No managed services." empty state is correctly scoped and is proven by dash-26. That half of the earlier objection is withdrawn.
- Confidence 85 · logic · Fix: relabel the preview as managed (e.g. "Managed running — N" / "no managed services running") and update the pinned strings.
- The preview now counts only managed, serving instances (
- [standing since 788a1a3, not re-found blind; confirmed by the follow-up check] The new manual-testing text describes
--managedbehaviour the CLI no longer has —docs/manual-testing.md:278- The doc says: "Without it,
rocm servestays attached to the log stream and the stop command above is not reachable." Butapps/rocm/src/serve_cmd.rs:542readslet _ = managed; // background is now the default; --managed is accepted as an explicit synonym., and only--foreground/--verboseattach (bothconflicts_with = "managed"). - AGENTS.md §5 requires docs to match the code path.
- Confidence 95 · mechanical · Fix: delete that sentence, or say that
--managedis the default.
- The doc says: "Without it,
- [standing since 788a1a3, not re-found blind; confirmed by the follow-up check]
SystemInfo.lemond_versionis now dead —crates/rocm-dash-core/src/metrics.rs:59,demo.rs:557- The PR removed its only reader. Its only writer is the demo, which sets it to
None. - Removing it is wire-safe: there is no
deny_unknown_fields, and a missingOptiondeserializes asNone. - Confidence 85 · mechanical · Fix: remove the field and its demo initializer. The siblings
llama_server_buildandccr_versionwere already dead before this PR and can go in a separate change.
- The PR removed its only reader. Its only writer is the demo, which sets it to
Non-blocking
- [new] First contact with the external endpoint swallows every error, not just "nobody is listening" —
crates/rocm-dash-daemon/src/runner.rs:879-880lemonade_discovery_warn = (!endpoint.rows.is_empty()).then(...)stays quiet on 401, 403, 5xx and body errors whenever the endpoint has never been seen. So a Lemonade server that is present but needs auth, or is broken, produces no warning.- docs/testing.md promises only that "a never-seen absent endpoint remains quiet".
- Confidence 80 · logic · Fix: stay quiet only on connection-refused or transport errors.
- [standing, re-found blind] dash-25's expectations row makes the whole journey advisory on every lane that can run it —
tests/e2e-cucumber/expectations.toml:103-111@requires-engine:lemonadeforceseffective_engine = "lemonade", and the scenario is@requires-os:linux. So the row{effective_engine="lemonade", os="linux"}withflaky = truematches every eligible lane.- As a result, the dashboard, instance-detail and post-quit inference checks never gate a merge. The description concedes this only for the installed wording.
- Confidence 80 · logic · Fix: keep only the serve step under the flaky scope, or say plainly that dash-25 is advisory everywhere.
- [new] No scenario covers the positive half of the new ownership rule —
crates/rocm-dash-tui/src/ui/services_manager.rs:95-105,tests/e2e-cucumber/features/dash.feature- dash-26 proves that external rows are excluded from the Services overlay.
- No scenario opens Services with a live managed service and sees it listed and stoppable. server-records-02 covers only non-running records.
- Unit and contract tests do catch a dropped
managedflag (mutation run). - Confidence 75 · mechanical · Fix: extend dash-07 to open the overlay, assert the managed row, and stage a stop approval.
- [new] Serving behaviour the CLI now chooses differently is covered only by unit tests —
apps/rocm/src/main.rs:15578-15598,engines/lemonade/src/direct_llama.rs(serving_route)- Two choices changed: Vulkan-backed router models now read as ready, and simple registered names now take the direct route on Linux.
- AGENTS.md §3 treats which runtime the CLI selects as user-observable.
serve-lemonade-inferenceexercises the direct route, but it neither asserts the route nor gates on Linux. - Confidence 70 · mechanical · Fix: assert the route (
load_response.method) in serve-lemonade-inference, and name or add the lane that covers router+Vulkan readiness.
- [new] serve-25 cannot tell old behaviour from new on Windows —
tests/e2e-cucumber/features/model_serving.feature:338- Registered names took the router on Windows both before and after this change.
- Confidence 75 · logic · Fix: tag the scenario
@requires-os:linux, or say that its Windows run only re-confirms existing behaviour.
- [new]
run_rocm_boundedwas applied to only some of its twin serve call sites —tests/e2e-cucumber/tests/e2e/serving_steps.rs:557,:928,:982setup_gpu_modeland twoserve --managedcalls still use the unboundedrun_rocm.- Confidence 75 · mechanical · Fix: use
run_rocm_bounded(..., serve_cli_budget(...))at those sites as well.
- [new] The isolated dash scenarios now probe a real
127.0.0.1:13305by default —crates/rocm-core/src/lib.rs(default_dashboard_enable_lemonade),tests/e2e-cucumber/tests/e2e/dash_steps.rs- Only dash-25 turns discovery off.
- dash-03/04/07/08/09 and the replay/GPU scenarios do not. On a self-hosted lane with Lemonade on 13305 they gain extra rows and warnings.
- Confidence 70 · mechanical · Fix: default the isolated e2e config to
enable_lemonade: false.
- [standing, re-found blind] The IPv6 contract tests return early and pass silently when IPv6 is unavailable —
crates/rocm-dash-daemon/tests/telemetry_contract.rs(lemonade_dual_stack_managed_bind_does_not_duplicate_external_row,…ipv6_record_replaced_by_ipv4_record…)- Confidence 75 · mechanical · Fix: print a skip, or mark the tests ignored when the bind fails.
- [standing, re-found blind]
Instance::is_external_lemonadekeys oncontainer_name == "lemonade"—crates/rocm-dash-core/src/metrics.rs:230-231,crates/rocm-dash-tui/src/ui/tabs/pane.rs:394-397- An unmanaged Docker container named
lemonademakes the Engines pane say "externally running". The effect is display-only. - Confidence 70 · logic · Fix: carry an explicit provenance or engine field.
- An unmanaged Docker container named
- [standing, re-found blind] No self-hosted GPU-lane check has reported on this head. The status rollup shows 23 hosted checks, all green, with Skill checks skipped, and no GPU lane. dash-25, serve-08 and serve-25 have not run anywhere for 5e0a33d. This is missing evidence, not a defect. Confidence 85 · mechanical.
Standing objections settled this round without being kept
These checks are anchored: each objection was put to the reviewer after its blind pass ended.
- Merge conflict with
main: resolved. The API reportsmergeable: truefor 5e0a33d. - Empty-state advice "check an external endpoint" points at a Serving action that does not exist: not kept. The Engines action in Serving does show live external availability. The
dash_characterizationcomment is accurate, because dash-27 asserts the empty message and that "externally running" clears. What remains is that nothing in the TUI names the probed host/port or thedashboard.daemon.lemonade_*keys. That is an observation. The next round re-readsinstances.rsblind. - Pinned CLI strings with no pointer to a behaviour test (help text, startup status note): does not hold. Both now point at tests that prove the behaviour: dash-26, and
managed_install_verdict_is_not_changed_by_live_events. The reinstall-advice part is kept, as the second Blocking item above. - Router and pull steps tested only through fakes: does not hold. Real-step tests against a loopback fixture and a fake executable exist and catch the alias/router-name mutations. As the author notes, "a router failure never spawns a direct server" is still guaranteed only by how the code is structured.
- The alias test and the managed-row e2e check could not fail: does not hold. Changing
--aliasfailed 2 of the 9 alias tests (run). The instance-detail matcher rejects a wrong status, port or model in its unit tests. dash-25 being advisory on every lane is the separate Non-blocking item above.
Decisions for the author
- Linux direct serving does not apply the router's per-model launch defaults — tradeoff
- Direct serving passes
-m,--n-gpu-layers 999,--aliasand the recipe flags. A simple catalog entry can therefore run with different settings, such as context size, on Linux than the same name gets through the router on Windows. In return, direct serving gives an exact alias and avoids the router. - One comment in
serving_routeon why Windows stays on the router would let a reader confirm the split is deliberate.
- Direct serving passes
- Direct-served models still get a
/v1/unloadon stop — tradeoffllama-serverhas no such route, soengine_stopreports an error, and the process is then stopped by PID.- The behaviour is pre-existing, but this PR makes direct serving reachable for registered names.
- The PR bundles three changes — non-blocking-improvement. The three are the serving-route and identity rework, default-on external discovery in the dashboard, and the Services ownership model. Each would review and revert more safely on its own.
Positive signals
- Ownership is a positive
managedflag that defaults to false (#[serde(default)]), so legacy snapshots fail safe: controls are hidden rather than wrongly offered. A staged approval whose target vanishes or turns external is dropped both on keypress and at draw time. - The external-protocol fixture records every request that is not a GET of health or stats, and dash-26 asserts the log is empty after the stop/restart keys and again after quit. That is the proof of behaviour AGENTS.md asks for.
router_serve.rshides the router steps behind a trait, so the failure-matrix tests check stop-then-failedstate for every step.
Deployment notes
rocm dashon Linux/WSL now polls127.0.0.1:13305on every discovery tick by default. To opt out, setdashboard.daemon.enable_lemonade: false.- On Linux, a failed router load no longer falls back to a direct llama-server, so affected hosts get an error on the first serve.
- The Services overlay no longer lists Docker-discovered containers.
- Version skew: a new
rocm dashthat attaches to an older dash's still-running embedded daemon gets snapshots with nomanagedfield. It then shows "No managed services." until the old dash exits.
What this covered
- Read: every file in
21be0034…5e0a33d9(prw-base...HEAD, 40 files), plus callers and siblings:backend_alignment.rs,install.rs,observation.rs, the readiness and stop paths inmain.rs, andlocal_server_records.feature. - Fan-out: two coupled-group workers (Lemonade engine/serving; dashboard/telemetry/docs), then one design worker.
- The engine worker split off a read-only sub-worker for the e2e files and the
main.rs/rocm-corehunks. That sub-worker read only the serving-related sections ofe2e.rs. - The dashboard worker read
event_loop.rs,app/mod.rs,app/types.rsandinstances.rsselectively, because their changes are one-line field plumbing. - The history and prior-changes passes were run by the consolidating context and were not independent.
- The engine worker split off a read-only sub-worker for the e2e files and the
- Runs:
cargo test -p rocm-engine-lemonade --lib: 167 passed, and mutations M1–M7 were all caught.- The
telemetry_contractLemonade tests: 14 passed, and every mutation was caught except one equivalent mutant. - The
services_managerandpanemutations were all caught. cargo xtask e2e -- -n 'External Lemonade protocol': 2 scenarios, 31 steps passed.- The follow-up objection checks ran the split-shard, cross-root, index-range and alias probes.
- Did not run: any GPU scenario (dash-25, serve-08, serve-25), because there is no GPU here; Windows
cfgarms (CI'swindows-build-and-testpassed); mutations for the e2e step files; local clippy (CI's passed). - No prompt-injection content was found.
- The findings were not reconciled against the PR discussion. Only the standing objections above were weighed against the author's replies.
5e0a33d to
aa28578
Compare
|
Thanks for round 6. I pushed Blocking
Non-blocking
dash-25 had a real bug that the flaky row hid. On the GPU lane my instance-detail check failed on the real screen, because a panel border touches the Verification: the full workspace suite passed serially (33 targets, 3,993 tests), and so did both clippy commands, fmt, crate edges, smoke, and the external protocol scenarios. After the last two rebases onto |
aa28578 to
dd0ac46
Compare
|
🔴 Automated review · pr-review-watcher · aa28578 This automation never files a GitHub approval, so no approving review will
Review — needs workFull review of the whole change.
Blocking
Non-blocking
Standing objections settled this round without being keptThese checks are anchored: each objection was put to the reviewer, with the author's reply, after the blind pass ended.
Decisions for the author
Positive signals
Deployment notes
What this covered
|
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · dd0ac46
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
No ticket named. The description is detailed and accurate except for the reinstall-proof claim (the fourth Blocking item).
Round 8 (at dd0ac46). This round reviewed dd0ac46 in full and blind, because the previous review point, aa28578, is no longer in the branch after the rebase. Each finding is marked [new] or [standing]. The five objections carried from round 7 (posted as an informational comment at aa28578) were not re-found blind, so each was put to the reviewer afterwards, one at a time, with the author's reply where there was one. That check is anchored: the reviewer had been told the objection. Four still hold and are appended under Blocking; one no longer holds and is listed at the end of Non-blocking. This change request replaces the round-6 one as this account's complete position.
The fourth new Blocking item (reinstall "proof") is the residual round 7 rated non-blocking. It is reported as the method found it, but it does not on its own decide this change request; the other blocking items do.
Blocking
-
[new] The new router serve path's real stop and its failed-exit branch are never tested —
engines/lemonade/src/router_serve.rs:121-133,engines/lemonade/src/router_serve.rs:220-225,engines/lemonade/src/process.rs:686-698
I made two changes together: turnedLemondRouterSteps::stop_serverinto a no-op, and made a non-zerowait_serverstatus returnOk(()).cargo test -p rocm-engine-lemonade --libstill passed all 177 tests.FakeSteps::wait_serveralways succeeds, and no test spawns a reallemondand then stops it. The architecture doc and the PR say this new module stops the server and marks state failed on any error. Neither claim is tested.
Confidence 100 · logic · Fix: add a failing-exit variant toFakeStepsthat asserts both the error and statefailed. Add a Linux test that spawns the fakelemond(exec sleep 30), calls the realstop_server, and asserts the process is gone. -
[new] New test helpers write fake executables with
std::fs::writeinstead of the sharedETXTBSY-safe helper —engines/lemonade/src/test_http.rs:112-116(write_executable), plus thefake_pull_manifest,fake_llama_serverand inline-script helpers inengines/lemonade/src/process.rs(thefrom_mode(0o700)sites the diff adds)
crates/rocm-core/src/fake_executable.rsexists to prevent this race. Writing a script in-process and then running it while other test threads fork fails intermittently withText file busy (os error 26).engines/lemonade/src/install.rsin this same crate already uses that helper. One worker reported 5 such failures in 70 serial runs ofcargo test -p rocm-engine-lemonade --lib, but its logs did not contain the error. I got 0 failures in 25 serial and 45 parallel runs, so I am relying on the in-repo explanation of the race, not my own reproduction. CI's per-process nextest would hide it. The localcargo test --workspacegate in AGENTS.md §8 would not. This also breaks AGENTS.md §6 (reuse the shared component).
Confidence 85 · mechanical · Fix: callrocm_core::fake_executable::write_fake_executablefrom these helpers. -
[new] Resetting telemetry when only
canonical_model_idchanges is never tested —crates/rocm-dash-daemon/src/runner.rs:650
The worker replacedold.canonical_model_id == new.canonical_model_idwithtrue, and all Lemonade telemetry contract tests stayed green. Themodel_refand host comparisons next to it are each caught by a test, so this is the one part of the new rewrite detection that nothing tests.
Confidence 100 · logic · Fix: add a case toassert_record_rewrite_resets_attributed_ratethat rewrites onlycanonical_model_id. -
[new] The reinstall advice is said to be "proved" by a step that never creates the condition the advice is for —
tests/e2e-cucumber/tests/e2e/runtime_steps.rs:532-575,tests/e2e-cucumber/features/runtime_setup.feature:337-339,docs/testing.md(theruntime-lemonade-backend-alignment-reportedparagraph),engines/lemonade/src/direct_llama.rs:775-785, PR description ("Local limits")
The serve error atdirect_llama.rs:74-79says to runrocm engines install lemonade --reinstallwhen the selected backend'sllama-serveris missing. Runtime-16 runs--reinstallon a runtime it never damaged, then checks that the binary exists. So it proves that an install leaves a binary, not that--reinstallrestores a missing one. AGENTS.md §3 requires the named command to "actually clear the condition that printed it", with the message and the resulting state asserted together. The step's doc comment,docs/testing.mdand the PR body all call this step "the proof", so the description overstates what the step covers.
Confidence 70 · logic · Fix: before the reinstall, delete<runtime_dir>/bin/llamacpp/<backend>/, ideally also check thatrocm serve <owner/repo:variant>prints the advice, then reinstall and assert the binary is back. Otherwise, weaken the claim in all four places. -
[standing since aa28578, not re-found blind; the follow-up check found it still holds] A managed Lemonade service with an API key shows a permanent "lemonade health … 401" warning and never shows throughput —
crates/rocm-dash-daemon/src/runner.rs(managed branch, ~L817-822 and ~L886-888),crates/rocm-dash-collectors/src/lemonade.rs(fetch_health_if_router, ~L335)engines/lemonade/src/process.rs:263-270setsLEMONADE_API_KEYwheneverrocm serveprotects the endpoint; its comment says Lemonade then gates/api,/v0and/v1on it.- Nothing in the collector or daemon sends credentials. A 401/403 from the health check is kept as an error on purpose (and tested as such), so the managed branch pushes a "health" warning and skips the row on every cycle.
- At aa28578 a managed record against a 401 fixture produced that warning on every snapshot. At dd0ac46 this was re-checked by reading, not re-run. That real lemond answers 401 there rests on the engine's own comment.
- Confidence 85 · logic · Fix: present the CLI-owned key for registry targets, or treat 401/403 on a managed target as "throughput unavailable" with no standing warning. Add a regression test.
-
[standing since aa28578, not re-found blind; reproduced again by the follow-up check] A stale partial download in an older snapshot revision makes a complete split model permanently unservable —
engines/lemonade/src/direct_llama.rs(hf_checkpoint_gguf_matches_in_roots~L514-534,first_shard_of_complete_set,is_incomplete_shard_set,ensure_model_with_download)- Matches are still de-duplicated by file name across every snapshot revision in a root, in path order. So revision
aaa/…-00001-of-00002shadows thebbb/…-00001of a complete set; the kept shards span two directories, the set reads as incomplete, and resolution returnsOk(None). - A probe test at dd0ac46 (
aaaholds shard 1 only,bbbholds both) returnedNone, thenensure_model_with_downloadfailed with "Lemonade did not downloadreqfor direct serving". - The author's earlier note ("a clean error, not a launch that fails after spawn") is accurate that nothing spawns, but nothing the user is told clears it: the automatic re-pull finds the same cache (inferred, not run against a real pull), and the message points at the download rather than the stale revision. That note also describes one root shadowing another; here both revisions are in the same root. The existing
shards_spread_over_snapshot_revisions_are_not_a_complete_setdoes not cover this case. - Confidence 100 · logic · Fix: group matches per snapshot directory, and choose a complete set within one directory.
- Matches are still de-duplicated by file name across every snapshot revision in a root, in path order. So revision
-
[standing since aa28578, not re-found blind; the mutation survived again] The "previously seen external endpoint went away" warning has no test —
crates/rocm-dash-daemon/src/runner.rs(let absent = endpoint.rows.is_empty() && …is_connect_error(&error))- Dropping the
endpoint.rows.is_empty()half silences that warning, andcargo test -p rocm-dash-daemonstayed green (47 unit, 9end_to_end, 25telemetry_contract). lemonade_external_health_failure_warns_and_removes_rows_without_graceuses HTTP 503, which is not a connect error, soabsentis false either way. A seen endpoint that stops, the commonest way a server goes away, is never exercised.- The code comment promises the code "reports why they disappeared". AGENTS.md §3 requires a message to be asserted together with its behaviour.
- Confidence 100 (mutation run) · mechanical · Fix: seed rows, close the listener, and expect a warning naming the endpoint.
- Dropping the
-
[standing since 788a1a3, not re-found blind; the follow-up check found the remaining part still holds] An idle Lemonade's last-request rate is presented as a live, fresh rate —
crates/rocm-dash-core/src/observation.rs(observe_direct, ~L249),crates/rocm-dash-daemon/src/runner.rs(stats scrape),crates/rocm-dash-tui/src/ui/tabs/serving.rs:57,crates/rocm-dash-tui/src/ui/tabs/instances.rs:57observe_directre-stampslast_observed_aton every scrape, so the held legend can never fire for an idle Lemonade. Serving and Instances show the value as plain "tok/s". No "last request" qualifier appears in README.md, docs/testing.md or the TUI; only the collector's code comment and the PR description explain it.- The author's reply explains why the deeper fix (not re-stamping an unchanged report) was reverted, and that is credible. It is also true that external rows already behaved this way on
main. But this PR newly scrapes managed Lemonade services through the same path, so a model started withrocm servenow shows its last request's rate as current throughput, which it did not before. - The cheap remedy needs no new scenario.
- Confidence 80 (anchored check, read not run) · logic · Fix: label the figure where it is displayed (for example "last req tok/s"), or document what it means in the user-facing docs.
Non-blocking
-
[standing, re-found blind] A missing space joins two tags —
tests/e2e-cucumber/features/runtime_setup.feature:340(@id:runtime-lemonade-backend-alignment-reported@requires-gpu)
gherkin 0.16 still reads it as two tags, becausetag_charexcludes@and__allows zero separators. Butfeature_naming.rssplits on whitespace, so it reads the id asruntime-lemonade-backend-alignment-reported@requires-gpu. The guard and the harness now see different ids for this scenario.
Confidence 100 · mechanical · Fix: restore the space. -
[new; the test-name part standing] Some new comments say things the code does not do —
engines/lemonade/src/runtime_dir.rs:45-46,engines/lemonade/src/direct_llama.rs:145-147,crates/rocm-dash-collectors/src/lemonade.rs:640-653runtime_dir.rssays the only callers ofrooted_atare the Linux tests inprocess.rs. It is also called fromlib.rs:898,router_serve.rs:543anddirect_llama.rs:1482.- The isolation comment says tests read no real user directories, but
hf_cache_rootsstill readsHF_HOME,HUGGINGFACE_HUB_CACHEandHOME. discover_is_clean_noop_when_endpoint_absentsays it checks "an empty discovery result", but it asserts thatfetch_healthandfetch_statsreturn errors.
Confidence 85 · mechanical · Fix: reword the comments, and rename the test or change what it asserts.
-
[new] Three host normalizers in the registry disagree on case —
crates/rocm-dash-daemon/src/registry.rs:67,:105,:301-306
host_reachlowercases and strips brackets; the other two are case-sensitive. ALOCALHOSTrecord only groups correctly because the grouping step happens to lowercase.
Confidence 70 · mechanical · Fix: route all three through one normalizer. -
[new] The docs describe the serve bound imprecisely —
docs/testing.md(dash-25 paragraph) and the comment on the newexpectations.tomlrow
Both say the serve is bounded byserve_timeout_secs(90s). Readiness polling does use that, but the CLI process is only killed after 90 + 300 s (serve_cli_budget).
Confidence 70 · mechanical · Fix: mention the margin.
Standing objections settled this round without being kept
This check is anchored: the objection was put to the reviewer after the blind pass ended.
windows-build-and-testfailed onlemonade_never_seen_absent_external_endpoint_is_quiet: no longer holds.send_errornow maps connect errors and timeouts alike to "nothing listening", andclosed_and_unanswered_endpoints_both_read_as_nothing_listeningcovers both cases. The test is not cfg-gated, and thewindows-build-and-testlog for dd0ac46 shows it ran and passed.
The seven other round-6 objections were settled at aa28578 as no longer holding (see the round-7 comment); none was re-found blind here.
Decisions for the author
- The router→direct fallback after a failed load is removed — tradeoff
An error now stops the server and fails the serve every time, which is more honest. The cost: a host where the router load fails but a direct server would have started now gets an error on its first serve. The PR states this, so this is only a request to confirm it is deliberate. - Vulkan counts as GPU-ready — tradeoff
service_model_reports_gpu_backend(apps/rocm/src/main.rs) and direct serving now accept whichever GPU backend Lemonade selected.docs/manual-testing.mdpreviously required "does not fall back to CPU or Vulkan". CPU is still refused, so the no-CPU-fallback rule holds. But a ROCm-capable host whose Lemonade picked Vulkan will now serve through Vulkan with no notice. - The real-GPU dashboard journey can never block a merge — tradeoff
The newdash-lemonade-managed-inference-journeyrow inexpectations.tomlmatches every lane where the scenario can run, so every assertion in it is advisory. This is disclosed. One option is to split the dashboard and instance-detail checks into a scenario that uses a planted managed Lemonade record, so they gate without the flaky serve. - The PR combines two independent changes — non-blocking-improvement
It is about 7.6k lines. Lemonade catalog routing and serve-path rework is one change; dashboard external discovery and ownership is the other. Each commit mixes both. Splitting them would make each easier to review and revert.
Positive signals
- Lifecycle ownership is a positive
managedflag that only the registry sets, with serde defaulting it to false for old snapshots. The Services overlay re-checks that the approval target is still managed before acting on a "yes". Mutation testing showed each of those branches is caught. serving_routedecides the route before any download or spawn and refuses CPU and unknown backends. The shard-resolution edge cases are covered by mutation-verified tests.- The external-discovery scenarios use an in-process fixture that records any request other than a
GETof health or stats. That is a real behavioural check that the dashboard never sends a lifecycle request to a server it does not own. Both scenarios pass locally (2 scenarios, 31 steps).
Deployment notes
dashboard.daemon.enable_lemonadedefaults totrue(new config fieldsenable_lemonade,lemonade_host,lemonade_port). Every Linux/WSL dashboard now probes127.0.0.1:13305read-only on each discovery tick.- Behaviour change: a failed router load no longer falls back to a direct llama-server.
What this covered
Every file in the change, 12342b3ac7be… (prw-base, equal to the merge-base) … dd0ac46a7141…. The code was split across three independent workers: Lemonade engine plus rocm-core, dashboard telemetry plus TUI, and the e2e harness plus docs. Some parts of large files were read only as diff hunks plus their surroundings, not end to end:
runner.rslines 1000-1739 (snapshot assembly and existing tests);- the rest of
registry.rsand of thepane.rsproduction code; direct_llama.rslines 1300-1412 and 1810-2001 (test bodies);- parts of
process.rsandlib.rs; - the unchanged parts of
serve_log.rs,serving_steps.rsandruntime_steps.rs.
Tests run, all in the checkout's own target/ with RUSTC_WRAPPER= empty:
- lemonade lib: 177 tests, plus mutations;
- dash-core, dash-collectors, dash-daemon (including telemetry_contract) and dash-tui: all green except 3 sandbox-only
agent::clientsfailures unrelated to this PR; - e2e-cucumber lib (161) and feature_naming (4);
- the two external Lemonade protocol scenarios.
CI is all green, but no self-hosted GPU or Windows-GPU lane appears in the rollup for this head. That is missing evidence for dash-25, serve-08 and serve-25, not a defect.
Did not run:
- Agent-instruction adherence, history, and design passes: run by the driver in a single context, so not independent.
- Prior-changes pass: not run (time).
- Windows
cfgarms: checked by reading only, not compiled locally. CIwindows-build-and-testpassed. - GPU scenarios: not run (no GPU).
Not reconciled against any discussion here. The standing objections above were weighed against the author's reply after the blind pass, and that is the only part of the discussion that reached the reviewer.
dd0ac46 to
a133657
Compare
|
Addressed in a133657, rebased on current main.
On the decisions:
Not verified here: no GPU lane or cucumber scenario ran locally. CI covers those. |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · a133657
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Review — needs work
Full review of the whole change.
No ticket named; the description was read and compared with the change (see the description finding below).
Round 9 (at a133657). This round reviewed a133657 in full and blind, because the previous review point, dd0ac46, is no longer in the branch after the rebase. Each finding is marked [new] or [standing]. Of the round-8 objections, one was re-found blind (the
canonical_model_idclause: the added test does not catch its removal) and one came back as non-blocking (the reinstall advice). The other six were not re-found blind, so each was put to the reviewer afterwards, one at a time, with the author's reply (comment 6216331509). That check is anchored, because the reviewer had been told the objection. None of the six still holds as a blocker; they are listed at the end. This change request replaces the round-8 one as this account's complete position.Two Blocking items do not on their own decide this change request. The merge conflict blocks the merge on GitHub anyway. The description items appear only in the PR description, not in any commit message, so they do not land on main. The fallback-window state, the vacuous leak test, the
canonical_model_idclause and the missing route scenario do decide it.CI: the one failing check,
E2E tests (Strix Halo, Windows, current), fails scenario diagnose-38. Main's own run of that lane at 1f19198 fails it the same way, and this PR touches no diagnose or fix files, so the failure is not counted against this PR.
Blocking
-
[new] The PR no longer merges, because its new dash scenario numbers collide with ones added on main —
tests/e2e-cucumber/features/dash.feature:560-613- The PR adds
dash-35,dash-36anddash-37. Main has since added its owndash-35(dash-gpu-vram-uses-binary-units) anddash-36(dash-onboarding-adopt-whitespace-cwd-quotes-approval-preview). git merge-tree --write-tree prw-base HEADreportsCONFLICT (content): Merge conflict in tests/e2e-cucumber/features/dash.feature. This is the only conflicting file.- Confidence 100 · mechanical · Fix: rebase, then renumber the three new scenarios to
dash-37–dash-39. Main's numbers stay because they are already merged.
- The PR adds
-
[new] While the router-to-direct fallback runs, the engine state says
failed, and any reader in that window locks the service record atfailedfor good —engines/lemonade/src/router_serve.rs:127-131,engines/lemonade/src/direct_llama.rs:110-149,crates/rocm-core/src/service_record.rs:224-246serve_routerstops lemond and callsmark_json_status(.., "failed").serve_direct_after_router_failurethen runsensure_model_in, which can be a long download, and spawns llama-server. Nothing rewrites the state until that spawn succeeds.refresh_from_engine_statecopiesfailedinto the record. Its callers then persist it:load_managed_serviceandload_managed_servicesinmain.rs, andproviders.rs:529. That coversrocm services, chat and dash.- Once the record says
failed,refresh_from_engine_statereturns early for any status outsidestarting|running|recovering|ready. A fallback that later succeeds therefore stays recorded asfailed. - The fallback test (
lib.rs:1195) asserts only the final state. - Confidence 85 (path traced, not run) · logic · Fix: write
startingbefore the fallback starts its work. Add a test that asserts the state during the fallback.
-
[new] A test that claims to prove the direct server is never leaked stays green when nothing kills it —
engines/lemonade/src/process.rs:1009-1016,process.rs:1107-1164(direct_server_stops_when_state_cannot_be_written_after_spawn)- The fake server is
#!/bin/sh\nexec sleep 30. After theexec, the process's/proc/<pid>/cmdlineissleep 30, so the test's search for the script path can never match. - I removed
terminate_pidandchild.wait()fromstop_and_failinsupervise_direct_server. This test still passed, and asleep 30process was left running. - The sibling
direct_server_step_failure_stops_the_child_and_marks_state_faileddid fail under the same mutation, so only this test is vacuous. - Confidence 100 · logic · Fix: have the fake write
$$to a file beforeexec, and check that PID.
- The fake server is
-
[standing since dd0ac46, re-found blind] One clause of the runner's same-endpoint check can be deleted without any test failing —
crates/rocm-dash-daemon/src/runner.rs:655-656- With
&& old.canonical_model_id == new.canonical_model_idremoved, all 19telemetry_contractLemonade tests passed. lemonade_managed_canonical_model_rewrite_resets_attributed_ratepasses only because the rewritten name stops matching health, and the unmatched-row tracker removal clears the rate anyway.- Confidence 100 · mechanical · Fix: add a case where
canonical_model_idchanges while a name that still matches health remains.
- With
-
[new] No scenario covers the new route choice for registered names: simple ones are now served directly on Linux —
tests/e2e-cucumber/features/model_serving.feature(serve-07/serve-08),tests/e2e-cucumber/tests/e2e/serving_steps.rs:746- Which runtime a serve uses is user-observable. AGENTS.md §3 says such a change needs a Gherkin scenario, and that "a unit test asserting the internal helper does NOT discharge this".
- The only route assertion in
tests/is"lemonade-cli", in serve-26 (router route)."lemonade-packaged-llama-server"appears nowhere undertests/. - serve-07 stays green whichever route a simple registered name takes.
- The description admits the gap and points to unit tests, which is the substitution §3 rules out.
- Confidence 85 (cited rule, verified with grep) · logic · Fix: add a
@requires-os:linuxThen step that reads the engine state for a simple registered name and asserts the direct method and a non-CPU backend.
-
[new; the reinstall part standing] The description overstates two things
- The non-ROCm notice. The description says "When the selected backend is not ROCm … the serve output and log name that backend". The direct path writes the notice to the log (
process.rs:524-548). The router path only callseprintln!(router_serve.rs:87-89). A managed serve runs underspawn_serve_http_background, whose stdout and stderr go toStdio::null()(state.rs:141-143), so a managed Vulkan router serve shows the notice nowhere.a_non_rocm_backend_is_named_in_the_notice_and_matches_the_statetests the helper, not the emit. - The reinstall advice. The description says the advice "is proved by a new last step of
runtime-lemonade-backend-alignment-reported". Later it says the step "does not prove that the command repairs a deleted one". The second statement is the accurate one. - It also names scenarios by stale numbers ("dash-25", "the nightly serve-25"); the PR's scenarios are dash-35 and serve-26.
- Confidence 85 · logic · Fix: for the notice, either share one emit-and-log helper between the two routes and assert it from the log, or narrow the description. For the rest, correct the wording.
- The non-ROCm notice. The description says "When the selected backend is not ROCm … the serve output and log name that backend". The direct path writes the notice to the log (
Non-blocking
-
[standing, re-found blind] The new reinstall advice is not shown to fix the condition it reports —
engines/lemonade/src/direct_llama.rs(the error added by this PR),tests/e2e-cucumber/tests/e2e/runtime_steps.rs(assert_lemonade_backend_server_installed)- The error added by this PR tells the user to run
rocm engines install lemonade --reinstall. AGENTS.md §3 requires that a named command actually clears the condition. - The step never deletes the binary first, so it passes even if the reinstall does nothing. It also runs only on Windows nightly, while the advice is printed on Linux too.
- The PR states this gap openly.
- Confidence 70 · logic · Fix: delete the selected backend's
llama-server, reinstall, then assert it is back.
- The error added by this PR tells the user to run
-
[new] Comments this change made stale
router_serve.rs:91-92says a load failure is preserved "rather than masking it with direct serving". The next call tags itRouterLoadFailure, which is what turns on direct fallback.crates/rocm-dash-tui/src/ui/mod.rs:258-260still refers to the "Running now" listing, which this PR renamed to "Managed running".ui/mod.rs:353-357saysservice_rowscan include an unmanaged Docker/Lemonade entry. It no longer can.- In each case the code governs, because the comment describes behaviour the code no longer has.
- Confidence 85 · mechanical.
-
[new] The
last req tok/slabel appears only in the Instances tab —crates/rocm-dash-tui/src/ui/tabs/pane.rs(OpenServicesgenspan),ui/services_manager.rs(gen_tps_compact)- Managed Lemonade rows on these surfaces show a stale last-request rate as plain
genthroughput. Home and Observe also add it into their totals. - The README scopes its claim to the Instances table and detail, so this is an inconsistency, not a false claim.
- Confidence 70 · logic.
- Managed Lemonade rows on these surfaces show a stale last-request rate as plain
-
[new] External discovery can flap and stall the daemon loop
- One failed or slow
/healthcall removes every external row with no grace period. That behaviour is pinned bylemonade_external_health_failure_warns_and_removes_rows_without_grace. parallel_scrapeis awaited inline in the tick loop (runner.rs:813-886). Up to three sequential 1.5 s requests can delay GPU ticks.- Both matter more now that discovery is on by default against
127.0.0.1:13305. - Confidence 70 · architectural · Fix: put an overall timeout on each endpoint and allow one missed discovery.
- One failed or slow
-
[standing] The PR does several separable things in about 8,400 lines. These are:
- Lemonade catalog routing;
- the router-to-direct fallback;
- dash external discovery;
- Services-overlay ownership;
- rate labelling.
AGENTS.md §11 asks for one logical change per PR. Splitting engine routing and fallback from the dash work would make each part reviewable.
Standing objections settled this round without being kept
Each was put to the reviewer after the blind pass ended, one at a time, with the author's reply. These checks are anchored, because the reviewer had been told the objection.
- A managed Lemonade service with an API key shows a permanent 401 warning: no longer holds. A 401 or 403 on a managed health check now maps to "no telemetry" with no warning. External discovery still errors on these statuses.
lemonade_protected_managed_endpoint_keeps_registry_status_without_telemetry_warningpasses, and fails with that branch disabled. - A stale partial download in an older revision makes a split model unservable: no longer holds. Complete sets within one snapshot directory now sort first.
complete_snapshot_is_not_shadowed_by_an_older_partial_downloadpasses, and fails with that ordering disabled. - The "previously seen external endpoint went away" warning has no test: no longer holds. With the round-8 mutation re-applied,
lemonade_seen_external_endpoint_closing_warns_and_removes_rowsfails in 3 of 3 runs. - The router's real stop and failed-exit paths are untested: no longer holds. Re-applying both mutations fails
nonzero_server_exit_returns_error_and_marks_state_failedandreal_router_stop_kills_and_reaps_its_server. The second checks the PID it spawned, so it does not share theexec sleepweakness above. - New fake-executable helpers skip the shared
ETXTBSY-safe helper: no longer holds, by reading only. Every fake executable this PR adds goes throughwrite_fake_executable. - An idle Lemonade's last-request rate is shown as a live rate: no longer holds as a blocker. README,
docs/testing.md,docs/manual-testing.mdand the Instances table and detail now label it. The Serving and Services surfaces still show plaingen, which is the non-blocking finding above.
Decisions for the author
- External Lemonade discovery is on by default — tradeoff
- For: zero-config visibility of a local Lemonade server.
- Against: every Linux/WSL dashboard now probes a third-party port. Any other service there that answers 404, 401 or 503 produces a standing warning, and the probe adds tick latency.
- The other option is off by default, with the dashboard telling the user how to turn it on.
- Registered aliases on Linux now go direct, launched with basic flags only — tradeoff
- Simple catalog entries no longer receive the router's per-model launch defaults.
- Restricting the route to "simple" entries limits this, but confirm that losing those defaults is acceptable for the default model.
Positive signals
serve_routerwraps every post-spawn step in one fallible block behind theRouterStepstrait, so any failure stops and reaps lemond. The old path could leak it.- Fallback is limited by a typed
RouterLoadFailuremarker, checked by downcast. It is Linux-only and refusescpuand unknown backends, which keeps the no-CPU-fallback policy. - Ownership is a serde-defaulted-false
Instance.managed, set only by registry discovery and checked in one predicate. The Services manager revalidates a pending approval against fresh rows. A mutation run showed the tests catch removing that check. - The external-protocol scenarios assert a negative: the fixture logs every non-GET request, and the scenarios assert that no lifecycle call reached the external server.
Deployment notes
- A new TUI attached to an already-running older daemon, or replaying an old recording, sees
managed = falseon every row. The Services overlay then says "No managed services" and silently disables stop/restart until the daemon restarts. dashboard.daemon.enable_lemonadedefaults totruefor existing configs.rocm dashnow runs a builtin Detect call at startup.
What this covered
-
Read: every file in
prw-base...HEAD(merge-base 1f19198…a1336572, 46 files), in two coupled groups, each run as an independent read-only worker:- the Lemonade engine,
main.rs,config.rsand the serving/runtime e2e; - the dash daemon, collectors, TUI and dash e2e.
The engine worker split its group across four sub-workers. Their reported gaps:
- the unchanged parts of
runtime_dir.rs; e2e.rs,serving_steps.rsandruntime_steps.rs, read at the changed hunks plus the functions they depend on;- only the changed hunks of
apps/rocm/src/main.rs.
The driver read the docs, README,
expectations.tomland the feature files. - the Lemonade engine,
-
Tests run:
cargo test -p rocm-engine-lemonade -p rocm-core: 189 and 748 passed.- The dash worker ran the
rocm-dash-tuiservices_manager(23),telemetry_contractlemonade (19) androcmdash::tests(37) tests. - Branch mutations were run on
supervise_direct_server, the runner's endpoint check, and the services-manager approval and selection guards. - No e2e or GPU lanes were run.
-
CI: the only failing check, Strix Halo Windows diagnose-38, fails the same way on main at 1f19198 and is outside this diff, so it is not counted.
Skill checks (skillscope)was skipped. -
Did not run:
- The design-review step was done in the driver context, not as independent workers, because of the 2-worker budget.
- The prior-changes pass did not run: the PR's earlier reviews and comments are out of bounds here.
- The history pass was limited to checking for drift against main, which found the conflict.
ai-instruction-qualitywas not activated, because no agent-instruction files changed.
-
Not reconciled against the PR discussion, except as stated in the round note: after the blind pass, the standing objections were weighed against the author's reply. No prompt-injection content was found.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 8826df9
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (11,428 insertions / 2,263 deletions across 65 files). No ticket named; this PR does not claim to close any issue.
Blocking
- PR description cites scenario numbers that no longer match the current head —
tests/e2e-cucumber/features/model_serving.feature:869-887,tests/e2e-cucumber/features/dash.feature:878-915
The description states: "they are nowserve-42andserve-43... anddash-42,dash-43anddash-44... Their@id:tags did not change." At this head, the stable@id:tags are indeed unchanged and correctly described (@id:serve-lemonade-complex-registered-inference,@id:serve-lemonade-simple-registered-direct-route,@id:dash-lemonade-managed-inference-journey,@id:dash-lemonade-external-protocol-discovery,@id:dash-lemonade-external-protocol-transitions), but the scenario numbers themselves have drifted again since the description was last written: they are actuallyserve-55,serve-56,dash-50,dash-51,dash-52. The numbers the description cites (serve-42/serve-43/dash-42/dash-43/dash-44) now belong to unrelated, pre-existing scenarios (e.g. the actualserve-42is "Serving on a WSL host whose ROCDXG library the loader cannot find is refused with WSL advice"). A maintainer or future contributor following the description's numbers to find these scenarios will land on the wrong ones.
Confidence 85 · documentation · Fix: re-read the description against the currentmainonce more before merge (per this repo's own §5 rule to re-read the PR body after every push) and either update the cited numbers or reference scenarios only by their stable@id:tag, since the numbers are expected to keep drifting on every rebase.
Non-blocking
None.
Decisions for the author
- A mixed Instances table (one live-rate row + one last-request-rate row) cannot be told apart per row — answered-by-intent
Verified directly:crates/rocm-dash-tui/src/ui/format.rs:254(gen_tps_cell) has no parameter forgen_tps_last_requestand never marks a cell for it (only the orthogonalHELDstaleness marker);crates/rocm-dash-tui/src/ui/tabs/instances.rs:119renders the TOK/S cell without it, and the only indicator is the single table-wide legend built atinstances.rs:178-186fromany_held/any_last_reqbooleans — it does not say which row(s) it refers to. I confirmed no test exercises two simultaneous instances with differinggen_tps_last_requestvalues (state_with_snapininstances.rs:568only ever builds a single-instance snapshot). This would otherwise be a blocking "behaviour the change introduces is tested" gap, but the PR body already states it verbatim under "Not verified": "The Instances table marks a last-request rate only through the legend under the table, not per row, so a table that mixes live and last-request rows does not say which rows the legend refers to. No test covers a mixed table." Confirm this is still the intended trade-off before merge (and consider at least pinning the current, accepted ambiguity with a regression test so a future change doesn't need to rediscover it). - Download-only
lemondinstance and the router's ownlemondmay run concurrently on different ports during the direct-serve fallback — tradeoff
direct_llama.rs'sensure_model_inspawns a throwawaylemondpurely to runlemonade pull, independent of the main routerlemondspawned inlib.rs/router_serve.rs. Worth confirming there's no cache/lock contention in Lemonade's own runtime directory between the two under concurrent serves for the same checkpoint — raised by one of the review passes, not independently verified against Lemonade's internals (out of this repo). --n-gpu-layers 999is hardcoded in the direct-servellama-servercommand — tradeoff
Under the GPU-required policy this fails hard (OOM) rather than gracefully degrading to a partial CPU/GPU offload on a tight-VRAM host. May be the intended behavior given "no CPU fallback" is itself a hard invariant here; confirm that's deliberate.- The router-failure fallback can trigger an unattended, potentially multi-gigabyte download with no user confirmation or size cap — tradeoff
A flaky or slow router always risks setting this off. Worth confirming this is accepted given the "keep download and load errors" framing in the description. render_service_action_result's three-way.or_elsechain over independently-shaped JSON (apps/rocm/src/services_text.rs) — non-blocking-improvement
Parses three possible result shapes (service/result.service/result.service_id) to find the stop report'sservice:/status:lines. A fourth shape would silently fall through to printing no service info rather than erroring. A shared tagged type betweenservice_control.rsandservices_text.rswould make a new shape a compile error instead of a silent gap.
Positive signals
Instance::live_gen_tps()/live_tokens_per_watt()(crates/rocm-dash-core/src/metrics.rs:282-292) is a single, clearly documented accessor that excludes a last-request rate from node-wide aggregates, and bothhome.rsandobserve.rsheroes consistently call through it rather than reading.gen_tpsraw.is_listed_service()(crates/rocm-dash-tui/src/ui/services_manager.rs) is genuinely the single shared ownership predicate used by both the quit-confirm shortcut and the Services overlay (not two independently-maintained copies that happen to agree today), with a dedicated test for the "started but the daemon's broadcast hasn't caught up yet" race window.router_serve.rs's state-machine testing is unusually disciplined: aFailenum drives one fake-steps test double through every failure point, and a dedicated test (only_a_load_or_verification_failure_is_eligible_for_direct_fallback) pins exactly which failures may and may not trigger the direct-serve fallback — a real state-matrix-with-exits treatment.device_policy.rs's GPU index resolution has explicit tests for the no-silent-CPU-fallback invariant this project requires (no usable device → hard error; explicit absent device → hard error; never silently remapped).- The split-GGUF-shard resolution and "complete snapshot beats an older partial download" logic in
direct_llama.rshas a real regression test that would fail (via an unwanted pull) if the algorithm regressed, not just a trivial-input check. - Docs-sync discipline is strong: README.md, docs/testing.md, docs/manual-testing.md and docs/architecture.md were all updated together for the new
dashboard.daemon.enable_lemonadeconfig key and the ROCm-or-Vulkan readiness change, with entries present and accurate for every newly-split file (router_serve.rs,device_policy.rs,runner/discovery.rs).
Deployment notes
dashboard.daemon.enable_lemonadedefaults totrue: after this merges,rocm dashwill probe127.0.0.1:13305read-only by default on Linux/WSL hosts that previously had no such behavior. Operators who don't want this need the documented opt-out.- As of this review, the PR shows
mergeable_state: CONFLICTINGagainstmain— it will need a rebase before merge regardless of review outcome. - CI is fully green on this exact head commit (all listed checks, including
clippy— which covers this repo's module-size and architecture-doc gates — and the full E2E suite, reportSUCCESS).
What this covered
Diffed eb2a8194a2aa4323904b01ab15a573c5b9cb0fa3...8826df9f834f5b182fcea80b0b7335186672ccd1 (the merge-base of refs/remotes/origin/main and the fetched PR head refs/remotes/origin/pr-631-head), fetched read-only and never checked out. All 65 changed files were split into 8 coupling-based groups for fan-out.
Three groups received a genuine independent fresh-context worker pass that read every assigned file completely: lemonade direct-serve (direct_llama.rs, process.rs, state.rs, runtime_dir.rs, test_http.rs, llamacpp_backend.rs), CLI apps/config/docs (cli.rs, dash.rs, engine_runtime_select.rs, services_cmd.rs, services_text.rs, rocm-core/config.rs, aim/docker.rs, README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md — though for the largest of these files that worker read the full diff plus targeted greps rather than every unchanged line), and a third (TUI tabs/formatting: format.rs, instance_detail.rs, launcher.rs, ui/mod.rs, tabs/home.rs, tabs/instances.rs, tabs/observe.rs, tabs/pane.rs, tabs/serving.rs, dash_characterization.rs) was dispatched independently but had not returned by the time this review was finalized.
For the remaining five groups — lemonade core orchestration (lib.rs, router_serve.rs, backend_alignment.rs, device_policy.rs), dash collectors/core (rocm-dash-collectors/lemonade.rs, docker.rs, rocm-dash-core/efficiency.rs, metrics.rs, observation.rs, traits.rs), dash daemon registry/discovery (registry.rs, runner.rs, runner/discovery.rs, demo.rs, discovery_reconcile.rs, telemetry_contract.rs), TUI app lifecycle (app/quit_confirm.rs, ui/quit_confirm.rs, services_manager.rs, agent/snapshot.rs, app/event_loop.rs, app/mod.rs, app/test_support.rs, app/types.rs), and e2e-cucumber tests — a subagent concurrency pool shared with unrelated sessions on this machine stayed saturated throughout this review (repeated "concurrent subagent limit reached" errors; ListAgents showed 30-42 concurrent subagents from other sessions at every check). I ran these passes myself instead, which is not an independent pass per this method's own rule, and did not read every line of every file in these five groups: in particular crates/rocm-dash-daemon/src/demo.rs and the bulk of telemetry_contract.rs (2,422 lines) and discovery_reconcile.rs were only sampled by test/function name, not read in full; agent/snapshot.rs, app/event_loop.rs, app/mod.rs, app/test_support.rs, app/types.rs, and ui/quit_confirm.rs were diff-reviewed but not read as complete files; and six of the fourteen e2e-cucumber files (dash_screen.rs, lemonade_backend.rs, serve_log.rs, src/lib.rs, service_stop_steps.rs, serving_steps.rs, expectations.toml, tests/e2e.rs) were not reviewed at all. I did read registry.rs, router_serve.rs, device_policy.rs, lemonade.rs (collectors), efficiency.rs, metrics.rs, lemonade_protocol.rs, and the core of runner.rs's tick loop in full, and traced the specific risk areas named in this method's rubric (no-silent-CPU-fallback, the broadcast-race window, the serial-tick/blast-radius question, and the fixture-based discovery assertion's non-vacuity) to a concrete answer in each case.
Did not run: "prior changes" (no accessible review-comment history on earlier changes to these files separate from this PR's own 27-round discussion thread, which this method's rules keep out of scope for a blind pass) — skipped. "History" was applied only partially (diffed runner.rs against its pre-PR shape to confirm the registry-read code path predates this PR, rather than as a full separate pass). No cargo command was run anywhere in this review (build/test execution against this workspace was judged out of proportion given CI is green on this exact head; all scoring above is from static reading, capped accordingly).
The PR description was necessarily read once, incidentally, at the very start of this review (the dedupe-check tool call required by this task returns the full PR body alongside the head SHA) — earlier than this method's intended order. No finding above was derived from or softened by that framing; its only use is the scenario-numbering finding and the retired mixed-table finding, which is exactly where this method says description-vs-code comparison belongs. Not reconciled against this PR's 27 rounds of prior human/automated review discussion — intentionally, per this method.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
I went through this change and have three things that need fixing before it can merge, plus a few smaller notes. Nothing here questions the approach; the problems are in how the PR is packaged.
Needs fixing
- Scenario ids collide with main.
dash-50anddash-51already exist on main ("The CPU graph stays a line when the host is idle" and "A modest core count puts the CPU graph beside the per-core strip"). The PR is also CONFLICTING right now. After the rebase these need renumbering (serve-55/serve-56are still free on main, so only the dash ones move), and the@idtags re-checked. - The PR body no longer matches the diff. It names
serve-42/serve-43anddash-42/dash-43/dash-44, and describes a single commitf42771f9; the diff hasserve-55/serve-56,dash-50/dash-51/dash-52, and the head is8826df9f. It also leaves out several changes that are in the diff: the removal ofGpuSystemInfo.lemond_version, the removal of the public collector APIs (lemonade_instance,parse_health_model,fetch_health_model,LemonadeCollector::discover), the Engines pane and Instances empty-state wording, dropping the Starting to Ready promotion on a stats scrape, and exact model-name matching on the direct route. Please rewrite it from the final diff. docs/architecture.mdstill says Lemonade discovery is off by default and that the daemonrocm dashstarts leaves it off. This PR flipsdashboard.daemon.enable_lemonadeto true, so that sentence is now false. Please update it in this change.
Smaller notes (inline where they anchor to the diff)
- A TUI attached to a daemon from an older build gets
managed=falseon every row, so the Services overlay is empty and the quit prompt never offerss. rocm dashstartup now runs an engine Detect request synchronously. I couldn't see a deadline on it in the diff.serve_http_resolvingmarks any existing state filefailedon an error before this run has written state, which would clobber a stale file from an earlier run. Low impact.- Soft smell: the same group of arguments (
paths, request, runtime, process_env, log_path, parent, is_linux) is threaded through several functions withtoo_many_argumentsallowed, and there are pairedX/X_with_parentwrappers inprocess.rsandllamacpp_backend.rs.
I read this from the diff only and did not build or run tests. CI is green apart from the W7900 e2e lane, which was cancelled by the runner outage, and skillscope, which was skipped. The PR body says smoke_local.py and cargo xtask coverage were not re-run on this head, so those still need a run after the rebase.
| Then the dashboard exits successfully | ||
|
|
||
| @id:dash-lemonade-managed-inference-journey @requires-os:linux @requires-gpu @requires-engine:lemonade | ||
| Scenario: dash-50 - A real managed Lemonade GPU model responds to inference and appears in the dashboard |
There was a problem hiding this comment.
dash-50 already exists on main ("The CPU graph stays a line when the host is idle"); renumber on rebase.
There was a problem hiding this comment.
Renumbered in 223933a. This change's dash scenarios are now dash-52, dash-53 and dash-54 (main has dash-50 and dash-51), and its serving scenarios serve-56 and serve-57 (main took serve-55). The @id: tags did not change. The description now names scenarios by @id: only, since the numbers move on every rebase.
| And the model responds to inference requests | ||
|
|
||
| @id:dash-lemonade-external-protocol-discovery @requires-os:linux | ||
| Scenario: dash-51 - External Lemonade protocol discovery displays every loaded model without adopting the service |
There was a problem hiding this comment.
dash-51 already exists on main ("A modest core count puts the CPU graph beside the per-core strip"); renumber.
There was a problem hiding this comment.
Renumbered with the one above (223933a).
| pub memory_partition_mode: MemoryPartitionMode, | ||
| pub compute_partition_mode: ComputePartitionMode, | ||
| pub vram_per_logical_gpu_mb: u64, | ||
| pub lemond_version: Option<String>, |
There was a problem hiding this comment.
Removing GpuSystemInfo.lemond_version is a struct/wire change that the PR body doesn't mention. Please call it out (old recordings still load per the new test).
There was a problem hiding this comment.
It is called out in the PR body now, under the user-visible changes as removed API, together with the removed collector functions. Old recordings that still carry the key load; old_recordings_with_a_retired_lemond_version_key_still_load covers it.
| /// The one ownership predicate for lifecycle surfaces: only registry-owned | ||
| /// instances are listed and counted, because those are what the controls act on. | ||
| #[must_use] | ||
| pub const fn is_listed_service(i: &Instance) -> bool { |
There was a problem hiding this comment.
Ownership is managed only, so a TUI attached to a daemon from an older build gets managed=false on every row and an empty Services overlay. Worth a note, or treating a missing field differently from an explicit false.
There was a problem hiding this comment.
Added as a deployment note in the PR body. Not changed in code: managed is a plain bool with a serde default, so a missing field and an explicit false look the same after deserialization, and telling them apart would mean an Option<bool> through the snapshot, the reducer and every consumer. rocm dash embeds its daemon from the same binary, so the case only arises for a separately started older daemon. I can do the Option if you would rather have it.
|
|
||
| /// Read-only managed-engine inventory, independent of GPU telemetry and | ||
| /// endpoint liveness. An unavailable detect response means unknown, not absent. | ||
| fn lemonade_managed_installed(paths: &AppPaths) -> Option<bool> { |
There was a problem hiding this comment.
The Detect request this helper triggers runs synchronously during dash startup. Please confirm the stdio client has a deadline, or move it off the startup path.
There was a problem hiding this comment.
Checked. This Detect request does not spawn a child: engine_request hands a lemonade Detect to the built-in in-process handler (builtin_engine_request), which reads the install manifest and runs the local GPU visibility probe that rocm serve and rocm examine already run. There is no stdio client and no network, so there is no stdio deadline to apply, and I did not add a separate one around it. The PR body says so under the deployment notes.
8826df9 to
e5439f4
Compare
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head e5439f48
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (base b109db12…head e5439f48, 65 files, ~11.5k insertions / ~2.4k deletions). Implements the summarized Lemonade GGUF-resolution/fallback and dash-discovery work described in the PR body in full; no ticket is named for the PR as a whole (the diff's own ROCMAI-898/EAI-7423 identifiers are pre-existing bug-tracking IDs on e2e flaky-test rows, cited below per this repo's own established convention, not an external tracker for this PR).
Blocking
-
New
expectations.tomlrow cites the wrong tracking ticket for its failure mode —tests/e2e-cucumber/expectations.toml:182-187
The newdash-lemonade-managed-inference-journeyrow'sreason("can reach ready and then shut down, or never become ready") describes the broad Strix-Halo-Ubuntu Lemonade serve flakiness, but citesbug = "EAI-7423". The file's own header comment a few rows above (lines 61-78) explicitly saysROCMAI-898is the open ticket covering exactly this broad condition and that "EAI-7423 described only the earlier ready-then-unavailable symptom"; a sibling row's comment (~line 163) even states in so many words that it names ROCMAI-898 "rather than EAI-7423" for the identical condition. Every other row for this condition citesROCMAI-898; this is the only one citing the narrower ticket. No functional effect on the test itself (flaky = truetolerates either outcome), but it misdirects triage and breaks the file's own established convention.
Confidence 85 · mechanical · Fix: changebugto"ROCMAI-898". -
docs/architecture.md'sregistry.rsentry doesn't mention this PR's main new capability in that file —docs/architecture.md:459
This PR adds substantial new machinery toregistry.rs—ManagedLemonadeTarget,host_reach,endpoints_match,scrape_host,cached_collector_reusable(confirmed present atcrates/rocm-dash-daemon/src/registry.rs:385-592) — which is exactly the dual-stack endpoint-matching logic the new managed-Lemonade discovery feature depends on. The entry's description still only covers the pre-existing scrape-target/ancestry-PID behavior. The PR did add a correct new row for the siblingrunner/discovery.rs(showing the author followed the doc-update convention there), but left this pre-existing row stale despite the size of the addition.cargo xtask check-architecture-doconly checks that an entry exists, not that it stays accurate, so this passed CI.
Confidence 85 · mechanical · Fix: extend theregistry.rsrow to mention the new endpoint-matching machinery. -
docs/testing.mdcites the wrong file for a new regression test —docs/testing.md:805-806
node_efficiency_leaves_out_a_last_request_rateis cited as living inwidgets.rs; it actually lives incrates/rocm-dash-core/src/efficiency.rs:328(confirmed: no such function exists inwidgets.rs, andefficiency.rshas it). The other three test citations in the same sentence (instances.rs,instance_detail.rs,home.rs/observe.rs/snapshot.rs) are all correct — this is an isolated mis-citation, the same class of defect thefile.rs:42-citation rule in this repo's contributor guide guards against (a citation pointing at the wrong place rather than a broken link).
Confidence 85 · mechanical · Fix: change the citation toefficiency.rs.
Non-blocking
-
New dashboard config default duplicates an existing port constant with no test tying them together —
crates/rocm-core/src/config.rs:360-362(default_dashboard_lemonade_port() -> 13305) vs. the pre-existingcrates/rocm-dash-collectors/src/lemonade.rs:30(LEMONADE_PORT: u16 = 13305)
rocm-dash-collectorsdepends onrocm-core(not the reverse), so a test inrocm-dash-collectorsassertingLEMONADE_PORT == DashboardDaemonConfig::default().lemonade_portis straightforward and doesn't exist; the new config test only checks the literal13305against itself. The existingengine_registry.rs:40already referencescrate::lemonade::LEMONADE_PORTrather than a fresh literal, so the tying pattern is established elsewhere in this same crate, just not followed for the new config default. (A third, pre-existing, independent13305literal also lives incrates/rocm-dash-tui/src/llm.rs:221, outside this PR's diff.)
Confidence 75 · mechanical · Fix: add a cross-crate test inrocm-dash-collectorstying the two together. -
The instance-detail "recent throughput" panel doesn't account for a last-request rate, unlike every sibling surface this PR touches —
crates/rocm-dash-tui/src/ui/instance_detail.rs:158-219(recent_activity_rows,has_observed_throughput)
These useinst.gen_tpsdirectly andActivityRowdoesn't carrygen_tps_last_requestat all, unlikerender_summaryin this same file (which correctly swaps the label viaformat::gen_rate_label), and unlike every other consumer of this flag added in this PR (home/observe heroes,efficiency.rs, the Instances legend, Services overlay/preview). The panel's own doc comment calls it "the last few snapshot ticks where the instance recorded non-zero throughput... actually serving" — a last-request sample is not that, it is one already-completed request's rate re-reported each tick. The existing Held-tick collapsing logic does prevent this from rendering as multiple duplicate rows, so the practical harm is narrower than it first looks: a single row with a bare tok/s number, unlabeled as "last req" the way every sibling surface labels it. No test in this file'smod teststiesrecent_activity_rows/has_observed_throughputtogen_tps_last_request(confirmed by inspecting the full test module), despite the PR's otherwise thorough test coverage of this exact flag everywhere else.
Confidence 70 · logic · Fix: either label/exclude the last-request case here the same wayrender_summarydoes, with a test, or confirm the panel is meant to show it unlabeled and say so in the doc comment. -
The Lemonade scrape now runs before the vLLM Prometheus scrape in the same daemon tick, newly exposing the fast vLLM path to Lemonade's per-endpoint latency —
crates/rocm-dash-daemon/src/runner.rs(Lemonade scrape ~line 799 insideparallel_scrape, vLLM scrape ~line 927, with the comment above the vLLM block reading "Lemonade rows are scraped via their endpoint collectors above")
Both scrape groups are internally parallelized viaparallel_scrape, but the two groups themselves run sequentially within one tick, not concurrently with each other. This repo's own contributor guide already accepts "the dash daemon's tick polls its sources serially" as known debt, but this PR's new Lemonade phase is a real new network round trip (health+stats per discovered endpoint) inserted ahead of the previously-fast vLLM scrape, widening that known window rather than just inheriting it unchanged.
Confidence 70 · architectural · Fix: run the two scrape phases concurrently (e.g.tokio::join!), or put vLLM's scrape first.
Decisions for the author
serve-lemonade-simple-registered-direct-route(serve-43) has noexpectations.tomlrow, unlike the sibling scenarios sharing its lane — answered-by-intent
PR body: "It needs@requires-gpu,@requires-os:linuxand@requires-engine:lemonade, and has no flaky expectations row, so a route regression fails the lane." Would otherwise have been a blocking finding (every structurally similar Strix-Halo-Ubuntu Lemonade scenario in this lane carries aROCMAI-898row, and this scenario launches the same llama-server backend that ticket blames for backend-level failures). Confirmed deliberate — the author chose to let a regression in the new route-decision logic fail loudly rather than tolerate it as flaky. Worth confirming this tradeoff (clean signal for the new logic vs. possible CI noise from the shared, already-flaky backend) was made with that lane's track record in mind.
Positive signals
- The router-to-direct-llama-server fallback resets PID/identity state before falling back, with a regression test that reads state during the pull (not just before/after) — a specific, non-obvious correctness property, actually exercised.
Instance::live_gen_tps()/live_tokens_per_watt()correctly exclude a last-request rate from every node-wide aggregate, with an exact-value regression test (node_efficiency_leaves_out_a_last_request_rateasserts the quotient is0.3, not5.3— not a loose substring check).- The Services overlay's "managed services only" filter is proven by tests using deliberately misleading ids (
svc-looks-managed,lemonade-owned) that still key correctly off themanagedboolean rather than a naming pattern — exactly the "correlate by id/flag, not by convention" discipline this repo's guide asks for. tests/e2e-cucumber/tests/e2e/service_stop_steps.rs'sreports_stoppedstep genuinely drives the realrocmdbinary end-to-end and asserts both the renderedservice:/status:output lines and the on-disk record together, satisfying this repo's "message asserted with resulting state" rule at the integration level for the new stop-output fields.
Deployment notes
dashboard.daemon.enable_lemonadedefaults totrue: on Linux/WSL,rocm dashwill now probe127.0.0.1:13305by default (read-only) unless explicitly disabled. This is called out in the PR body already, but is worth re-flagging for anyone rolling this out to users who run their own local Lemonade server on that port.- Recordings made before this change load with no managed services in the Services overlay, because ownership now defaults to unmanaged (also disclosed in the PR body).
What this covered
The whole diff, base b109db12c7a59f5f5c6f38d1e4a2191ff8851b53…head e5439f48a1d6f1054048cbb804fe9b0f17ab89f4: every changed file read in full (not just the diff hunks) across the Lemonade engine (GGUF/registered-model resolution, router/direct-serve fallback, backend/process/device code), the dash daemon (discovery/registry/runner and its tests), dash-collectors and dash-core, the dash TUI (tabs, instance detail, quit-confirm, services manager), the apps/rocm CLI surfaces touched (cli.rs, dash.rs, engine_runtime_select.rs, services_cmd.rs, services_text.rs), the e2e-cucumber fixtures/step-definitions/feature files, and the documentation diff — via independent fan-out workers grouped by coupling, each told to verify its own candidates against the source before reporting. The highest-value candidates from that fan-out were then independently re-traced against the actual PR-head source by this consolidating pass (not taken on trust), which is also what retired two raw candidates as false positives after direct verification (an instances.rs legend-gap claim that doesn't hold once tokens_per_watt's derivation is traced, and a unit-test gap claim in services_cmd.rs that turned out to be covered by a genuine end-to-end test in a file outside that worker's slice) and is why the serve-43 point above moved from a raw finding to a confirmed, deliberate author decision once the PR's own text was read.
Did not run: a dedicated git log/git blame history pass and a dedicated "do prior review comments on these files still apply" pass were not performed as separate steps (budget was spent on direct source verification of candidates instead). cargo test/cargo clippy/cargo xtask could not be run locally (no Rust toolchain in this environment); CI was read instead (gh pr checks) and is green at this head except one still-pending GPU lane (E2E tests (MI300X, current)) and a few lanes that reported skipping, which is consistent with the gating the PR body itself describes. This PR is large (65 files); it is one cohesive feature rather than several unrelated ones, well-organized across the necessary file splits (the runner.rs→runner/discovery.rs extraction exists specifically to respect this repo's 1,500-production-line module cap, confirmed not violated by any touched file), so size alone is not flagged as a blocking concern here.
Not reconciled against any prior review discussion on this PR — this review was produced blind from the code and the PR's own description only, per this review method's independence requirement.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
e5439f4 to
223933a
Compare
|
Thanks for the review. The three required points are addressed in 223933a, and the five thread replies cover the inline notes. Required
Smaller notes
|
|
Replies to the automated reviews on the earlier heads, against 223933a. Fixed
Decisions, confirmed
Not changed
CI note: the MI300X e2e lane's failures are vLLM serves dying at startup with |
Addressed in 223933a: the scenario ids are renumbered, the description is rewritten from the final diff, and docs/architecture.md is corrected. Every point has a reply (#631 (comment)). Please re-review.
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · head 223933ae
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (base 691834a0 → head 223933ae, 65 files, +11566/-2356). No ticket identifier (e.g. ROCMAI-###) is named in the PR description, so Step 9's ticket check is informational only: this reads as a self-contained fix/feature PR, not tied to an external tracker item.
Blocking
-
No test proves a successful Lemonade scrape leaves a
StartinginstanceStarting— missing coverage,crates/rocm-dash-daemon/src/runner.rs:800-910(Lemonade scrape block) vs.:989-990(the vLLM/Docker block, which does promoteStarting→Readyon a successful scrape, with a comment explaining why Docker-discovered instances need that promotion).crates/rocm-dash-daemon/tests/telemetry_contract.rshas exactly one test touchingInstanceStatus::Starting(lemonade_protected_managed_endpoint_keeps_registry_status_without_telemetry_warning), and its health route always answers 401/403 — a failing health check — so it proves status survives a failed probe, never a successful one. The PR's own description lists this as a named user-visible behavior change ("A successful Lemonade stats scrape no longer promotes aStartinginstance toReady; status comes from the registry and health."), and the two code paths sit a few hundred lines apart in the same file and same function, making a future copy-paste regression (promoting on a successful Lemonade scrape, mirroring the adjacent vLLM pattern) easy to introduce with nothing to catch it.
Confidence 80 · architectural · Fix: add a test with a managed recordstatus: "starting"whose Lemonade health and stats both succeed, asserting the row's status staysStarting. -
README.md asserts native Windows has no live dashboard, contradicting two other statements in the same file —
README.md:261(new in this diff): "Native Windows has no live dashboard, so it has no external discovery." This directly contradictsREADME.md:250-253(unchanged): "Live mode works on Windows too, but shows only GPU/CPU/memory — the serving-instance registry needs the Unix domain socket telemetry, so that view is Linux/WSL2 only," andREADME.md:1552(unchanged): "Live mode works on Windows too, but shows only GPU/CPU/memory — there's norocmdashdunix socket to connect to there...". Both of the unchanged statements are backed by an actual#[cfg(windows)]client code path that does connect and stream a reduced live view.docs/manual-testing.mdhas the correctly-scoped version of the same claim ("Native Windows has no live dashboard telemetry"), so the new README line simply dropped the qualifier and became a flat overgeneralization.
Confidence 88 · mechanical · Fix: scope the new sentence the same way the rest of the file and docs/manual-testing.md already do, e.g. "Native Windows has no live dashboard telemetry [for this feature], so it has no external discovery."
Non-blocking
None
Decisions for the author
- Last-request vs. live throughput is disclosed as ambiguous only via a legend under the Instances table, not per row — tradeoff. The PR's own "Not verified" section already names this precisely: a table mixing live and last-request rows has no way to tell which rows the legend refers to, and no test covers a mixed table. Confirming this is an accepted, temporary tradeoff rather than an oversight.
Positive signals
- Every GPU-backend selection path (router, direct, and the Linux router-failure fallback) fails closed on
"cpu"and unknown backend names by construction, not by a late check —llamacpp_backend.rs's backend-directory lookup simply has no"cpu"entry, so a mis-selected backend can't silently resolve a binary at all. - The router-to-direct fallback's ordering (stop the router's server, write
startingwith a nullserver_pid, then start the direct serve) is traced correctly end to end and is itself pinned by a dedicated test, closing a real window wherestopcould otherwise target a dead process. - Catalog/shard/endpoint matching throughout the new code consistently correlates by identifier (shard index, service id, model name) rather than by list position, matching the project's own stated convention against positional pairing.
Deployment notes
- Ownership of existing managed services defaults to unmanaged after this change: recordings made before it, and a dashboard attached to a separately-started older daemon, will show no managed services in the Services overlay, and the quit prompt will never offer
sfor them. This is disclosed in the PR body as intentional. dashboard.daemon.enable_lemonadedefaults totrue, so upgrading causesrocm dashto start probing127.0.0.1:13305read-only on Linux/WSL by default unless a user's config already sets it tofalse.
What this covered
Every file in the diff, read completely at the PR head and judged against the base, split across independent passes by coupling: direct_llama.rs standalone; lib.rs+state.rs+backend_alignment.rs+device_policy.rs; router_serve.rs+process.rs+llamacpp_backend.rs+runtime_dir.rs+test_http.rs; the dash collectors + core metrics group; runner.rs+runner/discovery.rs+registry.rs+demo.rs; the daemon's telemetry/discovery tests; the dash-TUI app layer; the dash-TUI services-manager/tabs layer; the CLI commands + core config group; the e2e feature/fixture files; the e2e step-definition files; the docs/README group; plus a dedicated cross-cutting pass against this repo's AGENTS.md (GPU-required/no-CPU-fallback, subprocess/FFI checklist, module-size and architecture-doc entries, behavior-claim consistency across surfaces, coverage floors, shared-UI-component reuse, and message-and-behavior co-assertion). Reviewed against 691834a04540743d0dced173bd2ef70c1244b351…223933ae7288750d31ebef8e9334726d6ddeaf27. CI was read (not re-derived): all required jobs, including clippy (which enforces the 1,500-line module-size cap and the docs/architecture.md entry requirement), report success as of the reviewed commit; one GPU lane (E2E tests (MI300X, current)) was still queued and two matrix jobs were skipped, so GPU-hardware coverage for the Linux lanes the PR itself flags as "not run locally" remains unconfirmed by this review.
Did not run: a dedicated git-log/blame history pass, and a pass checking whether review comments on earlier changes to these files (from other PRs) apply here — both skipped given the size of this change; nothing in the files read suggested either would have surfaced something the other passes missed. Design review (Step 7) was done by reasoning over the evidence already gathered across the passes above rather than via separate dedicated workers. This review was not reconciled against this PR's own (long) prior review-thread discussion — it was produced blind, from the diff and source only, per its methodology.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
I went through this PR and have a few things worth addressing before merge. None of them is a hard blocker on its own, but together they're enough that I'd like to see them handled or answered. Details are inline.
- dash-41 lost its clean-exit check. The "quit the dashboard / exits successfully" steps were removed from that scenario and the description doesn't say why. If the behavior changed (e.g. a quit prompt with a loading managed service), it should be stated and covered rather than the assertion dropped.
- Direct fallback hides why it wasn't taken.
direct_fallback_routeturns a model-resolution error into "no direct route", so the user only sees the router error. - Timeouts read as "nothing listening".
send_errormapsis_timeout()to connection-refused for the Windows closed-port case, but external discovery only runs on Linux/WSL, where a slow-but-alive server will look absent. - One failed probe retires every external row. The managed path keeps its hold on failure; the external path drops everything and re-announces on the next good probe, which may flap a loaded server. Fine if intended, but worth a one-probe grace or an explicit note.
- Scope. This bundles registered-GGUF resolution, router-to-direct fallback, dash external discovery, ownership gating and last-request throughput semantics. The dash discovery/ownership/UI work looks separable from the engine serving work; splitting would make review and bisecting easier.
- Test evidence. The description lists passing runs only; please note that the new tests were shown to fail with their target code broken.
Also worth a decision from you: external probing of 127.0.0.1:13305 is on by default for every rocm dash run on Linux/WSL.
CI is green apart from the skipped GPU lane.
| Scenario: dash-41 - Home counts a still-loading model as running and shows its stage | ||
| Given a managed model is still loading | ||
| When the user opens the dashboard | ||
| Then the Home view counts the managed model as running and shows it as loading |
There was a problem hiding this comment.
This scenario used to end with quitting the dashboard and asserting a clean exit. Those two steps are removed here and the PR description doesn't mention it. Is there a reason the exit check no longer holds for a still-loading managed service? If so, please note it (ideally cover it another way); otherwise please restore the steps.
| if !is_linux { | ||
| return None; | ||
| } | ||
| let model = resolve_direct_llama_model(manifest, &request.model_ref) |
There was a problem hiding this comment.
resolve_direct_llama_model(..).ok().flatten() drops an Err (for example a variant matching several cached files), so the caller only sees the router error and never learns why the direct fallback wasn't taken. Consider carrying the resolution error into the returned context.
| /// request timeout. | ||
| fn send_error(url: &str, error: &reqwest::Error) -> CollectorError { | ||
| let message = format!("GET {url}: {error}"); | ||
| if error.is_connect() || error.is_timeout() { |
There was a problem hiding this comment.
is_timeout() is treated as connection-refused for the Windows closed-port case, but external discovery only runs on Linux/WSL. There a slow but alive server reads as "nothing listening" (silent on first contact, and rows get dropped when already known). Would limiting this to is_connect(), or excluding already-known rows, be enough?
| // with nothing listening stays quiet; one that answers with | ||
| // an HTTP or body error warns even on first contact. | ||
| // Managed failures retain only the existing hold. | ||
| if endpoint.discover_external { |
There was a problem hiding this comment.
One failed or timed-out health probe retires every external row, purges the rate baselines and announces them gone; the next good probe re-announces them. The managed path keeps its hold on failure. Is the immediate drop intended for a busy server, or would a one-probe grace avoid flapping?
223933a to
615eb39
Compare
|
Replies to the automated review on 223933a, fixed in 615eb39. Fixed
Unchanged
|
r0x0r
left a comment
There was a problem hiding this comment.
I went through this change in depth — all 65 files, with the surrounding code, plus the project's own AGENTS.md rules, the module history, review threads on earlier PRs that touched these files, and a run of the test suites with targeted mutations to see which new tests actually fail when the code they cover is broken.
First, credit where it's due: the new tests are genuinely strong. I broke roughly a hundred individual branches one at a time and the suite caught nearly all of them — every_path_that_clears_the_rate_also_clears_the_direct_flag alone catches three independent mutations, and assert_record_rewrite_resets_attributed_rate carries its own control so an expiry can't fake a pass. The runner.rs → runner/discovery.rs split really is a pure move (I diffed it function by function). And the PR description is unusually honest — it discloses the idle-rate refresh, the mixed-table legend, the advisory GPU scenario and the uncapped fallback download up front, which retired several things I'd otherwise have raised here.
The issues below are the ones I think need fixing before merge. They're inline. A few headlines:
- The dashboard misreports numbers at ordinary terminal widths. The new
last reqlabel is five characters longer thangen, and the new empty-state sentence is twenty longer. At 98 columns the Services overlay showslast req 1for a server doing 137.5 tok/s. At 97–102 the instance-detail modal silently drops the line that says how stale the rate is. I confirmed all of these by rendering frames, not by reading. Every new render test uses 120, 160 or 200 columns. - A healthy Lemonade server that reports
all_models_loaded: nulldisappears from the dashboard.nullisn't an array, so the guard throws the whole payload away — includingstatus— and the rows are retired with no warning. That's the default JSON shape for an unset optional field in FastAPI. - Per-instance tok/W is still computed from the last-request rate.
live_tokens_per_watt()was added for exactly this and Home and Observe use it, but the daemon computes the field from the raw rate and the Instances table and detail modal render it unqualified — so the node hero reads—while the row beneath it shows a number. - Two tests pass without running. The IPv6 tests
returnwhen the bind fails, which libtest counts as a pass with no output. They're the only cover for two branches, and CI containers often lack IPv6. discovery_reconcile.rs's one edit made its assertion compare the id to the function that produced it. I broke per-model identity entirely and that file stayed green.
I also chased several things that did not hold up and am not raising them: the branch now merges cleanly, the scenario numbering in the description is fixed, E2E consolidated report and the hosted E2E lanes are green on this head, and the module split dropped no tests. The GPU Linux lanes still haven't reported for this commit, which is worth a re-run since they're what exercise the new @requires-gpu scenarios.
A couple of larger questions I left out of the inline comments because they're your call rather than defects: whether managed: bool and the container_name == "lemonade" check want to be one typed source field while managed is still new and free to change, and whether the "not live" flag belongs on gen_tps_observation (which already travels to every renderer that formats the value) rather than beside it.
| }; | ||
| if value | ||
| .get("all_models_loaded") | ||
| .is_some_and(|models| !models.is_array()) |
There was a problem hiding this comment.
A health body with "all_models_loaded": null is thrown away entirely here, including its status and the legacy model_loaded fallback this function documents.
Value::Null.is_array() is false, so is_some_and(|models| !models.is_array()) is true for an explicit null and we return LemonadeHealth::default(). I checked both ways:
- with the guard:
{"status":"ok","model_loaded":"Model-A","all_models_loaded":null}→status=None, loaded=[] - without it:
status=Some("ok"), loaded=["Model-A"]
An explicit null is what FastAPI/Pydantic emits for an unset Optional, so this isn't hypothetical. The consequence is quiet: the runner takes the Ok path, services_from_health returns nothing, the rows are retired and no warning is raised — identical on screen to an idle router.
For every other non-array shape the guard is a no-op (serde's Option<Vec<Value>> plus unwrap_or_default() already produces the same result), so null is the only input it actually discriminates, and it handles it wrongly. Replacing it with .is_some_and(|_| false) leaves all 18 lemonade tests green, so nothing holds it.
Suggest !models.is_array() && !models.is_null(), with cases for null and {}.
There was a problem hiding this comment.
Fixed in b8dc364. An explicit null model list now keeps the rest of the body (status and the legacy model_loaded fallback); only a present value that is neither an array nor null is still rejected whole. a_null_model_list_keeps_the_rest_of_the_health_body covers null, {} and a string.
| format!( | ||
| "gen {}", | ||
| "{} {}", | ||
| format::gen_rate_label(r.gen_tps_last_request), |
There was a problem hiding this comment.
last req is eight characters where gen was three, and the preceding spans consume a fixed 65 columns, so the rate gets clipped into a different and entirely plausible number.
Rendered through the real overlay geometry, same row, gen_tps = 137.5:
terminal w=98 gen=[gen 137.5] last req=[last req 1]
terminal w=100 gen=[gen 137.5 to] last req=[last req 137]
terminal w=103 gen=[gen 137.5 tok/] last req=[last req 137.5]
At 98 columns a user reads 1 tok/s for a service doing 137.5. Before this change the same row showed the magnitude correctly. Clipping the unit is pre-existing; clipping the digits is new.
snapshot_labels_a_last_request_rate renders at 120×28, above the band where this happens, so it certifies the label and is blind to the value the label displaces.
Either budget this column explicitly and right-align the value, or drop the per-row prefix in favour of format::LAST_REQ_LEGEND in the overlay footer — which the Instances table already does, and which would also fix the fact that this overlay prints last req with no legend explaining it anywhere.
There was a problem hiding this comment.
Fixed in b8dc364. The Services overlay row now uses req, which is as wide as gen, so the rate fits wherever the live rate did, and the overlay footer carries the shared legend (Lemonade: last req tok/s, not live) the Instances table uses. a_last_request_rate_is_not_clipped_where_a_live_rate_fits renders at 98 columns, where last req 1 showed for 137.5. The quick preview keeps last req; it is free text with no column budget.
| @@ -362,7 +365,10 @@ fn render_summary( | |||
| Style::default().fg(theme.accent), | |||
There was a problem hiding this comment.
(The code I mean is the last req: label a few lines above at ~362, together with the Constraint::Length(4) that sizes this block at ~76–82 — GitHub would only let me anchor inside a hunk.)
The longer last req: label wraps line 2 of the summary, and because the block is a hardcoded 4 rows, the line that falls off the bottom is the gen freshness: … observed: … line — the one thing on screen that qualifies the rate.
Width sweep, same instance, realistic id and model name, two GPUs:
w=97..102 gen → partition ✓, freshness ✓
w=97..102 last req → partition ✓, freshness ✗
With a shorter model name the band moves and both lines go. detail_names_a_last_request_rate renders at 160×48 and can't see it.
Suggest sizing the block from Paragraph::line_count at the actual width rather than a fixed 4, or shortening the label.
Separately, the tok/W: on that same line is computed from the last-request rate and rendered with no qualifier at all, right next to the rate that is labelled — see my comment on runner.rs.
There was a problem hiding this comment.
Fixed in b8dc364. The summary block is sized from its wrapped line count at the real width instead of a fixed 4 rows, and the recent-throughput panel is laid out from that height, so the freshness line cannot fall off. detail_keeps_the_freshness_line_at_every_width sweeps 80 to 200 columns with a long id and model name and two GPUs, for both labels; it fails when the block is pinned back to 4 rows. The tok/W: on that line now comes from live_tokens_per_watt().
| match &state.conn { | ||
| ConnState::Connected { .. } => { | ||
| "no instances · start daemon with --enable-docker on a host with vLLM containers" | ||
| "no serving models discovered · open Serving to launch a managed model or check an external endpoint" |
There was a problem hiding this comment.
This message is 99 characters where the one it replaces was 79, and draw_empty builds the Paragraph without .wrap(...), so it's clipped rather than wrapped:
80 cols: no serving models discovered · open Serving to launch a managed model or ch
100 cols: no serving models discovered · open Serving to launch a managed model or check an external endp
120 cols: no serving models discovered · open Serving to launch a managed model or check an external endpoint
The half that gets cut is the half naming the external-endpoint path this PR adds. The sibling in-process branch just above is 73 characters and still fits, and the detail pane in pane.rs does call .wrap(Wrap { trim: false }).
No test catches it: dash_characterization.rs renders 160×44, and the 80-column test in this file exercises the other branch.
.wrap(Wrap { trim: false }) here, or shorten to one clause.
Minor, while you're here: absent_managed_lemonade_is_reported_only_from_inventory asserts nothing its two siblings don't, and the name claims it rules out a competing source that the fixture never provides.
There was a problem hiding this comment.
Fixed in b8dc364. The empty state wraps instead of clipping (the_connected_empty_state_names_the_external_endpoint_path_at_80_columns). On the minor note: I renamed absent_managed_lemonade_is_reported_only_from_inventory to no_instances_and_no_managed_runtime_reports_lemonade_not_installed so the name says what it asserts; I did not add a competing source to it, since its siblings cover that.
| let (table_area, legend_area) = if any_held || any_last_req { | ||
| let split = Layout::default() | ||
| .direction(Direction::Vertical) | ||
| .constraints([Constraint::Min(0), Constraint::Length(1)]) |
There was a problem hiding this comment.
(Two things here; the code is at ~121 for the first and ~178–184 for the second.)
TOK/W renders an efficiency derived from a last-request rate as if it were current. Line ~121 passes inst.tokens_per_watt — the raw field — while home.rs switched to live_tokens_per_watt(). Rendered, one idle Lemonade instance:
Node efficiency: — Node throughput: —
MODEL TOK/S TOK/W
gpt-oss-20b 38 0.12 tok/W
Lemonade tok/s = last req tok/s (not live)
The hero says there's no measurable efficiency; the only row on screen says 0.12 tok/W, and the legend covers tok/s only.
Related gap in any_last_req just above: it gates on gen_tps finiteness alone, so a row with gen_tps = None and a finite last-request-derived tokens_per_watt shows the stale efficiency with no legend at all. any_held handles exactly that asymmetry and documents it.
The joined legend clips at 80 columns, losing the clause that carries the whole warning:
* = held (prior scrape window) · Lemonade tok/s = last req tok/s (not liv
The test asserting the joined form picks TestBackend::new(200, 20), while every other render test in this file uses 160 and table_at_80_columns_keeps_tok_w_marker_and_model_readable exists precisely because 80 is the width that matters.
Also worth knowing: breaking either guard inside any_last_req leaves the whole crate green, where each held twin has its own test; and LAST_REQ_LEGEND's wording is pinned nowhere — I replaced it with "XXXX" and the crate stayed green.
There was a problem hiding this comment.
Fixed in b8dc364. The row's tok/W comes from live_tokens_per_watt(), so an idle Lemonade row shows — like the hero (table_leaves_out_an_efficiency_derived_from_a_last_request_rate, with and without a rate). The legend is now Lemonade: last req tok/s, not live, short enough to join the held legend inside 80 columns (the joined-form test renders at 80), and its wording is pinned. table_has_no_last_request_legend_without_a_rate_to_explain holds the finiteness half of any_last_req; table_labels_a_last_request_rate_and_only_then holds the flag half.
| | [`test_http.rs`](../engines/lemonade/src/test_http.rs) | Test-only: a loopback HTTP fixture that answers each request with a chosen status and JSON body and records what it received, plus (Linux) fake `lemond` and `lemonade` executables that log their arguments. | flat | | ||
| | [`test_support.rs`](../engines/lemonade/src/test_support.rs) | Test-only fixtures shared by the crate's module tests. | flat | | ||
|
|
||
| Boundaries: |
There was a problem hiding this comment.
(The sentence is at 675 — anchoring to the nearest line in the hunk.)
This change inverts the import direction the bullet describes, so both halves are now false. Verified at both revisions:
- merge base
691834a04:llamacpp_backend.rs:10haduse crate::direct_llama::find_binary_in; - this head: that import is gone (
find_binary_inmoved intollamacpp_backend.rs), anddirect_llama.rs:15now hasuse crate::llamacpp_backend::{backend_server_dirs, find_llama_server_binary_for_backend};
So llamacpp_backend.rs no longer uses direct_llama.rs, direct_llama.rs imports it back, and the new router_serve.rs isn't in the chain at all. cargo xtask check-architecture-doc only checks the row exists — AGENTS.md §6 is explicit that "keeping an entry's description true is still on you".
A few other entries went stale in the same way and are worth a sweep while you're in here:
:398efficiency.rsstill saysnode_throughputis "Σgen_tps"; it's nowlive_gen_tps(), which is the behaviour change most likely to surprise a reader of that row:436lemonade.rsstill says "discovery of a running server" for a collector that now emits one row per loaded model:555services_manager.rsstill claims "live throughput"crates/rocm-dash-core/src/efficiency.rs:129andcrates/rocm-dash-tui/src/ui/services_manager.rs:8restate the same stale rules in doc comments
There was a problem hiding this comment.
Fixed in b8dc364. The layering bullet now matches the imports at this head (backend_alignment to llamacpp_backend to process; direct_llama uses both; router_serve uses process for its types). I also swept the entries you listed: efficiency.rs (live gen_tps, and instance_tokens_per_watt), lemonade.rs (one row per loaded model), services_manager.rs, and the two doc comments.
| model_names: vec![target.model_ref.clone(), target.canonical_model_id.clone()], | ||
| }); | ||
| } | ||
| managed_lemonade_collectors.retain(|endpoint, _| groups.contains_key(endpoint)); |
There was a problem hiding this comment.
Three new branches in this block have no test. I confirmed each by breaking it and running the full daemon suite — all green in every case.
1. This retain (line 776). It's the only bound on managed_lemonade_collectors, which holds a reqwest Client and its connection pool per (host, port) ever seen. Deleting the line leaves 151 tests green. A long-lived daemon whose managed services rebind ports accumulates them for the process lifetime.
2. The scrape_now && half of the stats gate (line 839). Removing it is green, because every Lemonade test sets instance_tick: TICK, making scrape_now always true. It's live under shipped defaults though: discovery_tick 5 s and instance_tick 2 s give discovery_ticks = 5, instance_ticks = 2, so ticks 5, 15, 25 are discovery-only and must not fetch stats. One case with instance_tick = 2 * TICK and discovery_tick = TICK would pin it.
3. The partial-registry filter (line 679). It keeps any previous ManagedLemonadeTarget whose id isn't in disc.seen — but a service this same pass read and found stopped never enters seen, it enters retired. ManagedDiscovery::retired exists for exactly this and keep_through_partial_registry already applies it to managed_roots; the new lemonade vector didn't get the same rule. Adding && !disc.retired.contains(&target.service_id) leaves everything green either way, so neither behaviour is held. Beyond wasted polls: if anything later binds that host:port, the stale row groups with the live one, two rows match the health model, attributed_id collapses to None, and the live service loses its throughput.
For 3, moving the filter into keep_through_partial_registry beside its sibling would both fix it and give it a seam a unit test can reach — right now it's inline in run_loop where there isn't one.
Unrelated but in the same block: attributed_id.expect("stats require attribution") at ~900 is held off only by the condition two statements earlier, which the tuple type can't express. It's unreachable today, but a panic there kills run_loop and every snapshot with it. Returning Option<(String, Result<InstanceSample, _>)> would make it unrepresentable.
There was a problem hiding this comment.
In b8dc364: (3) the partial-registry filter moved into DiscoveryState::keep_lemonade_through_partial_registry and now drops a target the pass read and found stopped (a_partial_registry_read_keeps_unseen_lemonade_targets_and_drops_stopped_ones); (2) the scrape_now half of the stats gate is held by lemonade_stats_are_fetched_on_the_instance_cadence_not_every_discovery_tick (discovery every tick, instance scrape every fifth; it fails with the half removed, 14 stats for 15 health requests); the expect is gone, since the stats now travel with the id attribution picked. Not changed: (1) the retain on the collector map. The map lives inside run_loop and holds only HTTP clients, so nothing observable depends on it, and I would have to expose it just to test it; the retain stays untested.
| parent: &ParentRuntimeEnvironment, | ||
| ) -> Result<LemondChild> { | ||
| #[cfg(windows)] | ||
| if let Some(log_path) = log_path { |
There was a problem hiding this comment.
(The &[] I mean is the env_overrides argument to spawn_hidden_console_with_log at ~86.)
This Windows branch returns before lemonade_process_environment_vars is ever called, and passes an empty env_overrides. It also ignores the parent argument this change added to the signature.
So on Windows lemond starts without LEMONADE_API_KEY, HIP_VISIBLE_DEVICES, ROCM_PATH or the prepended runtime PATH — and log_path is always Some for a managed serve, so this is the normal path, not an edge.
The part that makes this worth raising on a changed function rather than leaving as pre-existing is the comment further down this file, which says translating the key here "both secures the server and keeps our readiness checks authenticated". On Windows only the second half holds: rocm serve --require-api-key would report a protected endpoint while lemond serves it unauthenticated, and the readiness probe still passes because wait_for_lemonade_cli_status_with_parent applies the vars itself. --gpu pinning is dropped the same way.
I couldn't test this — no Windows host — so it's a code reading. Either build the vars in this branch and pass them through (env_overrides needs widening to OsString values first), or refuse to start when a key is configured and can't be delivered. Either way the comment needs correcting.
There was a problem hiding this comment.
Not changed here. This Windows branch is unchanged by this PR: origin/main has the same call with an empty env_overrides, so lemond already starts without the translated key, HIP_VISIBLE_DEVICES and the runtime PATH there. You are right that it matters (a configured API key leaves the server open while the readiness probe, which applies the variables itself, still passes). I would rather fix it in its own change, since it widens spawn_hidden_console_with_log's environment type in rocm-core and needs a Windows run to check the key reaches the server. I have not tested it either.
| /// non-ROCm backend notice) through another; without `O_APPEND` the child's next | ||
| /// write would land at its own offset, on top of what was appended. | ||
| fn open_service_log(log_path: &Path) -> Result<fs::File> { | ||
| fs::File::create(log_path) |
There was a problem hiding this comment.
fs::File::create truncates, and this runs for every lemond spawned — including the short-lived one ensure_model_in starts to pull a model, which is handed request.log_path, the same service log.
That's new behaviour on the fallback path. At the merge base a router failure went straight to resolving the GGUF from cache with no spawn; this change inserts ensure_model_in after the router failure, so a fallback that has to download wipes the router's stdout and stderr — exactly the diagnostics explaining why the fallback is happening. The reason line is appended afterwards, so the failure looks explained while the load error that caused it is gone.
A fresh: bool parameter, truncating only for the serve's own server, would do it.
While you're in this file: stop_server in router_serve.rs sends a single SIGTERM to the lemond PID and then waits unbounded — terminate_pid's force parameter is _force and discarded, so the true at the call site does nothing. rocm_core::terminate_process_tree exists for this and its own doc names the consequence ("leaves those workers reparented to init, where they keep the model resident and the device memory pinned"); vLLM and the managed stop path both use it. That's inherited rather than introduced, but this change adds the first path that kills the router and immediately launches a second GPU server on the same port and device — and the likeliest trigger of a RouterLoadFailure is an allocation failure, which is the case where an orphaned backend still holds VRAM.
There was a problem hiding this comment.
Fixed in b8dc364: LogMode. The serve's own server starts the log over; the helper lemond that pulls a model appends, so a fallback that downloads keeps what the router wrote (a_helper_server_keeps_the_log_it_is_given). The router's stop_server now uses terminate_process_tree, as the vLLM and managed stop paths do, so a backend that still holds device memory is signalled with it; on Windows this is the single-process terminate it was. That change has no unit test, since it needs a real process tree. I left terminate_pid's ignored force parameter alone; other callers use it.
| Scenario: dash-41 - Home counts a still-loading model as running and shows its stage | ||
| Given a managed model is still loading | ||
| When the user opens the dashboard | ||
| Then the Home view counts the managed model as running and shows it as loading |
There was a problem hiding this comment.
dash-41 lost its two closing steps here and I can't find a reason for it anywhere:
- When the user quits the dashboard
- Then the dashboard exits successfully
Every neighbouring dash scenario keeps the pair, the commit message doesn't mention it, and no comment was added beside the scenario. AGENTS.md §8: "Deleting or weakening a test is a change in its own right... say in the commit why."
I checked the obvious explanation and it doesn't hold — the seeded record is starting/loading and InstanceStatus::is_serving() is Ready | Running only, so the new ownership gate can't have opened the quit-confirm prompt here. Which leaves this as the one dash scenario with a live daemon that never quits the TUI, in a change that edits the quit-confirm path.
Restoring the two steps, or a one-line comment saying what made them wrong, would settle it either way.
There was a problem hiding this comment.
Restored in b8dc364. I found no reason for dropping the two closing steps of dash-41; it was a mistake. It passes in WSL with them.
juhovainio
left a comment
There was a problem hiding this comment.
Requesting changes on process grounds; I found no correctness defect in the engine-side code I read.
Blocking
- Merge conflict. The PR is CONFLICTING against main. It needs a rebase, and the current green CI predates that rebase, so it needs to be re-run on the rebased head.
- Scope. This bundles four-plus logical changes in roughly 16.6k diff lines (engine/CLI routing and GGUF resolution, the direct-llama fallback and download, and the dash Lemonade discovery/telemetry/Services work). AGENTS.md asks for one logical change per PR. Please split, at minimum engine/CLI changes from dash changes, so each can be reviewed and reverted independently.
Non-blocking
- Two inline comments on the default-on external probe (timeouts read as "absent"; one failed probe retires every external row).
- The fallback download runs with no size cap or confirmation. The PR body discloses this, but please consider a cap or an explicit prompt.
enable_lemonadenow defaults to true, sorocm dashprobes 127.0.0.1:13305 on Linux/WSL. It is read-only and documented, but it deserves a release-note line.- Test gaps: no scenario covers split-GGUF resolution or the router-to-direct fallback; the Linux GPU scenarios did not run here; the managed-inference journey scenario has a flaky expectations row, so it is not coverage. The PR body shows no evidence of the break-the-target mutation checks AGENTS.md asks for.
What I checked
- The router-to-direct fallback is GPU-safe (no silent CPU fallback) and its state handling holds up.
- The dash-side claims in the PR body (managed ownership, 401/403 handling for managed vs external endpoints, throughput exclusion from node aggregates) match the diff.
- CI is green on 615eb39.
- Not reviewed in depth: the e2e step definitions,
pane.rs/serving.rs/instance_detail.rsin the TUI, and the tests inrunner/discovery.rs.
| /// request timeout. | ||
| fn send_error(url: &str, error: &reqwest::Error) -> CollectorError { | ||
| let message = format!("GET {url}: {error}"); | ||
| if error.is_connect() || error.is_timeout() { |
There was a problem hiding this comment.
Any reqwest timeout is mapped to Io(ConnectionRefused), which is_connect_error reads as "nothing is listening". A live but slow server (e.g. lemond busy loading a model) is therefore treated as absent: silent on first contact, and on a known endpoint it falls into the retire-all-rows path. The comment justifies this by Windows closed-port hangs, but the feature is on by default for Linux/WSL where a refused connection is immediate. Consider scoping the timeout-to-absent mapping to Windows, or keeping rows through one failed probe.
There was a problem hiding this comment.
Not changed. A probe that times out reading as absent is the intended rule: on Windows a closed port can hang rather than refuse, and on Linux a busy lemond that does not answer within the bound is also not usable as a serving row. The result is a brief disappearance, not a wrong figure, and the row returns on the next good probe. I did not scope it by platform because the behaviour would then differ between Linux and Windows hosts for the same server. Keeping a known row through one failed probe is the same decision as on the runner thread.
| && rocm_dash_collectors::lemonade::is_connect_error(&error); | ||
| lemonade_discovery_warn = | ||
| (!absent).then(|| endpoint.warning("health", &error)); | ||
| (Some(Vec::new()), Vec::new(), None, None) |
There was a problem hiding this comment.
One failed external discovery probe (including a single timeout) yields an empty service set, which reconcile_lemonade turns into retire() of every row: telemetry is purged, InstanceGone is broadcast, and the rows are re-announced as new on the next good probe. A transient blip therefore flaps rows and resets the rate baseline. Managed targets get a hold on failure; external ones do not. Consider holding rows for N failed discovery passes, or retiring only on connect-refused/auth errors.
There was a problem hiding this comment.
Not changed here. A failed discovery pass retiring the external rows is the specified behaviour: @id:dash-lemonade-external-protocol-transitions asserts that models disappear when the server stops responding and return when it answers, and a hold of N passes would change what that scenario says and how long a dead server stays on screen. The cost you describe is real (the rows are announced again and the rate baseline restarts after a blip), and managed targets do hold. If you want external rows held through a pass or two, I would do it as its own change with the scenario updated to match; say so and I will.
…dels in dash Resolve registered GGUF models from the installed Lemonade catalog instead of falling back to the default model. Simple entries (one checkpoint with a variant, no extra files) are served directly by the selected GPU llama-server; anything else keeps the Lemonade router. On Linux a failed router load or verification falls back to the selected GPU backend's llama-server, never to a missing or CPU backend. Split (sharded) GGUF entries resolve to their first shard, and a complete set wins over an older partial download. Persist the name that is actually served, route complex catalog entries through Lemonade, and restrict lifecycle actions to registry-owned services. A non-ROCm GPU backend (Vulkan) is named in the serve output and log, on both routes, only once the model is actually serving. rocm dash discovers loaded models from a local Lemonade server on Linux and WSL (read-only, configurable host and port, opt-out), groups equivalent endpoints and never adopts an external service. A Lemonade rate is the last completed request's rate: it is labelled, left out of the node-wide tok/s and tok/W figures, and flagged in the agent snapshot. rocm services stop also prints the service id and status for the in-process stop report. Add routing, ownership, telemetry and user-journey regressions, including scenarios for the registered-name routes, external discovery, and the reinstall advice for a missing llama-server. Signed-off-by: Eugene Volen <eugene.volen@amd.com>
615eb39 to
b8dc364
Compare
|
Thanks for the thorough review. All twelve inline points have a reply on their threads; the fixes are in b8dc364, rebased onto current main. Not changed
Your two design questions
Verification is in the description: Windows and WSL, with the WSL signing workaround and the GPU scenarios that could not run here (the managed GPU journey needs about 25 GiB of disk; this machine has 17 GiB free). The GPU Linux lanes have not reported for this head yet. |
Addressed in b8dc364 and every inline point has a reply (#631 (comment)). The Windows lemond environment and the untested collector-map retain are deliberately not changed, with reasons on their threads.
|
Thanks for the review, answered against b8dc364. Blocking
Non-blocking
|
Rebased onto main with no conflict (CI re-running on b8dc364); both inline points have replies, and the PR stays one change by the author decision, explained at #631 (comment) and in the follow-up comment.
juhovainio
left a comment
There was a problem hiding this comment.
I read through the production changes in the Lemonade engine, the dash daemon/collector/core, the TUI quit/services paths and apps/rocm, and found nothing blocking. The behavior described in the PR body (registered GGUF resolution, router-to-direct fallback limits, dash discovery and ownership, quit-prompt gating) matches the diff.
A few non-blocking notes, left inline where they anchor to a line:
is_external_lemonadekeys on the container name, so an unrelated Docker container namedlemonadewould be misclassified.run_rocm_boundedduplicatesrun_rocmand polls with a blocking sleep on the executor thread.- The dash-52 row is marked flaky, so that journey can't fail its lane; worth a tracked follow-up to remove the flag once the serve is stable.
- The PR bundles several fixes plus harness changes (the bounded e2e runner is applied to unrelated serve steps), which makes it large to review; splitting would help next time.
This is a diff-only read: I did not build or run anything. The Windows e2e lane is red on serve-01/02 (vLLM unsupported on native Windows) and serve-52 (runner download failure), which look unrelated to this change since they also fail on other PRs; the MI300X lane is still pending, and the GPU scenarios (serve-57/58) need Linux GPU runners.
| /// External Lemonade discovery uses the engine marker, not the instance id. | ||
| /// This identifies provenance independently of the current serving status. | ||
| #[must_use] | ||
| pub fn is_external_lemonade(&self) -> bool { |
There was a problem hiding this comment.
is_external_lemonade identifies an external Lemonade by container_name == "lemonade" plus !managed. An unmanaged Docker-discovered container that happens to be named lemonade matches and gets reported as an external Lemonade runtime (and drives the Serving engine-status line). An explicit source/engine marker on Instance would be sturdier than a display name, with a test for a Docker row named lemonade.
There was a problem hiding this comment.
Agreed, and it is the same point r0x0r raised as a typed source field in place of managed: bool plus the name check. Not changed in this PR: it is a change to Instance and every renderer, so it should be its own change. No test covers a Docker row named lemonade today.
| /// child writes to files, so an inherited handle cannot keep us waiting, and the | ||
| /// child is killed once the deadline passes. A timeout is reported as rc `-1` | ||
| /// with a note appended to stderr. | ||
| pub fn run_rocm_bounded( |
There was a problem hiding this comment.
run_rocm_bounded duplicates run_rocm's spawn/isolation/recording path and polls with a blocking std::thread::sleep on the async executor thread. Sharing a helper would keep the two from drifting (env setup, diagnostics); if the blocking 200ms poll is intentional, a comment saying so would help.
There was a problem hiding this comment.
Not changed here. You are right that the two share a spawn, isolation and recording path and could drift; folding them into one helper is worth doing, and I did not attempt it in this PR. I have not checked whether the blocking poll is intentional, so I will not claim it is.
| serve_timeout_secs = 90 | ||
| flaky = true | ||
|
|
||
| # The dashboard journey does the same real lemonade serve as serve-lemonade-inference, |
There was a problem hiding this comment.
dash-52 is flaky = true (ROCMAI-898), so the only real-GPU managed-Lemonade dashboard journey can't fail its lane. Please keep a tracked follow-up so the row is removed once the serve is stable.
There was a problem hiding this comment.
The row names ROCMAI-898, the open ticket for the Linux managed-serve instability; the flag should come off when that is fixed. Not changed here.
#963 fixed Lemonade serving on Linux by resolving registered model names, but the ROCMAI-898 expected-failure rows stayed in expectations.toml. Every one of them is flaky, so their passes were tolerated as XPASS and never surfaced as stale rows. The lanes have been passing these scenarios since. In the main push and merge-queue E2E runs from #963's merge (2026-11-09) to 2026-11-13, the Strix Halo Ubuntu jobs passed every removed scenario that ran, MI300X passed serve-07 and serve-40 (as it also did on this PR's own MI300X job), and the W7900 jobs in that window passed them too. The dash-lemonade-managed-inference-journey row, added by #631, has passed in all seven Strix Halo Ubuntu jobs since that change merged. Remove twelve rows: the default-engine, readiness-contract, Anthropic inference/tool-use/streaming, explicit Lemonade inference (serve-07) and background-helper (serve-40) serving scenarios, the two chat scenarios, the bench real-serve scenario and the dash managed-inference journey. Keep serve-50 and serve-52 as flaky rows, re-attributed to ROCMAI-1023: their serve becomes ready, but `rocm services list` intermittently shows no `anthropic endpoint:` row within the 30s poll. They set no serve-timeout override, since the serve itself succeeds. Add a guard test that resolves each of the twelve scenarios from its real tag line on the modelled Linux Lemonade hosts (Strix Halo Ubuntu, a WSL2 guest with a usable GPU, gfx1100 bare metal) and, for the ids pinned to Lemonade, on MI300X, failing if any row matches; the pin is held per id and the failure message says how to proceed. Tests that used the deleted rows as fixtures now parse inline matrices. Update docs/testing.md and a model_serving.feature comment that still described the old rows. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Why
On Linux a registered Lemonade model name (for example
Qwen3-0.6B-GGUFor an alias) could not be served directly, a failed router load left servers and state behind,rocm dashknew nothing about a Lemonade server it did not start, and the dashboard showed a Lemonade last-request rate as live throughput. This change fixes those four together because they share one model-resolution and ownership model.Summary
startingwith no server pid, so a reader never records the service as failed while it is still running, andstopnever targets the dead router.model_ref, andcanonical_model_idholds the router name.rocm dashon Linux and WSL, with configurable host and port and an opt-out. Show every loaded model, remove unloaded or unreachable models, and group equivalent endpoints without adopting external services. A managed service bound to::is matched against both loopback families. An already-known external endpoint that starts answering 401 or 403 is reported on the scrape-only tick that finds out.s(open Services) only for an instance the overlay lists; for one started a moment earlier, before the daemon's next broadcast marks it managed, it prints therocm services stop <id> --yescommand. External Lemonade models and Docker-discovered containers stay visible in Observe but are not offered for stop or restart.Lemonade: last req tok/s, not live), the instance detail and quick preview readlast req, and an overlay row readsreq <rate>instead ofgen <rate>so it keeps the width ofgen. A row's tok/W is not computed from that rate either. The node-wide totals (the tok/W and tok/s heroes on Home and Observe, the Hardware efficiency line) leave a last-request rate out, through one shared aggregate, so an idle server does not show its last rate as current. The instance detail's recent-throughput panel does not list it as activity. The agent snapshot tools carrygen_tps_last_request.User-visible changes
rocm services stop <id>(and the stop that follows Ctrl-C on an attached serve) printsservice: <id>andstatus: <status>lines underLocal server stopped, next tostopped processes: N.rocm engines install lemonade --reinstallis the advice printed when the selected backend'sllama-serveris missing.dashboard.daemon.enable_lemonadedefaults totrue, so on Linux and WSLrocm dashprobes127.0.0.1:13305read-only unless the key isfalse.--enable-docker. The Engines pane reports Lemonade as "managed runtime installed", "managed runtime not installed", "externally running" or "unknown, runrocm engines list", from the managed inventory and external availability separately.Startinginstance toReady; status comes from the registry and health.GpuSystemInfo.lemond_version(recordings that still carry the key load), and the collector functionslemonade_instance,parse_health_model,fetch_health_modelandLemonadeCollector::discover, replaced by per-model discovery from the health payload.Deployment notes:
s.rocm dashembeds its daemon from the same binary, so this arises only for a separately started older daemon.rocm dashstartup reads the managed Lemonade inventory through the in-process Detect handler: a read of the install manifest and the local GPU probe, with no child process and no network.Regression coverage
Serving tests cover registered aliases, unsafe metadata, complex catalog entries, the pinned catalog entries, cache-root priority, the router route decision, the persisted served name, the pull command line, and failure states. Fallback tests cover a router load failure that falls back to the selected GPU backend, a non-load error, an unreachable verification probe, a non-Linux host, and a missing or CPU backend; each asserts the result with the state or log, and the state test also checks that the router's pid is cleared. The direct server's readiness, chat smoke, and exit paths are covered. Dashboard tests cover ownership, endpoint grouping, rate attribution and reset, scrape-tick holds, takeover of an external endpoint by a managed record, row order for models that share a name, and the shared node aggregate leaving out a last-request rate on Home, Observe and in the efficiency figure.
Scenarios (which lane runs what):
@id:dash-lemonade-external-protocol-discoveryand@id:dash-lemonade-external-protocol-transitionsuse an in-process protocol fixture, not real inference. The fixture records any request that is not a GET of health or stats, and the scenarios assert none arrived and that no managed record was created. They need no GPU.@id:dash-managed-service-visibleis extended with the ownership check on the Services overlay.@id:dash-lemonade-managed-inference-journeyneeds Linux, a GPU, and Lemonade. It has a bounded serve and a flaky expectations row, like the other Lemonade serve scenarios on Linux, so it cannot fail its lane and does not yet count as coverage of that journey.@id:serve-lemonade-complex-registered-inferenceserves the smallest catalog entry that needs a vision projector and checks that the router loaded it. It is@requires-gpu,@requires-os:linuxand@nightly.@id:serve-lemonade-simple-registered-direct-routeserves the simple registered entryQwen3-0.6B-GGUFon its own port and reads the engine state:load_response.methodmust belemonade-packaged-llama-serverand thellama-serverthat served must belong to the backend Lemonade selected. It then runs inference, stops the service, and asserts the service id in the stop output. It needs@requires-gpu,@requires-os:linuxand@requires-engine:lemonade, and has no flaky expectations row, so a route regression fails the lane.Startingrow stayingStartingon a successful scrape (a temporary promotion made it fail), the instance detail keeping its freshness line at every width from 80 to 200 columns (pinning the block to four rows made it fail), and the stats cadence (removing thescrape_nowhalf of the gate made it fail). The rest of the new tests were not mutation-checked.@id:service-stop-yes-stops-a-running-serviceand@id:service-stop-cli-confirms-a-stop-and-clears-the-recordassert the newservice:andstatus:lines on a default lane.@id:runtime-lemonade-backend-alignment-reportedtests the reinstall advice: it deletes the selected backend'sllama-server, reinstalls, and requires the binary to be back. It runs only on the Windows nightly GPU lane.direct_llama.rs,lib.rsandrouter_serve.rs;@id:serve-lemonade-complex-registered-inferenceremains the hardware check of the router route.Verification
Run locally on a Windows 11 host (Strix Halo) and in WSL2 Ubuntu.
cargo fmt --check, workspacecargo clippy --all-targets -- -D warnings,cargo clippy -p e2e-cucumber --test e2e -- -D warnings,cargo xtask check-module-size,cargo xtask check-architecture-doc,cargo xtask smoke, andcargo nextest runfor the affected crates (rocm,rocm-core,rocmd, the dash crates,rocm-engine-lemonade,e2e-cucumber,xtask): 6438 passed on the tree before the last two small fixes (a doc comment and the wording a Linux-only test pins), and 2205 passed for the lemonade, dash and e2e-cucumber crates on the final tree.cargo clippy -p e2e-cucumber --test e2e,cargo xtask smokeand the same affected crates on the final tree: 6851 passed. The scratch git repos twobuild_metadatatests create could not sign, because this WSL has no SSH signing key, so that run setcommit.gpgsign=falsethroughGIT_CONFIG_*. The scenariosdash-41,dash-53,dash-54,service-stop-01andservice-stop-06pass there, the last two being the default-lane stop scenarios that assert the newservice:andstatus:lines.cargo xtask coverage, and the GPU scenarios below.@id:dash-lemonade-managed-inference-journeyneeds a download of about 25 GiB and this machine has 17 GiB free, so it fails at the serve step here.Not verified:
@id:serve-lemonade-simple-registered-direct-route,@id:serve-lemonade-complex-registered-inference,@id:dash-lemonade-managed-inference-journey) were not run locally, and neither was the Windows nightly@id:runtime-lemonade-backend-alignment-reported. Real GPU inference through the new serving paths was not run locally.@id:dash-lemonade-external-protocol-transitionsflaky on hosted CI, so it is not part of this change.lemondbeside the router's own, and the directllama-serveris launched with--n-gpu-layers 999, so a GPU out-of-memory is a hard failure rather than a partial offload. Both follow from the GPU-required rule.terminate_process_tree) and a helperlemondappends to the service log instead of truncating it; the log behaviour has a unit test, the stop has none because it needs a real process tree.lemondspawn passes no environment (no translated API key, device pinning or runtimePATH); that is unchanged from main and left for its own change.Safety and scope
External discovery never creates managed service records or stops external processes. Tests use isolated roots and owned endpoints. Native Windows live dashboard support and general damaged-runtime repair are outside scope.
router_serve.rs, and the runner's discovery state intorunner/discovery.rs.