Skip to content

fix(lemonade): resolve registered GGUF models and discover serving models in dash - #631

Merged
volen-silo merged 1 commit into
mainfrom
worktree-lemonade-serving-discovery
Nov 16, 2026
Merged

volen-silo merged 1 commit into
mainfrom
worktree-lemonade-serving-discovery

Conversation

@volen-silo

@volen-silo volen-silo commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Why

On Linux a registered Lemonade model name (for example Qwen3-0.6B-GGUF or an alias) could not be served directly, a failed router load left servers and state behind, rocm dash knew nothing about a Lemonade server it did not start, and the dashboard showed a Lemonade last-request rate as live throughput. This change fixes those four together because they share one model-resolution and ownership model.

Summary

  • Resolve registered GGUF models from the installed Lemonade catalog instead of falling back to the default model. Only simple entries are served directly: one checkpoint with a variant, and no extra files or options. Anything else stays with the Lemonade router, including entries that need a vision projector. The direct path validates checkpoint metadata, uses the selected GPU backend, requires the requested alias at readiness (an exact model-name match), and keeps download and load errors.
  • Keep a Linux fallback from a failed router model load or load verification to a direct llama-server, with two limits. It runs only when the registered model resolves to a safe checkpoint, and it uses the llama-server of the GPU backend that Lemonade selected, never a missing or CPU backend. A verification probe that cannot reach the endpoint is a plain failure and does not start the fallback or its download. Every other error after a serve starts returns its own error, stops a spawned server, and marks the engine state failed, for the router and the direct route. While the fallback runs, the router's server is stopped first and the state says starting with no server pid, so a reader never records the service as failed while it is still running, and stop never targets the dead router.
  • State and health use the name that is actually served. The requested text stays in model_ref, and canonical_model_id holds the router name.
  • A registered entry whose GGUF is split into shards resolves to the first shard, and a complete split model wins over an older partial download. The CLI readiness check accepts a router that selected the Vulkan backend as well as ROCm.
  • Enable read-only local Lemonade discovery in rocm dash on Linux and WSL, with configurable host and port and an opt-out. Show every loaded model, remove unloaded or unreachable models, and group equivalent endpoints without adopting external services. A managed service bound to :: is matched against both loopback families. An already-known external endpoint that starts answering 401 or 403 is reported on the scrape-only tick that finds out.
  • Lifecycle ownership comes only from the managed service registry. The Services overlay lists only managed services. The quit prompt offers s (open Services) only for an instance the overlay lists; for one started a moment earlier, before the daemon's next broadcast marks it managed, it prints the rocm services stop <id> --yes command. External Lemonade models and Docker-discovered containers stay visible in Observe but are not offered for stop or restart.
  • Separate managed runtime installation from external availability. Leave throughput unavailable when last-request metrics cannot be attributed to one loaded model. A managed direct llama-server row stays visible with no throughput and no standing warning, and so does a managed Lemonade server that needs an API key (health answers 401 or 403); no credentials are read or sent, and external discovery still warns on those statuses.
  • When the selected backend is not ROCm (for example Vulkan), the serve output and the serve log name that backend and say it is not ROCm, on both the router and the direct route, and only once the model is actually serving. CPU is still refused.
  • A Lemonade rate is the rate of the endpoint's most recent completed request, not live throughput. The Instances table and the Services overlay show a legend (Lemonade: last req tok/s, not live), the instance detail and quick preview read last req, and an overlay row reads req <rate> instead of gen <rate> so it keeps the width of gen. A row's tok/W is not computed from that rate either. The node-wide totals (the tok/W and tok/s heroes on Home and Observe, the Hardware efficiency line) leave a last-request rate out, through one shared aggregate, so an idle server does not show its last rate as current. The instance detail's recent-throughput panel does not list it as activity. The agent snapshot tools carry gen_tps_last_request.
  • Update command help, README, architecture, testing, and manual-testing documentation.

User-visible changes

  • rocm services stop <id> (and the stop that follows Ctrl-C on an attached serve) prints service: <id> and status: <status> lines under Local server stopped, next to stopped processes: N.
  • rocm engines install lemonade --reinstall is the advice printed when the selected backend's llama-server is missing.
  • dashboard.daemon.enable_lemonade defaults to true, so on Linux and WSL rocm dash probes 127.0.0.1:13305 read-only unless the key is false.
  • The Instances table's empty state reads "no serving models discovered · open Serving to launch a managed model or check an external endpoint" instead of advising --enable-docker. The Engines pane reports Lemonade as "managed runtime installed", "managed runtime not installed", "externally running" or "unknown, run rocm engines list", from the managed inventory and external availability separately.
  • A successful Lemonade stats scrape no longer promotes a Starting instance to Ready; status comes from the registry and health.
  • Removed API: GpuSystemInfo.lemond_version (recordings that still carry the key load), and the collector functions lemonade_instance, parse_health_model, fetch_health_model and LemonadeCollector::discover, replaced by per-model discovery from the health payload.

Deployment notes:

  • Ownership defaults to unmanaged, so recordings made before this change, and a dashboard attached to a daemon from an older build, show no managed services in the Services overlay and the quit prompt never offers s. rocm dash embeds its daemon from the same binary, so this arises only for a separately started older daemon.
  • rocm dash startup reads the managed Lemonade inventory through the in-process Detect handler: a read of the install manifest and the local GPU probe, with no child process and no network.

Regression coverage

Serving tests cover registered aliases, unsafe metadata, complex catalog entries, the pinned catalog entries, cache-root priority, the router route decision, the persisted served name, the pull command line, and failure states. Fallback tests cover a router load failure that falls back to the selected GPU backend, a non-load error, an unreachable verification probe, a non-Linux host, and a missing or CPU backend; each asserts the result with the state or log, and the state test also checks that the router's pid is cleared. The direct server's readiness, chat smoke, and exit paths are covered. Dashboard tests cover ownership, endpoint grouping, rate attribution and reset, scrape-tick holds, takeover of an external endpoint by a managed record, row order for models that share a name, and the shared node aggregate leaving out a last-request rate on Home, Observe and in the efficiency figure.

Scenarios (which lane runs what):

  • @id:dash-lemonade-external-protocol-discovery and @id:dash-lemonade-external-protocol-transitions use an in-process protocol fixture, not real inference. The fixture records any request that is not a GET of health or stats, and the scenarios assert none arrived and that no managed record was created. They need no GPU. @id:dash-managed-service-visible is extended with the ownership check on the Services overlay.
  • @id:dash-lemonade-managed-inference-journey needs Linux, a GPU, and Lemonade. It has a bounded serve and a flaky expectations row, like the other Lemonade serve scenarios on Linux, so it cannot fail its lane and does not yet count as coverage of that journey.
  • @id:serve-lemonade-complex-registered-inference serves the smallest catalog entry that needs a vision projector and checks that the router loaded it. It is @requires-gpu, @requires-os:linux and @nightly.
  • @id:serve-lemonade-simple-registered-direct-route serves the simple registered entry Qwen3-0.6B-GGUF on its own port and reads the engine state: load_response.method must be lemonade-packaged-llama-server and the llama-server that served must belong to the backend Lemonade selected. It then runs inference, stops the service, and asserts the service id in the stop output. It needs @requires-gpu, @requires-os:linux and @requires-engine:lemonade, and has no flaky expectations row, so a route regression fails the lane.
  • Mutation checks run against the new tests: the managed Starting row staying Starting on a successful scrape (a temporary promotion made it fail), the instance detail keeping its freshness line at every width from 80 to 200 columns (pinning the block to four rows made it fail), and the stats cadence (removing the scrape_now half of the gate made it fail). The rest of the new tests were not mutation-checked.
  • @id:service-stop-yes-stops-a-running-service and @id:service-stop-cli-confirms-a-stop-and-clears-the-record assert the new service: and status: lines on a default lane.
  • @id:runtime-lemonade-backend-alignment-reported tests the reinstall advice: it deletes the selected backend's llama-server, reinstalls, and requires the binary to be back. It runs only on the Windows nightly GPU lane.
  • No scenario covers split (sharded) GGUF resolution or the router-to-direct fallback. Split resolution needs a real multi-shard checkpoint (several GiB) or a staged Lemonade runtime that the harness cannot fabricate, and forcing a real router load failure has no safe hook. Both are covered by unit tests in direct_llama.rs, lib.rs and router_serve.rs; @id:serve-lemonade-complex-registered-inference remains the hardware check of the router route.

Verification

Run locally on a Windows 11 host (Strix Halo) and in WSL2 Ubuntu.

  • Windows: cargo fmt --check, workspace cargo clippy --all-targets -- -D warnings, cargo clippy -p e2e-cucumber --test e2e -- -D warnings, cargo xtask check-module-size, cargo xtask check-architecture-doc, cargo xtask smoke, and cargo nextest run for the affected crates (rocm, rocm-core, rocmd, the dash crates, rocm-engine-lemonade, e2e-cucumber, xtask): 6438 passed on the tree before the last two small fixes (a doc comment and the wording a Linux-only test pins), and 2205 passed for the lemonade, dash and e2e-cucumber crates on the final tree.
  • WSL2: workspace clippy, cargo clippy -p e2e-cucumber --test e2e, cargo xtask smoke and the same affected crates on the final tree: 6851 passed. The scratch git repos two build_metadata tests create could not sign, because this WSL has no SSH signing key, so that run set commit.gpgsign=false through GIT_CONFIG_*. The scenarios dash-41, dash-53, dash-54, service-stop-01 and service-stop-06 pass there, the last two being the default-lane stop scenarios that assert the new service: and status: lines.
  • Not run locally: cargo xtask coverage, and the GPU scenarios below. @id:dash-lemonade-managed-inference-journey needs a download of about 25 GiB and this machine has 17 GiB free, so it fails at the serve step here.

Not verified:

  • The Linux GPU lanes (@id:serve-lemonade-simple-registered-direct-route, @id:serve-lemonade-complex-registered-inference, @id:dash-lemonade-managed-inference-journey) were not run locally, and neither was the Windows nightly @id:runtime-lemonade-backend-alignment-reported. Real GPU inference through the new serving paths was not run locally.
  • The throughput that Lemonade reports is the rate of the most recent request, and the dashboard refreshes that observation on every successful scrape. An idle server keeps showing its last rate in the Instances views, the Services overlay and its quick preview (all labelled) until a new request or the row goes away. Stopping an unchanged report from refreshing the observation made @id:dash-lemonade-external-protocol-transitions flaky on hosted CI, so it is not part of this change.
  • The Instances table marks a last-request rate only through the legend under the table, not per row, so a table that mixes live and last-request rows does not say which rows the legend refers to. No test covers a mixed table.
  • After a router failure on Linux the fallback can download a model with no size cap or confirmation, through a short-lived lemond beside the router's own, and the direct llama-server is launched with --n-gpu-layers 999, so a GPU out-of-memory is a hard failure rather than a partial offload. Both follow from the GPU-required rule.
  • Existing flaky serving expectations stay, because the historical shutdown failure is not shown fixed on hardware.
  • The router stop now signals the whole process tree (terminate_process_tree) and a helper lemond appends to the service log instead of truncating it; the log behaviour has a unit test, the stop has none because it needs a real process tree.
  • The Windows lemond spawn passes no environment (no translated API key, device pinning or runtime PATH); that is unchanged from main and left for its own change.

Safety and scope

External discovery never creates managed service records or stops external processes. Tests use isolated roots and owned endpoints. Native Windows live dashboard support and general damaged-runtime repair are outside scope.

  • Reviewed existing expected failures. None were removed without hardware evidence.
  • Reused existing domain modules. The router path moved into its own module, router_serve.rs, and the runner's discovery state into runner/discovery.rs.
  • Discovery, ownership and backend messages are asserted together with the resulting state.

@volen-silo
volen-silo requested a review from a team as a code owner October 12, 2026 08:48
@volen-silo
volen-silo requested a review from fredespi October 12, 2026 08:48
@volen-silo volen-silo changed the title Fix Lemonade GGUF serving and dashboard model discovery fix(lemonade): resolve registered GGUF models and discover serving models in dash Oct 12, 2026
rominf
rominf previously requested changes Oct 12, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · ace6ca1

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

First automated round on this PR: every finding below is new.

Review — needs work

Full review of the whole change.
No ticket named. The description claims registered-GGUF resolution, read-only dash discovery, separate install and availability status, and doc updates. Findings 1, 3 and 4 below conflict with parts of those claims.

Blocking

  • A registered llamacpp catalog entry the resolver doesn't fully understand now fails rocm serve on Linux, or is served without its extra files — engines/lemonade/src/direct_llama.rs:257-291, engines/lemonade/src/lib.rs:514-519
    resolve_direct_llama_checkpoint requires a plain string checkpoint of the form owner/repo:variant and returns an error otherwise. serve_http passes that error up with ? on Linux for every model reference. Before this change these names went to Lemonade's router.
    I ran a probe test against the head code:

    • an entry with a checkpoints map returns Err(registered model Vision-GGUF has no checkpoint);
    • an entry whose checkpoint has no variant returns Err(... needs an explicit quantization variant ...);
    • an entry with checkpoint plus a top-level mmproj returns Ok(direct), and direct_llama_server_command never passes --mmproj, so the model is served without its vision projector and nothing reports it.
      Lemonade documents the checkpoints map (main/mmproj/draft) as the current format and shows built-in models using it (custom models, API). I could not confirm the pinned 11.5.1 catalog itself, because the repo has no fixture of it. A server_models.json that can't be read or parsed also hard-fails, including for names the file wouldn't contain.
      Confidence 85 · logic · Fix: serve directly only shapes the resolver fully understands (a single checkpoint with a variant, no mmproj/checkpoints/extra checkpoint types). Return Ok(None) for everything else and for catalog read or parse errors, so the router handles them. Add a test for each shape.
  • The new serve routing decisions have no test that fails when they break — engines/lemonade/src/lib.rs:453-458, :514-519, :546-553; direct_llama.rs:297
    I mutated each of these on its own and ran cargo test -p rocm-engine-lemonade --lib; all 129 tests still passed every time:

    • disabled the bail for an HF checkpoint when no selected backend binary exists;
    • inverted the Linux check in launch_service;
    • forced direct_model to None, so registered names never take the direct path;
    • removed the backend_recipe != llamacpp guard in direct_serve_binary.
      The GPU scenarios serve-07, serve-08 and dash-25 cover only the happy path.
      Confidence 100 · logic · Fix: pull the direct / router / bail decision into a pure function (inputs: platform, parsed reference, catalog result, binary present) and unit-test each outcome, including the bail message.
  • launch_service stops normalizing the model reference on Linux, with no stated reason, and engine state can then disagree with the served name — engines/lemonade/src/lib.rs:453-458, engines/lemonade/src/state.rs:152-153
    rocm serve already passes the canonical name (serve_cmd.rs → resolve_model_response), so for that command this branch changes nothing.
    On the engine's own launch path, the state file now records the raw alias (for example qwen, or text with spaces around it) as model_ref and canonical_model_id. The router path, however, serves resolve_lemonade_model_ref(...). healthcheck_service reads the state value back, and its inference probe then uses the raw alias.
    WSL also counts as Linux here. With this branch, model-name normalization now happens in three separate places.
    Confidence 70 · logic · Fix: normalize once in launch_service as before. If the requested text must be kept, store it separately and keep canonical_model_id as the name actually served.

  • External models found by discovery can be selected for "stop"/"restart" in the Services overlay, and the approval text calls them managed — crates/rocm-dash-tui/src/ui/services_manager.rs:93-110, :169-200
    service_rows lists every daemon instance. Pressing s or r on a lemonade-<host>-<port>-model-<hex> row stages rocm services stop <id> --yes, with the text "This runs the CLI services stop for this managed service."
    The stop itself is harmless: the record lookup fails for an id that isn't in the registry, so nothing is killed. But the text is false for these rows, and the action can only fail.
    The code was already there, but this change makes discovery default-on in rocm dash, so these rows now appear on every host running Lemonade. docs/manual-testing.md works around it ("Do not use managed stop or restart actions…") instead of the code preventing it. AGENTS.md §3 says a promise that something will not happen must be tested together with the behavior.
    Confidence 85 · logic · Fix: give discovered rows an external/read-only marker. Hide or refuse lifecycle actions on them, and add a test that s on such a row stages nothing.

  • A contract test's step claims something it cannot detect — crates/rocm-dash-daemon/tests/telemetry_contract.rs (step "adding another model clears previously attributed rate immediately")
    I deleted gen_trackers.remove(&row.id) in runner.rs:897, so nothing clears the old rate. lemonade_multiple_models_keep_identity_and_do_not_share_last_request_rate still passes; it took 7.1s instead of 1.2s because it waits for the held value to expire on its own. So the step does not prove "immediately".
    For contrast, the "last access is not last request" step is a real check: breaking stats_model's single-model rule fails it at line 192.
    Confidence 100 · logic · Fix: put a time bound on that wait that is shorter than the hold window, or check the first snapshot after the health change.

  • Code left with no caller after the runner rewrite — crates/rocm-dash-collectors/src/lemonade.rs:169 (parse_health_model), :313 (fetch_health_model), :334 (discover), :117-121
    The runner now calls only discover_models and fetch_health. The other three have no non-test callers in the workspace (it is publish = false), yet their doc comments say they exist "for compatibility/legacy callers".
    In stats_model, the trailing if loaded.len() == 1 { … } else { None } can't be reached with any other length, because of the len() != 1 early return above it.
    Confidence 85 · mechanical · Fix: delete them, or make them test-only and move their tests to discover_models/fetch_health. Collapse the dead branch.

Non-blocking

  • The new bail message has no remediation, and it is wrong for some cases — engines/lemonade/src/lib.rs:546-553. The old message said "run rocm engines install lemonade". The new "no selected GPU llama-server backend X is installed" also fires when backend_recipe isn't llamacpp or the backend is cpu, where "not installed" is not the problem. Confidence 70 · mechanical · Fix: handle the recipe and cpu cases separately, and name a real command (rocm engines install lemonade --reinstall is already used in backend_alignment.rs).
  • Comments made stale by this change — Confidence 85 · mechanical · Fix: update each:
    • tests/e2e-cucumber/features/model_serving.feature:73-77 and tests/e2e-cucumber/tests/e2e/serving_steps.rs:613 cite the removed serve_hf_checkpoint and the removed "managed-lemonade fallback".
    • crates/rocm-dash-daemon/src/runner.rs:67-69 and :311-312 still say discovery is "Opt-in … off by default", but rocm dash now turns it on.
    • The registry.rs:26-31 module doc says record.host is honored only for the co-located daemon, but managed Lemonade is now scraped at record.host.
  • Lemonade HTTP calls run one after another inside the daemon tick loop — runner.rs:~678-686, ~833-925. Each external or managed endpoint gets a health request and possibly a stats request, each with a 1.5s timeout (doubled by the 404 fallback), and a new reqwest client is built per managed endpoint on every tick. A hung endpoint delays every snapshot. The vLLM path next to it uses parallel_scrape. Confidence 70 · architectural · Fix: run the endpoint futures concurrently and reuse the collectors.
  • One failed health poll removes a discovered model — lemonade.rs:323-330 (unwrap_or_default), runner.rs:~697-709. A timeout or 5xx while Lemonade is busy emits InstanceGone, and the next good poll emits InstanceDiscovered again, so the row flaps. Before, this applied to one instance and discovery was opt-in; now it applies to every loaded model by default. Health errors are logged at trace! only. Confidence 70 · logic · Fix: require several consecutive misses before removal, and report health errors through scrape_warns.
  • dash-27 ends on a step that checks nothing — tests/e2e-cucumber/features/dash.feature:372-373. After "selects engine status in the Serving view", the final Then only waits for the heading text "Serving actions". Nothing checks that "Lemonade — externally running" is gone once the endpoint is unreachable. Confidence 85 · logic · Fix: replace it with an absence check on the engine pane.
  • "External" is recognized only by string matching — crates/rocm-dash-tui/src/ui/tabs/pane.rs:393-398. The TUI matches container_name == "lemonade" plus a lemonade- id prefix against how the collector happens to build ids. Removing the starts_with half fails none of the four pane::tests, as I confirmed by running them. Confidence 70 · architectural · Fix: share a single predicate, or add an origin field to Instance; the Services-overlay fix above needs the same marker.
  • One commit holds several separate changes — AGENTS.md §11. It bundles the engine serving rewrite, a new dashboard discovery feature with config schema, a default flip, the engine-pane status, and a protocol fixture, about 1,900 inserted lines, with an empty commit body. Consider splitting the serving fix from the dash discovery work. Confidence 75 · architectural.

Decisions for the author

  • Direct llama-server is now tried first for every registered llamacpp name on Linux, and the router's load error no longer falls back to direct — tradeoff
    For: router load errors are no longer hidden, and the binary is chosen by the selected backend. Against: router-only per-model wiring (vision projector, draft and other checkpoint types, labels) is skipped whenever direct applies. Finding 1 is the correctness half of this; the question is whether bypassing the router for all registered names is intended.
  • The direct-serve binary lookup is now limited to manifest.backend_name — tradeoff
    For: a stale ROCm or Vulkan install can no longer be picked. Against: a host whose manifest says rocm but has only vulkan on disk (or the reverse) used to serve HF checkpoints and now fails.
  • rocm dash now probes 127.0.0.1:13305 by default — tradeoff
    It's documented. The cost is a poll on every discovery tick on hosts with no Lemonade, and any local listener on that port can put rows into the dashboard.

Positive signals

  • The /api → /v1 fallback happens only on a 404, never on an auth failure or 5xx, and health_http_failure_never_falls_back_to_successful_route locks that in with .expect(0) plus verify().
  • Throughput from Lemonade's server-wide "last request" stats is shown only when exactly one model is loaded, and breaking that rule fails both the unit test and the contract test, as I checked by mutation.
  • Catalog metadata is treated as untrusted: checkpoints that look like path traversal or local paths are rejected and tested, and CPU or unknown backend names are refused.
  • dash-26 checks the external fixture is still alive after the dashboard quits and that no service record was created, so the behavior is asserted, not just the wording.

Deployment notes

  • rocm dash now reads the new dashboard.daemon.enable_lemonade setting (default true) and probes lemonade_host:lemonade_port (default 127.0.0.1:13305). Existing config files pick up these defaults. A standalone daemon built from RunnerOptions::default() stays opt-in.
  • On Linux, rocm serve <registered llamacpp name> --engine lemonade now runs the packaged llama-server directly on the selected backend whenever the name resolves in the catalog, rather than going through Lemonade's router.

What this covered

Every file in the change at 34fbdfe9…ace6ca15, plus callers and siblings. Workers looked at:

  • services_manager.rs and the services stop path;
  • serve_cmd.rs and main.rs readiness, and the engine's state.rs, install.rs and backend_alignment.rs;
  • expectations.toml and the serve step definitions.

Four coupled code passes, three design questions, and the four source-reading passes (agent-instruction adherence, history, prior PRs #80/#94/#104/#242/#295/#404, code comments) ran as separate workers.

What I ran myself:

  • The unit and contract tests for the lemonade engine, the dash collectors and daemon, and the TUI pane, plus seven mutations and one probe test. Each run's test names and counts were checked.
  • The leak scan was clean.
  • At review time the Strix Halo Ubuntu (release) GPU lane had passed serve-07, serve-08 and dash-25 through dash-27.

Did not run or only partly ran:

  • The e2e suite was not run locally. The dash-27 finding is from reading the step definitions.
  • The TUI worker read event_loop.rs, app/mod.rs and instances.rs only as their one-line diffs plus call sites, not completely; it also did not read the rubric reference.
  • MI300X, Strix Halo Windows and WSL2 had not reported; Ubuntu nightly was still running.
  • The Lemonade catalog shape comes from Lemonade's public docs, not from the pinned 11.5.1 file.

Test verdicts:

  • The engine serving_* tests and the rocm-core/dash.rs config tests would fail if the production change were reverted. The routing branches inside the change are not covered (finding 2).
  • In the contract test, the "immediately clears" step passes with the clearing removed (finding 5); its other steps are real checks.
  • The pane tests do not cover the id-prefix half of the "external" check.

This report is not reconciled against the PR discussion.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from ace6ca1 to 5f26173 Compare October 12, 2026 11:34
@volen-silo
volen-silo dismissed rominf’s stale review October 12, 2026 11:35

Addressed in 5f26173: preserve router wiring for complex catalog models, persist the served identity, restrict lifecycle controls to managed services, and strengthen routing and telemetry regressions. Rebased onto current main; local workspace and dashboard protocol checks passed. Updated CI is running.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch 2 times, most recently from ac02905 to 93324b5 Compare October 13, 2026 06:53
@rominf

rominf commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator

🔴 Automated review · pr-review-watcher · ac02905

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Informational — describes ac029055, not the current head. The branch moved to 93324b5e while this round was running, so these findings were read against the older commit. Some may already be fixed, and anything the new commits changed was not reviewed. They are not attached inline for that reason, and this round files no change request: until the next round reviews 93324b5e, nothing here gates the merge. Round 2: the commit round 1 reviewed (ace6ca15) had been rewritten, so this was a full review. standing marks a finding round 1 also raised; the rest are new.

Review — needs work

Full review of the whole change.
No ticket named. The description claims the external endpoint staying alive after dashboard quit is "covered with resulting-state assertions". It is not; see Blocking.

Blocking

  • [new] The new real-GPU scenario dash-25 hangs both Strix Halo Ubuntu lanes until they are cancelled — tests/e2e-cucumber/features/dash.feature:332-344
    • What CI shows. On jobs 114933031640 (X2-LA03-S03-D02) and 114933031567 (X2-LA03-S04-D08), dash-25 started at 17:57. Its first step passed. The next log line is ##[error]The operation was canceled. at 18:33. Both lanes produced an empty report.json, so none of the Linux GPU scenarios on this PR were verified there.
    • Likely cause. dash-25 serves with a GGUF model is being served on lemonade, the same step as serve-07. Unlike serve-07 (model_serving.feature:66), it omits Given a managed runtime is active. So the serve runs against an empty, isolated runtimes tree, inside a run_rocm call with no timeout.
    • Missing expectations row. That same serve is xfail/flaky on Linux under EAI-7423 in expectations.toml (serve-lemonade-inference and siblings, serve_timeout_secs = 90). The new id has no row.
    • Confidence 100 on the hang (lane logs); 70 on the cause. · logic
    • Fix: add the runtime precondition, add an EAI-7423 row with a bounded serve timeout, and re-run the Ubuntu lanes.
  • [new] discover_models has no production caller; its three new tests check a copy of the logic the daemon never runs — crates/rocm-dash-collectors/src/lemonade.rs:292, crates/rocm-dash-daemon/src/runner.rs:800-814
    • On the base, the runner called LemonadeCollector::discover(). This change replaces it with discover_models(), which only tests call (lemonade.rs:388, 491, 512, 531, 542, 553).
    • The runner re-implements the same health → loaded_models → lemonade_service mapping inline, with different error handling: it propagates errors, while discover_models swallows them via unwrap_or_default.
    • So discovery_rejects_invalid_health_and_healthy_empty_router, discovery_recognizes_modern_loaded_model_without_legacy_field and discovery_accepts_v1_health_only_when_api_prefix_is_absent pass or fail independently of production.
    • Confidence 85 · architectural
    • Fix: extract one pure services_from_health(host, port, &LemonadeHealth) used by both, or delete discover_models and point the tests at the runner's path.
  • [new] No test covers dropping external rows when a managed record takes over the endpoint — crates/rocm-dash-daemon/src/runner.rs:723-745
    • I deleted the for gone in &lemonade_ids { … } loop and lemonade_ids.clear(). All 7 lemonade_* contract tests and every other rocm-dash-daemon test (45 lib + 9 + 16) stayed green.
    • lemonade_external_endpoint_deduplicates_managed_owner_then_reappears_after_record_removal has the managed record present from tick 1, so lemonade_ids is always empty when this branch runs.
    • Without the loop, external rows that existed before the record appeared would stay "Ready" forever next to the managed row.
    • Confidence 100 (mutation run) · logic
    • Fix: add a contract case where external rows exist first and a matching registry record then appears.
  • [standing — round 1's "routing decisions have no test", narrowed to this branch] The router-to-direct fallback removal has no test; the test added for it exercises unchanged code and is not hermetic — engines/lemonade/src/lib.rs:588-606, engines/lemonade/src/process.rs:815-856
    • The behaviour change is untested. On Linux, router load or verify failures no longer fall back to direct serving. Nothing tests that.
    • The test passes without the change. serving_gpu_model_load_process_error_is_returned drives only run_lemonade_model_load, which this diff does not change, so it passes with the production change reverted.
    • The test fails here. It fails deterministically on this host (cargo test -p rocm-engine-lemonade --lib: 134 passed, 1 failed; it also fails when run alone). The real error was preparing a runtime directory … at /home/rinflian/.rocm/data/lemonade (XDG_RUNTIME_DIR is unset …): Permission denied.
    • It writes outside its tempdir. LemonadeProcessEnvironment::default() reaches ParentRuntimeEnvironment::current(). Without XDG_RUNTIME_DIR that writes into the real user data dir; with XDG_RUNTIME_DIR set to a checkout dir the test passes. The runtime_dir.rs tests avoid this by passing an explicit parent.
    • Confidence 100 for the failure and the revert verdict; 85 for the untested behaviour · logic
    • Fix: inject the parent runtime env, or pass a LemonadeProcessEnvironment rooted in the tempdir. Then cover the no-fallback decision, for example by extracting it into a testable function.
  • [new] The "external server remains alive after quit" check cannot fail, and the description cites it as coverage — tests/e2e-cucumber/tests/e2e/dash_steps.rs:194-205, tests/e2e-cucumber/src/lemonade_protocol.rs:25-28
    • The fixture is an in-process axum server with only GET /api/v1/health and /api/v1/stats routes. No product action can stop it or change its loaded list, so all_models_loaded.len() == 2 always holds.
    • That step backs the --help promise "quitting leaves external servers running" (apps/rocm/src/main.rs:596). It also backs the description's "covered with resulting-state assertions, including the external endpoint remaining alive after dashboard quit". AGENTS.md §3 requires "will not happen" promises to be exercised.
    • Only the data/services emptiness check in that step can go red.
    • Confidence 85 · logic
    • Fix: have the fixture record any non-GET or unload/load request and assert none arrived. Otherwise narrow the description.
  • [new] Cache-root priority is lost when the same GGUF exists in several roots — engines/lemonade/src/direct_llama.rs:428-446, :476-478
    • On the base, hf_checkpoint_gguf_matches walked hf_cache_roots in order (HUGGINGFACE_HUB_CACHE > HF_HOME/hub > app cache > ~/.cache) and sorted within each root only.
    • Now files from every root are merged first. filter_ggufs_by_variant then runs ggufs.sort() over full paths before both the exact-filename find and the name dedup. So the alphabetically first path wins, not the configured root: /home/u/.cache/... beats /srv/hf/....
    • Confidence 85 (base vs head traced) · logic
    • Fix: filter per root in priority order, as the base did, and add a two-root test that asserts the chosen path, not just the file name.
  • [new] lemonade_managed_installed has no test that tells "installed and managed" apart from "installed" — apps/rocm/src/dash.rs:252-261
    • Dropping the detect.managed_env == Some(true) conjunct, or mapping a failed Detect to Some(false) instead of None, is reached only by dash-25 on a GPU lane, and that scenario hangs (first finding).
    • Unit tests in dash.rs exercise only the config defaults.
    • Confidence 75 (read; I did not build the rocm bin here) · logic
    • Fix: inject the Detect result and unit-test the three outcomes.

Non-blocking

  • [new] The Services details pane counts instances the Services overlay no longer shows — crates/rocm-dash-tui/src/ui/tabs/pane.rs:450-455, crates/rocm-dash-tui/src/ui/services_manager.rs:98
    • service_rows now filters i.managed, but the OpenServices preview still counts every is_serving() instance.
    • With only external Lemonade running, the pane says "Running now — 2" and pressing Enter shows "No managed services". service_rows governs, because it is what the lifecycle controls act on.
    • Confidence 75 · logic
    • Fix: apply the same filter in the pane, or label external rows.
  • [standing] The "off by default" comments are now stale — crates/rocm-dash-daemon/src/runner.rs:88-90, :345-347
    • This change edited line 89 but kept "Off by default so hosts with no Lemonade server never poll a dead port", and "off unless a Lemonade endpoint is configured".
    • rocm dash now passes d.enable_lemonade, which defaults to true (crates/rocm-core/src/lib.rs default_dashboard_enable_lemonade). The config default governs; RunnerOptions::default remains false only for tests.
    • Confidence 85 · mechanical
    • Fix: reword both comments.
  • [new] A failed /v1 fallback is reported against the wrong URL — crates/rocm-dash-collectors/src/lemonade.rs:263-274
    • After a 404 on /api/v1/..., a failing status from the retry is reported as GET {url} with the original /api URL.
    • A transport error on the retry prints only legacy_path, with no host or port. Both strings reach the dashboard warning badge.
    • Confidence 85 · mechanical
    • Fix: rebind the URL to the one actually requested.
  • [new] dash-26/27 each bundle about five behaviours, and some step text leaks implementation — tests/e2e-cucumber/features/dash.feature:346-…
    • Examples of leaked implementation: "protocol fixture", "keeps only its first model loaded", "with truthful throughput".
    • In external_lifecycle_keys_do_nothing, the real lifecycle-refusal assertions sit inside a When step.
    • Confidence 70 · mechanical
    • Fix: split by behaviour and move the assertions into Then steps.

Decisions for the author

  • Linux direct-vs-router routing is decided by an exhaustive key and label allow-list over Lemonade's external catalog (direct_llama.rs:339-355) — tradeoff
    • For: anything unrecognised falls to the router, so a model is never served directly on metadata it does not understand.
    • Against: the default model (Qwen3-4B-Instruct-2507-GGUF) no longer has a hard-coded direct path. Any extra label or key in a future catalog moves it, silently, onto the Linux router path that EAI-7423 records as broken. No in-repo fixture of the real entries pins its route.
  • About 230 lines of endpoint grouping, scraping, attribution and reconciliation were added inline to run_loop (runner.rs:706-935) — non-blocking-improvement
    • A pure reconcile function in its own module would make the untested branches above testable without driving the whole loop.

Positive signals

  • Instance.managed is #[serde(default)] false, and a legacy-snapshot test asserts that. Ownership is fail-closed: missing evidence never grants stop/restart.
  • Removing the router-failure fallback ends a path that could quietly serve a different model than requested. It is replaced by an error that keeps the original cause.

Deployment notes

  • rocm dash on Linux/WSL now polls 127.0.0.1:13305 every discovery tick by default. New config keys: dashboard.daemon.enable_lemonade, lemonade_host, lemonade_port.
  • The Services overlay now lists only managed instances. Docker containers, external Lemonade, and snapshots or replays from an older daemon (which have no managed field) no longer appear there.

What this covered

  • What was read. All 33 changed files, completely, plus callers and callees (registry/runner, the TUI consumers, engine_request, expectations.toml, the e2e harness). Range fe47c4891d5e…ac029055f224.
  • How the review was split. Four parallel workers grouped by coupling: the Lemonade engine, the dash daemon and collectors, the TUI/app/e2e, and docs/AGENTS.md/history/prior-PR comments. Two more design-question workers followed.
  • What I ran.
    • rocm-dash-daemon contract tests, plus one mutation.
    • The full rocm-dash-daemon suite.
    • rocm-engine-lemonade --lib, which reproduced the failure above.
    • The rocm-dash-tui services/pane tests.
    • CI rollup and lane logs for this head.

Did not run:

  • rocm bin unit tests and local e2e, skipped for disk and build cost.
  • No GPU on this host.
  • The MI300X lane was still queued at review time.

Real Lemonade server_models.json entries were unavailable, so the default model's actual route is unverified.

Not reconciled against any PR discussion.

rominf
rominf previously requested changes Oct 13, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 93324b5

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Round 3. ac029055, the commit the previous round described, was force-pushed away, so this is a full review of 93324b5e. [new] marks findings this round raised for the first time. [standing] marks objections from round 2 (the informational comment on ac029055) that still hold. The last four Blocking items were not re-found by this round's blind review. They were settled afterwards by putting each one, in its original wording, to the reviewer that had just read this head. That check is anchored, meaning it was told what to look for, so it is weaker evidence than a blind finding; each item carries its verdict. The description finding would not gate on its own, because the squash commit is built from commit messages; the other blockers do gate.

Review — needs work

Full review of the whole change.
No ticket named. The description covers the registered-GGUF resolution and the dash discovery work. It leaves out several behaviour changes from the second commit (see Blocking).

Blocking

  • [standing, in a new form: round 2's "dash-25 hangs both Strix Halo Ubuntu lanes"] The required GPU lanes never ran on this head, so the hardware-only scenarios are unverified. Locations: tests/e2e-cucumber/features/dash.feature:332 (dash-25), tests/e2e-cucumber/features/model_serving.feature:326 (serve-25), tests/e2e-cucumber/expectations.toml:96-101.

    • The self-hosted e2e run 38359301083 for 93324b5e has sat pending with no jobs since 06:53Z. No GPU lane appears in statusCheckRollup.
    • That means three things never ran: dash-25 (per-PR, @requires-gpu), serve-08, and the nightly serve-25. serve-25 is the only end-to-end check that complex catalog entries stay on the Lemonade router.
    • dash-25 reuses the step "a GGUF model is being served on lemonade". serve-lemonade-inference uses the same step and has a Linux/lemonade flaky xfail row (EAI-7423). dash-25 has no row, so it may fail hard on the Strix Ubuntu lane.
    • Confidence 100 · architectural · Fix: get the self-hosted run scheduled (it is stuck, not failed). Then either add a matching expectations row for dash-25 or show on hardware that EAI-7423 is fixed. Dispatch include_nightly once for serve-25 and cite that run.
  • [standing: round 2's "router fallback removal has no test", now with new mutations] Several lemonade engine branches this change introduces are not exercised by any test. Locations: engines/lemonade/src/lib.rs:546, :569, :578, :585, :588-611; engines/lemonade/src/direct_llama.rs:152-159; engines/lemonade/src/process.rs:814-856.

    • Each mutation below left the whole lemonade lib suite green (143 tests):
      • persisting request.model_ref instead of router_model_ref at :546;
      • passing request.model_ref to the load, loaded-model and chat-smoke calls;
      • pulling the alias instead of checkpoint.model_ref();
      • swallowing the pull error with .ok().
    • The removed router-failure fallback has no test either. serving_gpu_model_load_process_error_is_returned calls run_lemonade_model_load, whose body this diff does not change, so it would still pass with the fallback restored.
    • That same test is also not hermetic. With XDG_RUNTIME_DIR unset it tries to create the real ~/.rocm/data/lemonade. On a clean tree here it failed with assertion failed: error.to_string().contains("Lemonade model load failed") (env -u XDG_RUNTIME_DIR cargo test -p rocm-engine-lemonade --lib serving_gpu_model_load_process_error_is_returned: 1 run, 1 failed).
    • Confidence 100 · logic · Fix:
      • Make the router branch's state persistence and failure handling testable, for example with a seam around spawn and load, and assert the id that gets persisted.
      • Assert the pull owner/repo:variant argv and that a non-zero exit is propagated, using the existing fake-executable pattern.
      • Give the load test an explicit ParentRuntimeEnvironment or a tempdir runtime dir.
  • [new] Several dashboard branches this change introduces are not exercised by any test. Locations: crates/rocm-dash-daemon/src/runner.rs ~641-656, ~741-757, ~858-868; crates/rocm-dash-tui/src/ui/services_manager.rs:142; crates/rocm-dash-tui/src/ui/tabs/pane.rs (mark(...)).

    • Each of these mutations left the relevant suite green:
      • Model/host-change telemetry reset: turning it into a no-op left all 7 lemonade contract tests green.
      • Scrape-only cadence: forcing discover_external: true left everything green, because every new contract test sets discovery_tick == instance_tick. That leaves the "hold rows on a scrape-tick health failure" path unexercised.
      • Selection clamp: removing it left all 25 TUI tests green.
      • Pane glyph: forcing mark(false) left all 25 TUI tests green.
    • Confidence 100 · logic · Fix:
      • Add a contract test that rewrites a managed record's model while the clock is held, and asserts the rate resets.
      • Add one test with discovery_tick = 3 × instance_tick that covers the scrape-tick hold.
      • Add a TUI test that shrinks the rows below selected and then presses s.
      • Assert the ✓/· glyph.
  • [standing in part: round 2's "external server remains alive after quit check cannot fail"; the other checks are new] Some new checks claim coverage they cannot fail on. Locations: engines/lemonade/src/direct_llama.rs:584-587, :461 and :875-876; tests/e2e-cucumber/tests/e2e/dash_steps.rs external_survives_quit (dash.feature:356,361) and real_lemonade_visible.

    • assert_eq!(requested.model_ref, "qwen-smoke", "direct aliases are not normalized") checks a value serving_route only borrows, so it is always true.
    • The test comment says the is_file() filter in collect_gguf_files is covered. The fixture is a directory that the earlier is_dir() branch handles first, so removing is_file() leaves everything green.
    • The e2e step "the protocol fixture remains alive" checks an in-process fixture thread that nothing the dashboard does can stop.
    • "Real managed Lemonade model is displayed" only waits for "Qwen3-0.6B" anywhere on screen. Managed rows were already shown before this change.
    • A check that names a branch it cannot fail on is worse than no check, because it stops the next reader from looking.
    • Confidence 85 · mechanical · Fix:
      • Assert the --alias produced by direct_llama_server_command along the launch_service path.
      • Add a dangling-symlink fixture (#[cfg(unix)]) and correct the comment.
      • Run the fixture as a child process and check its PID, or rename the step to what it really asserts ("no managed records").
      • Anchor the managed-row check to that row's port or status.
  • [standing in part: round 2 flagged the "remaining alive" overclaim; the omissions are new] The description misstates the change's scope.

    • It does not mention four things the diff does:
      • It removes the Linux fallback to direct llama-server after a router load or verify failure (lib.rs:588-611).
      • The Services overlay now lists only managed instances, so Docker-discovered containers that used to appear are gone (services_manager.rs:98).
      • Complex catalog entries now go to the Lemonade router.
      • It adds the nightly scenario @id:serve-lemonade-complex-registered-inference.
    • It claims "the external endpoint remaining alive after dashboard quit" is covered "with resulting-state assertions". That assertion cannot fail (previous finding).
    • Confidence 85 · mechanical · Fix: update the description to list these changes. Either narrow the "remaining alive" claim or make the assertion real.
  • [standing, not re-found blind, settled by objection check: HOLDS] discover_models has no production caller; its three tests check a copy of the logic the daemon runs — crates/rocm-dash-collectors/src/lemonade.rs:292-300, crates/rocm-dash-daemon/src/runner.rs:800-814

    • Only tests in the same file call discover_models, and a workspace grep finds no production caller. The runner repeats the health → loaded_models → lemonade_service mapping inline. Its error handling differs: it propagates the fetch_health() error with ?, while discover_models uses unwrap_or_default().
    • Partly mitigated: the shared helpers (fetch_health/get_text, parse_health, loaded_models, lemonade_service) are production code. Only the way they are put together, and the error-to-empty behaviour, are tested on a copy.
    • Fix: extract one pure function both paths use, or delete discover_models and point the tests at the runner's path.
  • [standing, not re-found blind, settled by objection check: HOLDS, confirmed by a run] No test covers dropping external rows when a managed record takes over the endpoint — crates/rocm-dash-daemon/src/runner.rs:726-745

    • With the for gone in &lemonade_ids { … } loop and lemonade_ids.clear() deleted, RUSTC_WRAPPER= cargo test -p rocm-dash-daemon stayed fully green: lib 45, a second binary 9, telemetry_contract 16. All 7 lemonade_* contract tests ran by name.
    • lemonade_external_endpoint_deduplicates_managed_owner_then_reappears_after_record_removal writes the managed record before the runner starts (telemetry_contract.rs:328-329). So the transition from external rows to a managed record never happens.
    • Fix: add a contract case where external rows exist first and a matching registry record then appears.
  • [standing, not re-found blind, settled by objection check: HOLDS, read from code] Cache-root priority is lost when the same GGUF exists in several roots — engines/lemonade/src/direct_llama.rs:428-446, :477, :115

    • The base served hf_cache_roots(..).into_iter().find_map(..), so the first root in priority order won (prw-base direct_llama.rs:327-329).
    • Now every root is flattened, filter_ggufs_by_variant sorts by full path, and matches.into_iter().next() is served. Nothing canonicalizes afterwards, so the alphabetically first path wins. It matters when copies differ, for example a stale ~/.cache snapshot against the configured HUGGINGFACE_HUB_CACHE.
    • The only multi-root test (:865-893) asserts file_name() alone, so it cannot see which root won.
    • Fix: filter per root in priority order, and add a two-root test that asserts the full path.
  • [standing, not re-found blind, settled by objection check: HOLDS, read from code] lemonade_managed_installed has no test that tells "installed and managed" apart from "installed" — apps/rocm/src/dash.rs:252-261

    • The only unit test in dash.rs covers config. The TUI tests set the Option directly.
    • Hosted dash-26/27 reach only a Detect that succeeds with installed=false. So neither dropping the managed_env conjunct nor mapping a Detect error to Some(false) is caught there.
    • Only dash-25 exercises Some(true), and dash-25 did not run on this head (first finding). The check could not confirm round 2's claim that dash-25 hangs.
    • Fix: inject the Detect result and unit-test the three outcomes.

Non-blocking

  • [standing] Stale "opt-in / off by default" comments — crates/rocm-dash-daemon/src/runner.rs:88-90, :345-347. rocm dash now enables discovery by default through DashboardDaemonConfig.enable_lemonade (crates/rocm-core/src/lib.rs:2413), which apps/rocm/src/dash.rs:61 passes through. Confidence 85 · mechanical · Fix: say discovery is on by default in rocm dash via config, and that only the bare RunnerOptions::default() is off.
  • [new] rocm dash --help promises discovery on every platform — apps/rocm/src/main.rs:598-602. The embedded daemon returns None on Windows (apps/rocm/src/dash.rs:975-978), so nothing is discovered there. README already scopes the claim to Linux/WSL. Confidence 85 · mechanical · Fix: prefix the help text with "On Linux/WSL,".
  • [new] A failed router serve leaves engine state at running, and this now happens on Linux too — engines/lemonade/src/lib.rs:588-611. Both error arms kill the child and return without mark_json_status(.., "failed"), unlike the direct path (process.rs:536-540). Before this change, Linux took the fallback instead. The rocmd record does flip to failed. Confidence 85 · logic · Fix: mark the state failed in both arms.
  • [new] Remediation hints dropped — engines/lemonade/src/direct_llama.rs:56-75, :98-112. The missing-backend error no longer says to run rocm engines install lemonade (that command does fix it), and the variant errors lost their {model}:Q4_K_M examples. Confidence 85 · mechanical · Fix: restore both.
  • [new] "open to check" points the user at an overlay that never shows install status — crates/rocm-dash-tui/src/ui/tabs/pane.rs (Lemonade and managed-runtime lines). draw_engine_manager (engine_manager.rs:252-300) shows no Detect status. The wording copies the existing vLLM line. Confidence 85 · logic · Fix: say "unknown — run rocm engines list", or show status in the overlay.
  • [new] Install status is captured once per session — apps/rocm/src/dash.rs:314, crates/rocm-dash-tui/src/app/event_loop.rs:805. Installing Lemonade from the Engines overlay leaves "managed runtime not installed" on screen until restart. Confidence 85 · logic · Fix: re-run Detect when an engines job completes.
  • [standing] Wrong URL in errors after the /v1 fallback — crates/rocm-dash-collectors/src/lemonade.rs get_text ~309-325. The message reports the original /api/v1 URL with the fallback's status, and the transport error drops the host. Confidence 85 · mechanical · Fix: format errors with the URL that was actually requested.
  • [new] Leftover parameter from the fallback design — engines/lemonade/src/process.rs:451, lib.rs:528. reason: &anyhow::Error is now always a fixed string and only gets logged. Confidence 85 · mechanical · Fix: make it &str or drop it.
  • [new] Stale doc and comment wording — three places:
    • docs/testing.md:852 still calls direct serving "the packaged llama-server fallback".
    • direct_llama.rs:240-244: the DIRECT_LLAMA_SERVER_BACKENDS doc describes lookup ordering it no longer drives.
    • direct_llama.rs:1098-1111: test names and comments refer to the removed find_llama_server_binary.
    • Confidence 85 · mechanical.
  • [new] A managed service bound to :: can show up twice — crates/rocm-dash-daemon/src/registry.rs ~257-261 and endpoints_match. A dual-stack :: bind on a non-default port 13305 is also reached by the external probe, giving a false "externally running" row and double-counted gen_tps. Confidence 70 · logic · Fix: match a :: record against both loopback families, and add a test case.
  • [new] A twin construction drops ownership — apps/rocm/src/dash.rs:621-628. launcher_serving_instances builds Instance from registry records without managed: svc.managed, while merge_instance carries it. Harmless today, because the launcher does not read the flag. Confidence 70 · mechanical.
  • [new] A stale approval swallows one keypress — services_manager.rs:132-139. The modal is hidden, but the next key only clears it. Confidence 70 · logic · Fix: fall through and handle the key normally.
  • [new] Managed direct llama-server rows may carry a standing "lemonade health" warning — registry.rs:257-270, runner.rs:862-868. Every Lemonade-engine record is scraped at /api/v1/health, which a direct llama-server likely does not serve the way the collector expects. Inferred from the code, not run. Confidence 70 · logic.
  • [new] Match arms the current writers never reach — direct_llama.rs:258-259 (rocm-stable/rocm-nightly). Manifest writers only ever store rocm or vulkan; deleting both arms left all tests green. Confidence 70 · mechanical.
  • [new] A dead branch in lemonade_service — lemonade.rs ~247-266. Every caller passes a trimmed, non-empty name, so the empty-name path is unreachable and untested. Confidence 85 · mechanical.
  • [new] The manual-testing sequence cannot run as written — docs/manual-testing.md:219-224. rocm serve qwen-smoke … --port 11435 without --managed attaches to the log, so the next rocm services stop line is unreachable. Confidence 70 · mechanical.
  • [new] README never says where config.json lives — README.md:196-199. Editing the file is the only way to set dashboard.daemon.*, but the path and the ROCM_CLI_CONFIG_DIR override are not documented. Confidence 70 · mechanical.

Decisions for the author

  • Removing the router-failure → direct llama-server fallback on Linux — tradeoff. Pro: a real load error is no longer hidden behind a cached-model direct serve. Con: the narrow set of users whose router load failed while a direct server was available now gets an error on first serve. Confirm this is intended, and say so in the description.
  • The default model's Linux route now depends on catalog labels — non-blocking-improvement. resolve_direct_llama_checkpoint serves directly only when every label is in {reasoning, tool-calling, tools}. If the pinned 11.5.1 server_models.json labels Qwen3-4B-Instruct-2507-GGUF with anything else (Lemonade's generated docs show hot), default rocm serve silently moves to the router path, which is the one the EAI-7423 xfail rows describe. A unit fixture with the pinned entry would lock this down.
  • Provenance as managed: bool plus container_name == "lemonade" — non-blocking-improvement. A Docker vLLM container named lemonade is labelled "Lemonade — externally running". The lifecycle gate uses managed alone, so the impact is cosmetic. An explicit enum Provenance { Managed, Docker, ExternalLemonade { host, port } } would make the confusion unrepresentable and carry the endpoint the dedup needs.
  • The external probe is awaited inside the daemon tick loop — non-blocking-improvement. With an unreachable or slow lemonade_host, each discovery tick can stall GPU sampling and the vLLM scrape for 1.5–3 s. This follows the vLLM collector's existing pattern. A separate task feeding a channel would isolate it.
  • Scope — non-blocking-improvement. 33 files across two separable changes: lemonade serving-route resolution, and dash external discovery/ownership. Splitting them would make each easier to review and to revert.

Positive signals

  • serving_route decides direct vs router before any download or spawn, and re-checks the GPU-required policy there for both platforms, so no CPU path is introduced (direct_llama.rs:44).
  • Direct-server readiness now requires an exact alias match (process.rs:650). Mutating it back to fuzzy matching fails the tests.
  • Ownership is assigned in exactly one place (registry.rs:169), and legacy snapshots default to managed: false. An old snapshot can never unlock stop/restart.
  • stats_model refuses to attribute a last-request rate unless exactly one model is loaded. The dash-26 fixture puts the rate's model first so that naive attribution fails.
  • The sr + single-Esc check in external_lifecycle_keys_do_nothing is a sound black-box probe: any staged approval or job console would keep the overlay open.

Deployment notes

  • rocm dash now probes 127.0.0.1:13305 for Lemonade on every discovery tick by default (Linux/WSL embedded daemon). It adds the config keys dashboard.daemon.enable_lemonade, lemonade_host and lemonade_port. They have serde defaults, so existing configs keep loading.
  • The Services overlay no longer lists unmanaged instances, Docker-discovered containers included.
  • On Linux, a router load or verify failure in rocm serve with the lemonade engine is now an error instead of a fallback to direct llama-server.

What this covered

  • What was read. Every file in prw-base…93324b5e (a02a859a…93324b5e, two commits), split across seven read-only workers grouped by coupling:
    • lemonade direct_llama/state;
    • lemonade lib/process;
    • dash daemon, collectors and core;
    • dash TUI plus apps/rocm/src/dash.rs;
    • e2e features and steps;
    • docs plus AGENTS.md adherence;
    • history, prior changes and code comments.
  • Design questions. Two more workers took the design questions in Step 7. The callers of changed symbols were traced.
  • Prior changes. The pass consulted the review comments of #80, #94, #242, #404 and #457.
  • Tests run. Mutation verdicts came from real test runs, mostly in per-worker copies. The non-hermetic test failure was reproduced on a clean tree.
  • Did not run / limits:
    • The TUI worker read only the diff hunks of crates/rocm-dash-tui/src/app/mod.rs and app/event_loop.rs (4- and 1-line changes), not those files in full.
    • Workers shared this checkout and briefly mutated tracked files in it. The local e2e run of dash-26/27 (2 scenarios, both passed, and also green in hosted CI) may have been built with another worker's pane.rs glyph mutation in place.
    • The apps/rocm test lemonade_discovery_defaults_and_overrides_reach_daemon was not run because of the time box.
    • GPU scenarios (dash-25, serve-08, serve-25) could not run: there is no GPU here, and CI never scheduled them.
    • The pinned Lemonade 11.5.1 catalog was not available offline, so the default-model label question is unresolved.
    • The ai-instruction-quality check was not needed, since no agent-instruction files changed.
    • The leak scan of the diff was clean, and no prompt-injection content was found.
  • Not reconciled. This report was not reconciled against the PR's discussion.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch 2 times, most recently from cd8558a to 7778139 Compare October 13, 2026 10:52
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the round 3 review. I pushed 77781396, which is rebased on current main. It replaces 93324b5e, so the review and the round 2 comment were written against older heads. Here is the status of each item.

Blocking

  • GPU lanes: not fixed, and I cannot fix it from here. The self-hosted run for the earlier head never started. For dash-25 I added the active-runtime step, a bounded serve (a stuck rocm serve is now killed at the serve timeout instead of stalling the lane), and a flaky expectations row like the other Lemonade serve scenarios. The nightly serve-25 still needs an include-nightly dispatch on a GPU lane. I have not run it.
  • Engine branches: fixed. The router path now lives in router_serve.rs behind a small trait. New tests check the persisted name, that the router name (not the alias) reaches load and verify, and that each failure stops the server and marks state failed. Other new tests check the exact pull owner/repo:variant command line and that a failed pull is returned. The load test no longer touches the real user directory. The removed direct fallback is covered only structurally: serve_router has no direct step, but a test cannot catch someone adding a fallback in serve_http itself.
  • Dashboard branches: fixed. New tests cover the model and host reset, the scrape-tick hold (discovery_tick of 3 times instance_tick), the selection clamp, and the glyphs. I checked each with a mutation that makes it fail.
  • Checks that cannot fail: fixed. The alias test now checks the --alias that direct_llama_server_command produces. A dangling-symlink test covers is_file(). The fixture now records any request other than a GET of health or stats, and the scenarios assert none arrived. I renamed the step to say what it checks. The managed-row check now needs a full Observe row. I could not run that last change, because dash-25 needs a GPU.
  • Description: updated. It now lists the removed router fallback, the managed-only Services overlay, the complex-entry routing, and serve-25.
  • discover_models: removed. Collector tests now use the function the runner calls.
  • Takeover of an external endpoint by a managed record: new test. It fails if the drop loop is disabled.
  • Cache-root priority: fixed, with a two-root test that checks the full path.
  • lemonade_managed_installed: the verdict is now a pure function with unit tests for all outcomes.

Non-blocking

  • Fixed: stale "off by default" comments; the /v1 fallback URL in errors; --help now says Linux/WSL; a failed router serve now marks state failed; the hints are back (rocm engines install lemonade --reinstall and owner/repo:Q4_K_M-style examples); the unused reason error parameter; stale doc and test names; the rocm-stable and rocm-nightly arms (removed, since no writer produces them); the dead branch in lemonade_service; managed in launcher_serving_instances; a stale approval no longer swallows a key; the "unknown, run rocm engines list" wording; the manual-testing sequence; and the README config path.
  • :: bind: a managed service bound to :: now matches both loopback families, with table tests and a contract test.
  • Standing warning on managed direct llama-server rows: I confirmed it by running a fixture. Such a row now stays visible with no throughput and no warning. A 503 or a transport failure on a real router still warns.
  • Install status is read once per session: I did not add a live refresh. That needs a new event path from engine jobs. The pane now says that status is read at startup.
  • dash-26 and dash-27: assertions moved out of the When step into Then steps, and the wording describes what the user sees. I did not split the scenarios.

Decisions

  • Removing the router-to-direct fallback is intended. The description now says so.
  • I added a test with the real 11.5.1 catalog entries for the default and canary models. Entries that carry other labels, such as hot or coding, go to the router on purpose.
  • I left the provenance enum, the probe in the tick loop, and the split of this PR as they are.

Verification: the full workspace suite passed serially (33 targets, 3,775 tests), and so did both clippy commands, fmt, crate edges, smoke, and the external protocol scenarios. I could not run GPU scenarios, the Windows build, or lychee locally. The previous head failed the Windows job on a test-only helper with no caller there. I gated it the same way as its callers. CI will confirm.

rominf
rominf previously requested changes Oct 13, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 7778139

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
No ticket named; the description's claims were checked against the code (see the first and third Blocking findings).

Blocking

  • [new] Several serve error paths never mark the engine state failed, while the docs and description say every error after the spawn does — engines/lemonade/src/router_serve.rs:49-56, :95-111, :112; engines/lemonade/src/lib.rs:509-529; docs/architecture.md:161
    serve_router only stops the server and calls mark_json_status(.., "failed") for errors raised inside the outcome closure. Three ? exits outside it skip both: the write_running_state_for_model right after the spawn, the merge_json_state once the model is ready, and wait_server(). LemondChild has no Drop, so on those paths lemond is left running and the state stays running or ready.
    The direct route has the same gap. Errors from serving_route and ensure_direct_llama_model_available (missing binary, invalid checkpoint, ambiguous variant, a failed pull) leave the state at the starting that serve_http wrote. This change fixed the router copy of this pattern but not the direct one.
    docs/architecture.md:161 says the module will "mark engine state failed on any error after the spawn". The code is what governs here, since it is what actually runs, so the doc claim is false.
    Confidence 85 · logic · Fix: one guard in serve_http that marks the state failed (and stops any spawned child) on every Err after starting is written. Otherwise narrow the doc line to what the code does.

  • [new] The only end-to-end scenario for the managed engine pane's "installed" state can never go red — tests/e2e-cucumber/expectations.toml:106-111, tests/e2e-cucumber/features/dash.feature:332
    dash-25 is tagged @requires-os:linux @requires-gpu @requires-engine:lemonade. Its new row matches { effective_engine = "lemonade", os = "linux" } with flaky = true, and that covers every lane the scenario can run on. flaky accepts both pass and fail.
    So the "Lemonade — managed runtime installed" assertions have no scenario that can fail, which AGENTS §3 requires for user-observable behaviour. The dash assertions are hidden behind an unrelated serve bug (EAI-7423). The description says the row exists, but not that it leaves this behaviour without a gate.
    Confidence 85 · logic · Fix: assert the managed pane from a scenario that doesn't need the flaky real serve (for example a planted registry record plus a real Detect), or narrow the row to the lanes where the serve bug actually occurs.

  • [new] Text about the CLI's own behaviour is pinned with no pointer to a test of that behaviour, despite the description's checkbox — apps/rocm/src/main.rs (dash_help_scopes_lemonade_discovery_to_linux_and_wsl); crates/rocm-dash-tui/src/ui/tabs/pane.rs:432-438 and startup_snapshot_note_shown_only_with_a_verdict; engines/lemonade/src/direct_llama.rs:65-73 and its test at :660-672
    AGENTS §3: "Where the text genuinely has to be pinned on its own, the assertion carries a comment naming the test that proves the behaviour". None of these three does:

    • "quitting leaves external servers running" (help text);
    • "Install status is read at startup. Restart the dashboard to refresh." (engine pane);
    • the advice to run rocm engines install lemonade --reinstall, which §3 says must be shown to actually fix the condition.
      The description ticks "Discovery and ownership messages are asserted together with the resulting state."
      Confidence 85 · mechanical · Fix: add the pointer comments (dash-26 / @id:dash-lemonade-external-protocol-discovery exists for the first), and add a test that restores the backend through the reinstall path, or name one that does.
  • [new] A tracked file is cited as a bare backtick path in new docs/ text — docs/testing.md (the dash-25 bullet: "serve_timeout_secs in expectations.toml")
    AGENTS §5 requires repo files cited from docs/ to be relative links. The same section already links ../tests/e2e-cucumber/expectations.toml a few lines later.
    Confidence 85 · mechanical · Fix: make it [expectations.toml](../tests/e2e-cucumber/expectations.toml).

  • [standing in part since 93324b5 — not re-found blind; settled by putting it to the reviewer after its report was frozen (anchored verdict): HOLDS IN PART] The GPU lanes have not run on this head, so the hardware-only scenarios are unverified. The part about dash-25 lacking an expectations row no longer holds — that row now exists (and is the second new finding above).

    • The only self-hosted run for 7778139, "E2E self-hosted" run 38386320833, has been pending with zero jobs since 10:52Z. No GPU, Strix or WSL check run appears for this head, so dash-25, serve-08 and serve-25 have no hardware result.
    • No include_nightly dispatch exists for this head. serve-25 is still the only end-to-end check that complex catalog entries stay on the Lemonade router.
    • Fix: get the self-hosted run scheduled (it is stuck, not failed) and dispatch include_nightly once for serve-25; cite both runs.
  • [standing in part since 93324b5 — not re-found blind; settled by putting it to the reviewer (anchored verdict, read-only, no mutations run): HOLDS IN PART] Some lemonade engine connections this change introduces are still exercised only through fakes. The new router_serve.rs tests, the pull-argv test, the pull-error tests and the now-hermetic load test cover most of the original mutations. What remains:

    • The removed router-to-direct fallback is covered only structurally; nothing fails if one is re-added in serve_http (the author agrees).
    • The real LemondRouterSteps::load_model, model_loaded and chat_smoke (router_serve.rs:176-205) could ignore the name they receive and use self.request.model_ref; router_name_not_alias_reaches_load_loaded_and_smoke_checks drives FakeSteps only.
    • The real pull closure in ensure_direct_llama_model_available (direct_llama.rs:137-169) could pull request.model_ref instead of the checkpoint ref and stay green.
    • Fix: one test through the real LemondRouterSteps against a fake HTTP server asserting the name on the wire, and one through the real pull closure with the existing fake-executable pattern.
  • [standing in part since 93324b5 — not re-found blind; settled by putting it to the reviewer (anchored verdict): HOLDS IN PART] Two checks still claim coverage they cannot fail on. The is_file() dangling-symlink test and the renamed stray-request step now fail as they should.

    • The alias test (direct_llama.rs:605-628) builds the command from its own requested value, so the launch_service / serve_http path to direct_llama_server_command is never exercised. That is also why the launch normalization change listed under Non-blocking is uncaught.
    • "Real managed Lemonade model is displayed": observe_row_cells (dash_steps.rs:302-307) needs a cell starting with "Qwen3-0.6B" followed by 7 more cells, with no port or status check. Managed rows already appeared before this change, so the step passes with it reverted — and dash-25 sits under a flaky row anyway.
    • Fix: drive launch_service or serve_http and assert the served --alias; anchor the managed-row check to that row's port or status.

Standing objections from 93324b5 that no longer hold

Each was not re-found by this round's blind review, then put to the reviewer one at a time with the author's reply (anchored verdict, from reading; no mutations could run). All were judged no longer to hold:

  • Dashboard branches (telemetry reset, scrape-tick hold, selection clamp, glyph): each now has a named test that would plausibly fail under the original mutation.
  • discover_models: removed; the collector tests call the runner's services_from_health.
  • External rows dropped on managed takeover: lemonade_managed_record_taking_over_endpoint_drops_existing_external_rows exercises the transition.
  • Cache-root priority: per-root filtering in priority order; checkpoint_matches_prefer_the_highest_priority_cache_root orders roots so alphabetical and priority order disagree, and asserts the full path.
  • lemonade_managed_installed: the pure managed_installed_from_detect is unit-tested on all outcomes.
  • Description scope: the description now lists the removed fallback, the managed-only Services overlay, complex-entry routing and serve-25. The commit messages still do not mention the removed fallback or the Services-overlay change; that is an omission, not a false claim, and does not gate.

Non-blocking

  • [new] A stale cached collector can take over a new managed endpoint and dial the wrong IP family — crates/rocm-dash-daemon/src/runner.rs:770-797, crates/rocm-dash-daemon/src/registry.rs:251-272
    Grouping chains managed_lemonade_collectors.keys() into endpoints_match, which is deliberately non-transitive: :: matches 127.0.0.1. Say a :: record is removed and a 127.0.0.1 service takes the same port. It keeps the cached collector, which dials scrape_host("::") = "::1" against an IPv4-only listener, and the warning never clears. This contradicts the in-code comment "without … conflating IPv4 and IPv6 listeners". No test exercises the cached-key reuse branch.
    Confidence 70 · logic · Fix: reuse a cached collector only when scrape_host(cached) == scrape_host(target.host).

  • [new] Managed Lemonade rows now show the last request's throughput as a fresh live rate indefinitely — crates/rocm-dash-daemon/src/runner.rs:925-935, crates/rocm-dash-collectors/src/lemonade.rs:121
    /api/v1/stats reports the most recent request (module doc, lines 8-10). Every scrape calls observe_direct, which refreshes last_observed_at, so an idle server keeps showing, say, 42 tok/s as Fresh. External rows already behaved this way; this change extends it to every managed Lemonade row. The doc comment at :121 calls the value "instantaneous".
    Confidence 70 · logic · Fix: fingerprint the stats tuple and stop refreshing freshness (or report 0) while it is unchanged.

  • [new] The engine's launch method no longer normalizes model_ref, and the description doesn't mention it — engines/lemonade/src/lib.rs:451 vs :409
    Through Launch, qwen-smoke is now served with --alias qwen-smoke, and direct readiness requires that exact name. resolve still reports Qwen3-0.6B-GGUF. rocm serve and rocmd are unaffected, because they pass canonical_model_id to serve-http. This still changes the rocm-engine-protocol contract (AGENTS §6).
    Confidence 70 · architectural · Fix: keep normalizing in launch_service, or pin the new identity with a protocol test and mention it.

  • [new] The new lib.rs test reads with unwrap from sockets that are non-blocking on Windows — engines/lemonade/src/lib.rs:694-726 (serving_regression_persisted_identity_drives_health_chat_model)
    listener.set_nonblocking(true), then stream.read(..).unwrap() on the accepted streams. On Winsock, accepted sockets inherit non-blocking mode, so an early read returns WouldBlock and panics. The Windows lane runs cargo test; it passed once on this head.
    Confidence 70 · mechanical · Fix: call stream.set_nonblocking(false) after accept.

  • [new] run_rocm_bounded limits the whole CLI serve to the harness's readiness window — tests/e2e-cucumber/tests/e2e/serving_steps.rs:150-151, tests/e2e-cucumber/tests/e2e.rs (run_rocm_bounded)
    For @serve-timeout:2400 the harness clock starts at spawn, but the CLI's own wait (ROCM_CLI_VLLM_READY_TIMEOUT_SECS) starts after its pre-launch work. The harness therefore kills the CLI first, and the CLI's own diagnostic is lost in the slow cold-load case. The 90 s rows now also have to fit any cold backend or weights fetch into that window.
    Confidence 70 · logic · Fix: give the CLI call timeout + margin, or a separate launch budget.

  • [new] serve-25 runs rocm serve --managed through the unbounded run_rocm and has no EAI-7423 row — tests/e2e-cucumber/tests/e2e/serving_steps.rs:676-689, tests/e2e-cucumber/features/model_serving.feature:326
    This change adds run_rocm_bounded precisely because the unbounded call "stalls the whole lane with no report". serve-25 is the same Linux managed Lemonade serve as the rows that carry EAI-7423, but has no row.
    Confidence 70 · logic · Fix: use run_rocm_bounded, and add a row or say why the router path isn't affected.

  • [new] write_dashboard_config overwrites config.json and drops the runtime that the previous step activated — tests/e2e-cucumber/tests/e2e/dash_steps.rs:92-101
    This is harmless today only because Lemonade skips the active-runtime check (serve_cmd.rs:722-730).
    Confidence 70 · mechanical · Fix: read the file, set only dashboard.daemon.*, and write it back.

  • [new] No gating scenario asserts the new Linux direct route for simple registered names — engines/lemonade/src/lib.rs:512-530; tests/e2e-cucumber/features/model_serving.feature:61-70
    The only scenario serving Qwen3-0.6B-GGUF is covered by a flaky row on Linux and checks inference only. If the route were reverted to the router, every scenario would stay green.
    Confidence 70 · logic · Fix: assert load_response.method on a gating lane, or name the unit test covering the route in the PR text.

  • [new] The README says unrecognized catalog shapes use the router, but on Linux a malformed registered checkpoint fails the serve — README.md (~681); engines/lemonade/src/direct_llama.rs:46-47, :359-385
    resolve_direct_llama_checkpoint runs before direct_serve_binary and only when is_linux || explicit_checkpoint. On Linux the malformed entry fails even if no direct binary exists, while on Windows the same entry goes to the router. The comment "must not escape validation via routing" holds only on Linux. The code governs, because it is what runs.
    Confidence 70 · logic · Fix: send a malformed registered entry to the router unless the user gave an explicit owner/repo reference, or document the error.

  • [new] A sharded GGUF variant in a registered entry goes direct, downloads every shard, then fails as ambiguous — engines/lemonade/src/direct_llama.rs:97-127, :508-540
    collect_gguf_files recurses into the shard folder. The substring filter then matches every -0000N-of-0000M file and bails "matches multiple files". This contradicts the new docs line that multi-file entries stay with the router. It was not a working path before either. Whether the pinned catalog actually contains such an entry was not checked.
    Confidence 70 · logic · Fix: when every match shares one shard stem, pick shard 00001, or route split matches to the router before downloading.

  • [new] A Lemonade router serving on Vulkan never reaches the CLI's readiness check, and more Linux serves now take the router — apps/rocm/src/main.rs:19055, :19113-19120
    The CLI accepts a router model only when its backend starts with rocm, while the engine side accepts vulkan (router_serve.rs tests record backend_requested: "vulkan"). The check existed before this change. What is new is that complex entries on Linux are now routed through the router, so a Vulkan host serving one waits forever.
    Confidence 70 · logic · Fix: accept the backend the engine recorded, and add a Vulkan router case to the readiness tests.

  • [new] No GPU-lane check ran for this head — CI rollup
    Every hosted check passes. No self-hosted GPU-lane checks appear, so dash-25, serve-08 and nightly serve-25 have no result for 7778139.
    Confidence 70 · mechanical · Fix: run the required GPU lanes before merging.

Decisions for the author

  • This PR makes two logical changes — non-blocking-improvement
    The Lemonade serve-route rework (registered-GGUF resolution, router extraction, removal of the direct fallback) and the dash discovery and ownership rework share almost no code. Together they come to about 5,300 lines. AGENTS §11 asks for one logical change per PR; splitting would let each be gated and reverted on its own.
  • The Linux router-to-direct fallback was removed — tradeoff
    The description already discloses this. Router failures now surface as errors instead of being masked, at the cost of a first-serve failure on hosts where a direct server would have worked. Confirm this is the intended choice for Vulkan and WSL hosts, given the readiness gap above.

Positive signals

  • Moving the router path into router_serve.rs behind a RouterSteps trait makes the failure paths testable without a real server. router_failures_return_error_stop_server_and_mark_state_failed would fail on the old code.
  • Lifecycle ownership comes from a positive managed flag from the registry rather than id or engine string markers. The approval tests cover a stale target on both the key and draw paths.
  • The e2e protocol fixture records every request that is not a health or stats GET, which turns "never touches external servers" into something the scenarios actually assert.

Deployment notes

  • rocm dash now probes local Lemonade (default port 13305) by default on Linux and WSL; it can be turned off in config.
  • The Services overlay now lists only registry-managed services. Docker-discovered containers and external Lemonade servers no longer appear there.
  • A newer TUI attached to an older daemon gets no managed field, so its Services overlay is empty (by design: missing ownership is treated as unmanaged).

What this covered

Every file in the change was read completely (37 files), plus callers and callees in apps/rocm, apps/rocmd, rocm-core, the vLLM engine and the e2e harness. Diff prw-base…77781396 (merge base 2844d79).

What did not run, and why:

  • No build or test ran. The host's free disk fell below the build floor during the review, so builds were stopped. Every finding is from reading, none is scored 100, and all mutation checks — including the ones behind the standing-objection verdicts above — are unverified.
  • telemetry_contract.rs lines outside the diff were not re-read.
  • The pinned Lemonade 11.5.1 server_models.json is not on this host, so the sharded-catalog question is unverified.
  • The prior-changes pass was limited to inline comments on about seven earlier merged PRs.
  • GPU and Windows execution is not possible here.
  • No prompt-injection content was found.

The review ran fanned out across six independent workers: two for the engine, one for dash collectors/daemon, one for the TUI, one for e2e and docs, and one for the instruction, history, prior-change and comment passes. The findings were reached without the PR's discussion; the standing objections above were then settled against the author's reply of 10:53Z.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from 7778139 to 11b6452 Compare October 14, 2026 09:15
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for round 4. I pushed 11b64524, rebased on current main. Here is the status of each item.

Blocking

  • Failed state on every error: fixed. One wrapper in serve_http now marks the state failed on any error after starting, for the router and the direct route, and returns the original error. The router path also stops its server on the three exits you listed. I corrected the architecture line. One gap is left: the direct llama-server path still does not stop its child at its own post-spawn state writes. The state does end at failed.
  • dash-25 cannot go red: agreed, and I did not add a hardware scenario that I cannot run. docs/testing.md now says dash-25 does not gate the installed wording, and names the unit tests that do. No other scenario shows the installed state through a real Detect.
  • Pointer comments: added for the help text (dash-26), the startup status note (a new test shows the status does not change after startup), and the reinstall advice. For the reinstall advice I found no scenario that proves the backend binary is restored, and the comment says so. The two --reinstall scenarios prove only that the command and flag exist.
  • Bare path in docs/testing.md: now a relative link.
  • GPU lanes: unchanged. The self-hosted run is not mine to schedule, and the nightly serve-25 still needs an include-nightly dispatch.
  • Fakes only: fixed. New tests drive the real router steps against a loopback fixture and a fake CLI, and the real pull closure with a fake executable. Both fail if the alias is used instead of the router name or checkpoint. A new test also runs the serve path and asserts --alias. The removed fallback is still covered only structurally.
  • Checks that cannot fail: the alias test now goes through the serve path. For the managed row, the Observe row has no status or port column, so I check status and port in the instance detail view. The matcher has unit tests for wrong status, wrong port, and truncated rows. I could not run it on a GPU.

Non-blocking

  • Stale cached collector: fixed. A cached collector is reused only when it dials the same host. A new contract test fails with the old behavior.
  • Stale throughput shown as fresh: fixed. The collector keeps the last stats and refreshes freshness only when they change.
  • Launch normalization: restored in the engine's launch path. The requested text is not kept separately there, because the serve request has one field. The protocol contract is unchanged.
  • Non-blocking sockets in the Windows test: fixed through a shared fixture that sets blocking mode after accept. I could not reproduce the failure on Linux.
  • run_rocm_bounded: the CLI budget is now the readiness timeout plus a margin.
  • serve-25: now bounded. I did not add an expectations row, because it takes the router route, and a comment explains why.
  • write_dashboard_config: now merges instead of overwriting.
  • Simple-name direct route: no gating scenario. The description names the unit tests that cover it.
  • README on malformed checkpoints: reworded. A malformed registered checkpoint is rejected on Linux with an error. This is intended.
  • Sharded GGUF: confirmed. Two eligible entries in the 11.5.1 catalog are sharded. A complete shard set now resolves to the first shard, and an incomplete set is pulled again.
  • Vulkan readiness: confirmed and fixed. The CLI check now accepts any GPU backend the engine selects, never cpu.

Not changed: the provenance enum, the probe in the tick loop, and the PR split.

Verification on the rebased tree: the full workspace suite passed serially (33 targets, 3,897 tests), and so did both clippy commands, fmt, crate edges, smoke, and the external protocol scenarios. GPU scenarios, the Windows build, and lychee were not run locally.

@volen-silo
volen-silo dismissed rominf’s stale review October 14, 2026 09:20

Addressed in 11b6452. Details are in the reply comment.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch 2 times, most recently from 788a1a3 to 5e0a33d Compare October 14, 2026 14:38
@rominf

rominf commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

🔴 Automated review · pr-review-watcher · 788a1a3

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Informational — this describes commit 788a1a3, not the current head. The head moved to 5e0a33d while this round was running. A range-diff shows 5e0a33d is a rebase onto newer main that switches direct_llama.rs to crate::test_support::scratch_dir, which looks like it resolves the merge-conflict finding below. The other findings were not re-checked against 5e0a33d. This comment does not gate and files no change request. Its blocking findings are open, and the next round checks them against the current head.

Review — needs work

Full review of the whole change (two commits, 40 files, +6629/−741).
No ticket named. The description makes specific claims, and one of them is wrong (see the first Blocking item).

Blocking

  • [standing since 7778139, re-found blind this round] The direct llama-server route still leaks a running server when a step after spawn fails, and both the description and a new doc comment say it doesn't — engines/lemonade/src/process.rs:577 (write_running_state(...)?), process.rs:604 (merge_json_state(...)?), process.rs:~624 (child.wait().context(...)?), engines/lemonade/src/lib.rs:545
    • In serve_direct_llama_server, only the readiness and smoke-check closure kills and reaps the child on error.
    • The three ? returns listed above exit without terminate_pid or child.wait(). std::process::Child does not kill its process on drop. Then serve_http_resolving marks the state failed, so stop and health can no longer find the server, and it stays orphaned on the port.
    • The PR fixed exactly this for the router twin (router_serve.rs:50-52, with the StateWrite, ReadyMerge and WaitServer tests).
    • The description says: "Any error after a serve starts now returns its own error, stops a spawned server, and marks the engine state failed, for the router and the direct route." The new doc comment at lib.rs:545 says "Servers spawned inside serve_with_context are stopped by the code that spawned them." The code governs here, because it is what runs, so both claims are false for the direct route.
    • Two workers found this independently. I confirmed it by reading the function at HEAD.
    • Confidence 85 · logic · Fix: wrap every post-spawn step of serve_direct_llama_server in the same kill-and-reap-on-error block serve_router uses. Add a regression test that makes the state path unwritable after spawn, as Fail::ReadyMerge does. Alternatively, narrow the description and the comment.
  • [new] The branch conflicts with main and cannot merge as is — engines/lemonade/src/direct_llama.rs, engines/lemonade/src/lib.rs
    • git merge-tree prw-base HEAD reports content conflicts in both files, and the API reports mergeable: false, mergeable_state: dirty for head 788a1a3.
    • The cause is test: race-check scratch names in vllm and lemonade (follow-up to #603) #620 (6870dce), which moved the per-module scratch_dir helpers into crate::test_support::scratch_dir. This PR adds yet another local tempfile-based scratch_dir in direct_llama.rs (~L625-636).
    • Confidence 100 · mechanical · Fix: rebase, drop the local helper and use crate::test_support::scratch_dir. Re-run the Windows check after the rebase.
  • [new] The suggested fix in the "matches multiple files" error does not resolve for split GGUF sets — engines/lemonade/src/direct_llama.rs:117-135 (message at L130), first_shard_of_complete_set (~L520), is_incomplete_shard_set (~L541)
    • Setup: a complete split set plus an unrelated file with the same label. The error then suggests the first shard's exact filename, e.g. o/r:model-Q4_K_M-00001-of-00002.gguf.
    • An exact-filename variant matches only that one shard, so the set reads as incomplete and direct_llama_model_path returns Ok(None). The serve then pulls and fails with "did not download". A probe test returned None.
    • serving_variant_errors_carry_concrete_examples checks that the example resolves, but only for non-split files.
    • AGENTS §3: remediation advice must actually clear the condition that printed it.
    • Confidence 100 · logic · Fix: when the matches include a split set, suggest the quantization label or shard folder instead. Add a split case to the test.
  • [new] A shard set can be assembled across cache roots or snapshot revisions — engines/lemonade/src/direct_llama.rs:478-497, 520-538
    • Matches are deduplicated by file name across every root and revision.
    • In a run, shard 1 in root h1 plus shard 2 in root h2 counted as complete and returned h1/.../-00001-of-00002.gguf. llama-server looks for the remaining shards next to shard 1, so the launch fails after a spawn.
    • Separately, removing the (1..=total) index-range check at ~L530 leaves the whole suite green (run), so an out-of-range set such as {00001, 00003}-of-00002 is untested.
    • Confidence 80 · logic · Fix: count a set as complete only when every shard shares the first shard's parent directory. Add tests for the split-root and out-of-range cases.
  • [new] A Lemonade last-request rate stays Fresh indefinitely, contradicting the PR's own contract comment — crates/rocm-dash-daemon/src/runner.rs:941, crates/rocm-dash-daemon/tests/telemetry_contract.rs:910-912, crates/rocm-dash-collectors/src/lemonade.rs:121
    • Every successful stats scrape calls observe_direct(rate, cycle_at), which refreshes last_observed_at. An idle server returning the same tokens_per_second body therefore never goes Held or expires.
    • The doc comment the PR adds at telemetry_contract.rs:910 says "An unchanged stats body is the same request, so it must not keep the rate fresh: once the validity window passes the rate becomes unavailable." The test beneath that comment checks something else (no standing warning on a direct llama-server row), so no test checks the stated contract.
    • The comment governs: it matches parse_stats's own doc ("not a live rate, stays the same until another request completes").
    • The runner logic predates the PR. Its blast radius is new: the PR turns on external discovery by default and starts scraping managed Lemonade.
    • Confidence 80 · logic · Fix: observe only when the stats body (or token counts) changes from the previous scrape for that row, and add the test the comment describes. Otherwise delete the orphaned comment.
  • [new] The Serving preview says nothing is running while external models are serving, and a new test pins that — crates/rocm-dash-tui/src/ui/tabs/pane.rs:471, test at pane.rs:~706-713
    • The preview now counts managed services only but still says "Running now — N". With two external models Ready, it shows "Running now — 0" in the same tab where Engines says "Lemonade — externally running".
    • services_preview_ignores_external_only_instances asserts that text. AGENTS §3: "a pin over a false claim holds the claim in place."
    • Confidence 85 · logic · Fix: relabel it as managed services (e.g. "Managed services running — N", with an external read-only count). Do the same for the overlay's "No managed services." empty state.
  • [new] The empty-state advice "check an external endpoint" points at a Serving action that doesn't exist — crates/rocm-dash-tui/src/ui/tabs/instances.rs:380, crates/rocm-dash-tui/tests/dash_characterization.rs:181-182
    • None of the Serving actions can check or configure an external endpoint, which is set by the dashboard.daemon.lemonade_host/lemonade_port config.
    • The characterization comment says dash-27 proves this message together with the behaviour. dash-27 only navigates to Serving.
    • Confidence 85 · logic · Fix: drop the clause or name the real control (the config keys), then correct the comment.
  • [new] The new manual-testing text describes --managed behaviour the CLI no longer has — docs/manual-testing.md:272-273
    • The text says: "Without it, rocm serve stays attached to the log stream and the stop command above is not reachable."
    • apps/rocm/src/serve_cmd.rs:542: let _ = managed; // background is now the default; --managed is accepted as an explicit synonym. The serve only attaches with --foreground/--verbose, and the rocm serve --help doc comment agrees.
    • Confidence 95 · mechanical · Fix: delete the "Without it…" sentence.
  • [new] SystemInfo.lemond_version is now dead — crates/rocm-dash-core/src/metrics.rs:59
    • The PR removed its only reader (the old Engines line in pane.rs). Its only writer, demo.rs:557, sets it to None.
    • Confidence 70 · mechanical · Fix: remove it, or mark it as a reserved or deprecated wire field if snapshot compatibility needs it.

Standing objections from 7778139 this round did not re-find

This round's blind review did not re-find these, and they were not settled this round. The next round settles them:

  • Pinned CLI strings with no pointer to a behaviour test (help text, startup status note, reinstall advice).
  • Some router-step and pull-step methods tested only through fakes.
  • The alias test and the managed-row e2e check could not fail (the managed-row half overlaps the dash-25 item under Non-blocking).
  • The bare-path citation in docs/testing.md (below the gating floor on its own).

Non-blocking

  • [standing since 93324b5, re-found blind] No GPU or self-hosted lane ran on this head. statusCheckRollup lists only the hosted checks (all green). The router-route change, including removal of the Linux router→direct fallback, has no hardware evidence on this commit. dash-25, serve-08 and serve-25 have not run. The PR text acknowledges this. The dirty state may be what is keeping those lanes off. Confidence 85 · architectural.
  • [standing since 7778139 as blocking, re-found blind and rated non-blocking this round] dash-25 can never gate. The flaky row at expectations.toml:106-111 matches every lane the scenario can run on (@requires-os:linux @requires-engine:lemonade), so its Observe-row and instance-detail assertions are advisory too. docs/testing.md:200-202 only concedes that the "installed" wording is ungated, which implies the other assertions do gate. Confidence 80 · mechanical · Fix: say in the docs that the scenario is advisory, or cover the Observe row and instance detail in a non-GPU scenario with a planted managed record.
  • [new] dash-27's wording is stronger than what it shows.
    • "Lemonade returns to the managed installation status" (dash.feature:382): this scenario never shows the externally-running state first.
    • "stops responding" (dash.feature:380): the step only produces connection-refused, not a hung server.
    • Confidence 70 · mechanical.
  • [new] Three host normalisers have already drifted — registry.rs:230-247 (host_reach, which lowercases), 271-277 (scrape_host) and 302-309 (discover_managed_services). As a result, LOCALHOST matches in endpoints_match but fails cached_collector_reusable. Today the only cost is a rebuilt collector. Confidence 75 · mechanical · Fix: one shared normaliser.
  • [new] Inaccurate comments.
    • runtime_dir.rs:45-46 says rooted_at's "only callers" are in process.rs. It is also called from direct_llama.rs, lib.rs and router_serve.rs.
    • The launch_serve_request doc in lib.rs says "serve-http itself never normalizes", but the router route does normalize.
    • docs/testing.md:215 credits the rocm dash --help test to rocm-dash-tui. It lives in apps/rocm/src/main.rs (dash_help_scopes_lemonade_discovery_to_linux_and_wsl).
    • Confidence 75–90 · mechanical.
  • [new] The two IPv6 contract tests return silently when the bind fails (lemonade_dual_stack_managed_bind_does_not_duplicate_external_row, lemonade_managed_ipv6_record_replaced_by_ipv4_record_on_same_port_recovers). On a host without IPv6, or on Windows where [::] is v6-only, they pass without testing anything. Confidence 70 · mechanical · Fix: emit a skip message or cfg_attr(..., ignore) with a reason.
  • [new] is_external_lemonade keys on container_name == "lemonade" (metrics.rs:229-232). A Docker container a user names lemonade would show as "Lemonade — externally running". Confidence 70 · architectural · Fix: an explicit engine/source field.
  • [new] The lemonade_managed_installed wiring in resolved_args is untested (apps/rocm/src/dash.rs:256-265, 328). Only the pure managed_installed_from_detect is tested, and only dash-26 catches the event_loop.rs:805 assignment. Confidence 70 · mechanical.
  • [new] Pre-existing, made more visible: serve_http_resolving adds a second late failed write. This widens an existing race where a user-initiated stop (writes stopped) can end up recorded as failed. Confidence 70 · logic.

Decisions for the author

  • This PR does two separable things — non-blocking-improvement
    • One is Lemonade serve routing and identity (engines/lemonade). The other is dashboard discovery and ownership (rocm-dash-*, TUI, e2e).
    • At 6.6k lines, each half reviews worse for sharing a PR with the other. Splitting would also let the dash half land without waiting on GPU-lane evidence for the serving half.
  • Removing the Linux router→direct fallback — tradeoff
    • Errors are now honest, and a failed router load no longer hides behind a different serving path.
    • The cost: hosts where the router fails but a direct server would have worked now fail on the first serve.
    • The description states this; confirm it is deliberate given that no hardware lane has run.
  • A malformed registry entry fails the serve on Linux even when no direct binary is installed — tradeoff
    • The ? on resolve_direct_llama_model (direct_llama.rs:48-50) runs before the direct_serve_binary check, so Linux errors where Windows would route to the router.
    • The comment near L378 calls the error deliberate. Confirm it is also meant for hosts where the direct path cannot run.

Positive signals

  • Extracting router serving behind the RouterSteps trait (router_serve.rs) made post-spawn failure paths testable with a fake. Seven single-branch mutations in that area each turned a test red.
  • Ownership is a positive, opt-in managed flag with #[serde(default)], so a legacy snapshot never gains lifecycle control. Tests confirm that id prefixes cannot fake it. dash-26 failed under the is_listed_service → true mutation (run).
  • The e2e Lemonade protocol fixture logs every non-GET-health/stats request, which gives a real negative check against lifecycle calls reaching external servers.
  • The 404-only /v1 fallback never falls back on 401/403/5xx, and that is tested with .expect(0).
  • find_llama_server_binary_for_backend now rejects cpu and unknown backends instead of mapping them to vulkan, so the no-CPU-fallback rule is enforced and tested on every route.
  • The docs/ citations the PR adds are relative links, README stays link-free, and the leak scan of the diff came back clean.

Deployment notes

  • rocm dash now enables read-only local Lemonade discovery by default on Linux and WSL (dashboard.daemon.enable_lemonade defaults to true, 127.0.0.1:13305). Existing users will see external models appear in Observe on upgrade.
  • Docker-discovered containers are no longer listed in the Services overlay, so they can no longer be stopped or restarted from the dashboard.
  • On Linux, a failed router load no longer falls back to a direct llama-server; it now fails the serve.

What this covered

Every file in prw-base...HEAD (6870dce…788a1a3f). Six independent workers covered: lemonade direct_llama/runtime_dir/state; lemonade lib/process/router_serve/test_http; dash collectors/core/daemon plus apps/rocm/src/dash.rs; the dash TUI; main.rs, rocm-core and the docs; the e2e harness and features.

  • Not read end to end: the TUI worker read only the changed regions of app/mod.rs, event_loop.rs, types.rs and instances.rs (1–4 changed lines each in files of 500–3400 lines), plus every user of the new symbols.
  • What was run: lemonade lib tests (166 passed), daemon telemetry_contract (14), the collectors/core/daemon/TUI lib tests, clippy -D warnings on the touched crates, and e2e dash-26/dash-27 (both passed, both failed under targeted mutations). About 25 production mutations in total.
  • Caveat on the runs: the workers ran concurrently in the same disposable checkout, and some mutation runs overlapped another worker's edits. The kills reported were each attributable to that worker's own mutation.

What did not run, and why:

Not reconciled against any PR discussion here. The driver read the author's reply of 09:19Z. The standing objections that reply addresses and this round did not re-find are listed above as unsettled.

rominf
rominf previously requested changes Oct 14, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 5e0a33d

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
No ticket named. The description matches the change closely. It discloses the removed Linux router-to-direct fallback, that the Services overlay now hides Docker containers, and the scenario gaps it knows about.

Round 6. This round reviewed 5e0a33d in full and blind, because the previous review point, 788a1a3, is no longer in the branch. Each finding is marked [new] or [standing]. Standing objections the blind pass did not re-find were each put to the reviewer afterwards. That check is anchored: the reviewer had been told the objection. Each is listed below with its outcome.

Blocking

  • [standing since 788a1a3, re-found blind] An idle Lemonade's last-request rate stays "Fresh" forever, against the contract the PR's own test file states — crates/rocm-dash-daemon/src/runner.rs:932-941, crates/rocm-dash-daemon/tests/telemetry_contract.rs:910-912, crates/rocm-dash-collectors/src/lemonade.rs:117-122
    • /api/v1/stats tokens_per_second is the rate of the most recent request. The PR's own doc on parse_stats says it "stays the same until another request completes".
    • The runner calls observe_direct(rate, cycle_at) on every stats scrape, and observe_direct re-stamps last_observed_at without checking whether the body changed. So an idle server's old rate shows as Fresh on every cycle and never reaches the validity-window expiry.
    • The doc comment added in 5e0a33d says the opposite: "An unchanged stats body is the same request, so it must not keep the rate fresh: once the validity window passes the rate becomes unavailable". No code does this. The test the comment sits on (lemonade_managed_direct_llama_server_row_has_no_standing_warning) tests something else.
    • The comment governs: it is the only statement of intent, and it agrees with parse_stats' own description of the value.
    • The base branch had the same weakness for external discovery, but discovery was off by default there. This PR turns it on by default and scrapes managed Lemonade services for the first time, which widens the blast radius.
    • dash-27's external_single_model_rate step asserts "38" tok/s from a fixture that serves no requests.
    • Confidence 85 · logic · Fix: call observe_direct only when the stats body changes (output tokens or decode times differ), and add the test that comment describes. Or remove the comment and label the value "last request".
  • [standing since 7778139, re-found blind] The new remediation advice is pinned as wording only, and its own comment says no test proves it — engines/lemonade/src/direct_llama.rs:67-75, :712-733
    • The error tells the user to run rocm engines install lemonade --reinstall to restore a missing llama-server binary.
    • The test comment says: "This pins the wording only … NO test proves that the llama-server binary is back afterwards: that needs a real backend download, which no scenario asserts."
    • AGENTS.md §3 requires remediation advice to be asserted together with the command actually clearing the condition. A wording-only pin is allowed only when it names a test that proves the behaviour.
    • The comment's reason does not hold. @id:runtime-lemonade-backend-alignment-reported (runtime-16, @nightly @requires-gpu) already runs that exact reinstall with a real backend download. One added Then there would prove the binary is present afterwards.
    • Confidence 80 · mechanical · Fix: add a step to runtime-16 asserting that the selected backend's llama-server exists after the reinstall (or that a direct serve then starts). Point the pin's comment at that step.
  • [standing since 7778139, not re-found blind; confirmed in full by the follow-up check] The direct llama-server route leaves a running server behind when a step after spawn fails, while a doc comment says it doesn't — engines/lemonade/src/process.rs (serve_direct_llama_server, from L504), engines/lemonade/src/lib.rs:547-548
    • After command.spawn(), three ? returns exit without stopping the child: write_running_state(..., Some(child.id()), "running")?, merge_json_state(&request.state_path, …"ready"…)? and child.wait().context(…)?. Only the readiness/smoke closure's error arm calls terminate_pid and child.wait().
    • std::process::Child does not kill the process when dropped. serve_http_resolving then marks the state failed, so neither stop nor health can reach the server that is still running.
    • The doc comment on serve_http_resolving says "Servers spawned inside serve_with_context are stopped by the code that spawned them". That is false for these three sites. The PR description says the same for "the router and the direct route". The author's reply on 7778139 conceded the gap, and the code has not changed since.
    • The router twin got this fix (router_serve.rs StateWrite/ReadyMerge/WaitServer tests). No test makes a post-spawn step fail on the direct route.
    • Confidence 85 · logic · Fix: put one terminate-and-reap guard around every post-spawn step, as serve_router does. If wait() itself fails, still attempt terminate_pid. Add a failure-matrix test for the direct route.
  • [standing since 788a1a3, not re-found blind; confirmed by a run in the follow-up check] The suggested fix in the "matches multiple files" error does not resolve for split GGUF sets — engines/lemonade/src/direct_llama.rs (ambiguity error ~L117-135, first_shard_of_complete_set, is_incomplete_shard_set)
    • Setup: the existing split_gguf_variant_resolves_to_its_first_shard fixture, which has a complete two-shard set plus other-Q4_K_M.gguf. The error suggests …:model-Q4_K_M-00001-of-00002.gguf. Fed back through direct_llama_model_path, that suggestion returns Ok(None).
    • The reason: an exact-filename variant matches only that one shard, so the set reads as incomplete.
    • The serve would then pull and fail with "did not download". That part was read from the code, not run.
    • serving_variant_errors_carry_concrete_examples only covers non-split files. AGENTS.md §3 requires that remediation advice actually clears the condition that printed it.
    • Confidence 100 · logic · Fix: when the matches include a split set, suggest its quantization stem instead. Or let an exact shard filename resolve to its whole set. Add a split case to the test.
  • [standing since 788a1a3, not re-found blind; confirmed by runs in the follow-up check] A shard set can be assembled across cache roots or snapshot revisions — engines/lemonade/src/direct_llama.rs (dedup ~L484-497, collect_hf_checkpoint_ggufs ~L459-468, first_shard_of_complete_set ~L520)
    • Matches are deduplicated by file name only. Shard 1 in root h1 plus shard 2 in root h2 counted as complete and returned the h1 shard. The same happened with shard 1 under snapshots/revA and shard 2 under snapshots/revB.
    • llama-server looks for the remaining shards next to shard 1, so the launch fails after spawn. On the direct route it then also hits the cleanup gap above.
    • Deleting the (1..=total) index-range check leaves the engine suite green (166 passed), so the {00001, 00003}-of-00002 case has no test.
    • Confidence 85 · logic · Fix: count a set as complete only when every shard shares the first shard's parent directory. Add the cross-root and out-of-range cases as tests.
  • [standing since 788a1a3, not re-found blind; preview half confirmed, overlay half dropped] The Serving preview says nothing is running while external models are serving, and a new test pins that — crates/rocm-dash-tui/src/ui/tabs/pane.rs:463-477, test services_preview_ignores_external_only_instances (L706-713)
    • The preview now counts only managed, serving instances (is_listed_service(i) && i.status.is_serving(); base counted every serving instance). The label was not changed and still reads "Running now — {n}" / "none running".
    • With two external models Ready, the Serving tab says "Running now — 0 / none running", while the Engines pane in the same tab says "Lemonade — externally running". The new test asserts exactly that text. AGENTS.md §3: "a pin over a false claim holds the claim in place."
    • The overlay's "No managed services." empty state is correctly scoped and is proven by dash-26. That half of the earlier objection is withdrawn.
    • Confidence 85 · logic · Fix: relabel the preview as managed (e.g. "Managed running — N" / "no managed services running") and update the pinned strings.
  • [standing since 788a1a3, not re-found blind; confirmed by the follow-up check] The new manual-testing text describes --managed behaviour the CLI no longer has — docs/manual-testing.md:278
    • The doc says: "Without it, rocm serve stays attached to the log stream and the stop command above is not reachable." But apps/rocm/src/serve_cmd.rs:542 reads let _ = managed; // background is now the default; --managed is accepted as an explicit synonym., and only --foreground/--verbose attach (both conflicts_with = "managed").
    • AGENTS.md §5 requires docs to match the code path.
    • Confidence 95 · mechanical · Fix: delete that sentence, or say that --managed is the default.
  • [standing since 788a1a3, not re-found blind; confirmed by the follow-up check] SystemInfo.lemond_version is now dead — crates/rocm-dash-core/src/metrics.rs:59, demo.rs:557
    • The PR removed its only reader. Its only writer is the demo, which sets it to None.
    • Removing it is wire-safe: there is no deny_unknown_fields, and a missing Option deserializes as None.
    • Confidence 85 · mechanical · Fix: remove the field and its demo initializer. The siblings llama_server_build and ccr_version were already dead before this PR and can go in a separate change.

Non-blocking

  • [new] First contact with the external endpoint swallows every error, not just "nobody is listening" — crates/rocm-dash-daemon/src/runner.rs:879-880
    • lemonade_discovery_warn = (!endpoint.rows.is_empty()).then(...) stays quiet on 401, 403, 5xx and body errors whenever the endpoint has never been seen. So a Lemonade server that is present but needs auth, or is broken, produces no warning.
    • docs/testing.md promises only that "a never-seen absent endpoint remains quiet".
    • Confidence 80 · logic · Fix: stay quiet only on connection-refused or transport errors.
  • [standing, re-found blind] dash-25's expectations row makes the whole journey advisory on every lane that can run it — tests/e2e-cucumber/expectations.toml:103-111
    • @requires-engine:lemonade forces effective_engine = "lemonade", and the scenario is @requires-os:linux. So the row {effective_engine="lemonade", os="linux"} with flaky = true matches every eligible lane.
    • As a result, the dashboard, instance-detail and post-quit inference checks never gate a merge. The description concedes this only for the installed wording.
    • Confidence 80 · logic · Fix: keep only the serve step under the flaky scope, or say plainly that dash-25 is advisory everywhere.
  • [new] No scenario covers the positive half of the new ownership rule — crates/rocm-dash-tui/src/ui/services_manager.rs:95-105, tests/e2e-cucumber/features/dash.feature
    • dash-26 proves that external rows are excluded from the Services overlay.
    • No scenario opens Services with a live managed service and sees it listed and stoppable. server-records-02 covers only non-running records.
    • Unit and contract tests do catch a dropped managed flag (mutation run).
    • Confidence 75 · mechanical · Fix: extend dash-07 to open the overlay, assert the managed row, and stage a stop approval.
  • [new] Serving behaviour the CLI now chooses differently is covered only by unit tests — apps/rocm/src/main.rs:15578-15598, engines/lemonade/src/direct_llama.rs (serving_route)
    • Two choices changed: Vulkan-backed router models now read as ready, and simple registered names now take the direct route on Linux.
    • AGENTS.md §3 treats which runtime the CLI selects as user-observable. serve-lemonade-inference exercises the direct route, but it neither asserts the route nor gates on Linux.
    • Confidence 70 · mechanical · Fix: assert the route (load_response.method) in serve-lemonade-inference, and name or add the lane that covers router+Vulkan readiness.
  • [new] serve-25 cannot tell old behaviour from new on Windows — tests/e2e-cucumber/features/model_serving.feature:338
    • Registered names took the router on Windows both before and after this change.
    • Confidence 75 · logic · Fix: tag the scenario @requires-os:linux, or say that its Windows run only re-confirms existing behaviour.
  • [new] run_rocm_bounded was applied to only some of its twin serve call sites — tests/e2e-cucumber/tests/e2e/serving_steps.rs:557, :928, :982
    • setup_gpu_model and two serve --managed calls still use the unbounded run_rocm.
    • Confidence 75 · mechanical · Fix: use run_rocm_bounded(..., serve_cli_budget(...)) at those sites as well.
  • [new] The isolated dash scenarios now probe a real 127.0.0.1:13305 by default — crates/rocm-core/src/lib.rs (default_dashboard_enable_lemonade), tests/e2e-cucumber/tests/e2e/dash_steps.rs
    • Only dash-25 turns discovery off.
    • dash-03/04/07/08/09 and the replay/GPU scenarios do not. On a self-hosted lane with Lemonade on 13305 they gain extra rows and warnings.
    • Confidence 70 · mechanical · Fix: default the isolated e2e config to enable_lemonade: false.
  • [standing, re-found blind] The IPv6 contract tests return early and pass silently when IPv6 is unavailable — crates/rocm-dash-daemon/tests/telemetry_contract.rs (lemonade_dual_stack_managed_bind_does_not_duplicate_external_row, …ipv6_record_replaced_by_ipv4_record…)
    • Confidence 75 · mechanical · Fix: print a skip, or mark the tests ignored when the bind fails.
  • [standing, re-found blind] Instance::is_external_lemonade keys on container_name == "lemonade" — crates/rocm-dash-core/src/metrics.rs:230-231, crates/rocm-dash-tui/src/ui/tabs/pane.rs:394-397
    • An unmanaged Docker container named lemonade makes the Engines pane say "externally running". The effect is display-only.
    • Confidence 70 · logic · Fix: carry an explicit provenance or engine field.
  • [standing, re-found blind] No self-hosted GPU-lane check has reported on this head. The status rollup shows 23 hosted checks, all green, with Skill checks skipped, and no GPU lane. dash-25, serve-08 and serve-25 have not run anywhere for 5e0a33d. This is missing evidence, not a defect. Confidence 85 · mechanical.

Standing objections settled this round without being kept

These checks are anchored: each objection was put to the reviewer after its blind pass ended.

  • Merge conflict with main: resolved. The API reports mergeable: true for 5e0a33d.
  • Empty-state advice "check an external endpoint" points at a Serving action that does not exist: not kept. The Engines action in Serving does show live external availability. The dash_characterization comment is accurate, because dash-27 asserts the empty message and that "externally running" clears. What remains is that nothing in the TUI names the probed host/port or the dashboard.daemon.lemonade_* keys. That is an observation. The next round re-reads instances.rs blind.
  • Pinned CLI strings with no pointer to a behaviour test (help text, startup status note): does not hold. Both now point at tests that prove the behaviour: dash-26, and managed_install_verdict_is_not_changed_by_live_events. The reinstall-advice part is kept, as the second Blocking item above.
  • Router and pull steps tested only through fakes: does not hold. Real-step tests against a loopback fixture and a fake executable exist and catch the alias/router-name mutations. As the author notes, "a router failure never spawns a direct server" is still guaranteed only by how the code is structured.
  • The alias test and the managed-row e2e check could not fail: does not hold. Changing --alias failed 2 of the 9 alias tests (run). The instance-detail matcher rejects a wrong status, port or model in its unit tests. dash-25 being advisory on every lane is the separate Non-blocking item above.

Decisions for the author

  • Linux direct serving does not apply the router's per-model launch defaults — tradeoff
    • Direct serving passes -m, --n-gpu-layers 999, --alias and the recipe flags. A simple catalog entry can therefore run with different settings, such as context size, on Linux than the same name gets through the router on Windows. In return, direct serving gives an exact alias and avoids the router.
    • One comment in serving_route on why Windows stays on the router would let a reader confirm the split is deliberate.
  • Direct-served models still get a /v1/unload on stop — tradeoff
    • llama-server has no such route, so engine_stop reports an error, and the process is then stopped by PID.
    • The behaviour is pre-existing, but this PR makes direct serving reachable for registered names.
  • The PR bundles three changes — non-blocking-improvement. The three are the serving-route and identity rework, default-on external discovery in the dashboard, and the Services ownership model. Each would review and revert more safely on its own.

Positive signals

  • Ownership is a positive managed flag that defaults to false (#[serde(default)]), so legacy snapshots fail safe: controls are hidden rather than wrongly offered. A staged approval whose target vanishes or turns external is dropped both on keypress and at draw time.
  • The external-protocol fixture records every request that is not a GET of health or stats, and dash-26 asserts the log is empty after the stop/restart keys and again after quit. That is the proof of behaviour AGENTS.md asks for.
  • router_serve.rs hides the router steps behind a trait, so the failure-matrix tests check stop-then-failed state for every step.

Deployment notes

  • rocm dash on Linux/WSL now polls 127.0.0.1:13305 on every discovery tick by default. To opt out, set dashboard.daemon.enable_lemonade: false.
  • On Linux, a failed router load no longer falls back to a direct llama-server, so affected hosts get an error on the first serve.
  • The Services overlay no longer lists Docker-discovered containers.
  • Version skew: a new rocm dash that attaches to an older dash's still-running embedded daemon gets snapshots with no managed field. It then shows "No managed services." until the old dash exits.

What this covered

  • Read: every file in 21be0034…5e0a33d9 (prw-base...HEAD, 40 files), plus callers and siblings: backend_alignment.rs, install.rs, observation.rs, the readiness and stop paths in main.rs, and local_server_records.feature.
  • Fan-out: two coupled-group workers (Lemonade engine/serving; dashboard/telemetry/docs), then one design worker.
    • The engine worker split off a read-only sub-worker for the e2e files and the main.rs / rocm-core hunks. That sub-worker read only the serving-related sections of e2e.rs.
    • The dashboard worker read event_loop.rs, app/mod.rs, app/types.rs and instances.rs selectively, because their changes are one-line field plumbing.
    • The history and prior-changes passes were run by the consolidating context and were not independent.
  • Runs:
    • cargo test -p rocm-engine-lemonade --lib: 167 passed, and mutations M1–M7 were all caught.
    • The telemetry_contract Lemonade tests: 14 passed, and every mutation was caught except one equivalent mutant.
    • The services_manager and pane mutations were all caught.
    • cargo xtask e2e -- -n 'External Lemonade protocol': 2 scenarios, 31 steps passed.
    • The follow-up objection checks ran the split-shard, cross-root, index-range and alias probes.
  • Did not run: any GPU scenario (dash-25, serve-08, serve-25), because there is no GPU here; Windows cfg arms (CI's windows-build-and-test passed); mutations for the e2e step files; local clippy (CI's passed).
  • No prompt-injection content was found.
  • The findings were not reconciled against the PR discussion. Only the standing objections above were weighed against the author's replies.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from 5e0a33d to aa28578 Compare October 15, 2026 08:45
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for round 6. I pushed aa28578d, rebased on current main. Here is the status of each item.

Blocking

  • Idle rate stays fresh: not fixed. I tried to stop an unchanged stats report from refreshing the rate. It made dash-lemonade-external-protocol-transitions flaky on hosted CI, so I reverted it. I removed the comment that claimed this behaviour and rewrote the docs to say what the code does: the last request's rate stays shown until a new request or the row goes away. The description says so. A real fix needs a scenario that varies the stats, so I left it for its own change. External rows already behaved this way.
  • Reinstall advice: fixed. runtime-lemonade-backend-alignment-reported now ends with a step that checks the selected backend's llama-server exists after rocm engines install lemonade --reinstall. The helper that finds the binary has unit tests. The scenario needs a GPU and a real download, so I could not run it. The pinned test comment now names that step.
  • Direct route leaves a server running: fixed. All post-spawn steps now sit inside one guard that stops the child, reaps it, and marks state failed, including when wait() itself fails. New tests inject each failure with a fake llama-server and check that the process is gone and the state is failed. Mutating the guard makes them fail.
  • Split GGUF suggestion: fixed. An exact shard filename now resolves to its whole set. A test feeds the suggestion from the real error back through resolution.
  • Shard set across roots or revisions: fixed. A set counts as complete only when every shard is in the first shard's directory. New tests cover cross-root, cross-revision, and out-of-range indexes. One limit remains: matches are still deduplicated by file name, so a complete set in a later root is not found if an earlier root holds a same-named shard. The result is a clean error, not a launch that fails after spawn.
  • Serving preview label: fixed. It now reads "Managed running — N" and "no managed services running", and the pinned strings changed.
  • --managed doc: fixed. It says --managed is the default and only --foreground or --verbose attach.
  • lemond_version: removed, with a test that an older snapshot carrying the key still loads.

Non-blocking

  • First contact with the external endpoint: it now warns on a 401, a 5xx, or a body error. It stays quiet only when the connection is refused.
  • dash-25 row: it is advisory on every lane that can run it. The docs and description now say so. I cannot scope the row to one step.
  • Positive ownership: dash-07 now opens the Services overlay, sees the managed service listed with its port, and stages a stop approval, then declines it. It passes on the hosted lane.
  • serve-25 is now @requires-os:linux. The other serve --managed sites now use the bounded run.
  • Isolated dash scenarios now turn Lemonade discovery off unless a scenario sets it. dash-26 and dash-27 turn it on.
  • The IPv6 tests now print a skip line when they cannot bind.
  • The comment on why Windows registered names stay on the router is added.
  • Not changed: the Docker container named lemonade display case, a scenario that asserts the Linux direct route, and the /v1/unload call on stop for direct servers. The first is display-only. The other two need hardware runs.

dash-25 had a real bug that the flaky row hid. On the GPU lane my instance-detail check failed on the real screen, because a panel border touches the model: label. I fixed the matcher and added a fixture captured from that run.

Verification: the full workspace suite passed serially (33 targets, 3,993 tests), and so did both clippy commands, fmt, crate edges, smoke, and the external protocol scenarios. After the last two rebases onto main I re-ran the affected crates and clippy. I could not run the GPU scenarios, the Windows build, or lychee locally.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from aa28578 to dd0ac46 Compare October 15, 2026 09:15
@rominf

rominf commented Oct 15, 2026

Copy link
Copy Markdown
Collaborator

🔴 Automated review · pr-review-watcher · aa28578

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Informational: this review describes aa28578, not the current head. The head moved to dd0ac46 while this round was running, and the code differs. These findings are not anchored to the current diff, are not a binding round, and file no new change request. The change request already standing from round 6 is untouched and still gates. The next round reviews dd0ac46 and settles everything below. Until then, a blocker first found here at aa28578 is not gating.

Review — needs work

Full review of the whole change.
No ticket named. The description matches what the diff does: catalog resolution, removal of the router-to-direct fallback, persisted served identity, read-only dashboard discovery, and registry-only ownership.

Round 7 (at aa28578). This round reviewed aa28578 in full and blind, because the previous review point, 5e0a33d, is no longer in the branch after the rebase. Each finding is marked [new] or [standing]. The eight blocking objections from round 6 were not re-found blind as written, so each was put to the reviewer afterwards. That check is anchored: the reviewer had been told the objection and the author's reply. One still holds in part and is kept below. The other seven are listed at the end of Non-blocking with their outcome.

The failing check at aa28578 is caused by this PR. windows-build-and-test fails on lemonade_never_seen_absent_external_endpoint_is_quiet. That test does not exist on main, and main's latest completed CI runs are green.

Blocking

  • [new] The required windows-build-and-test check fails on a test this PR adds — crates/rocm-dash-daemon/tests/telemetry_contract.rs:477 (lemonade_never_seen_absent_external_endpoint_is_quiet), crates/rocm-dash-collectors/src/lemonade.rs:122-137, crates/rocm-dash-daemon/src/runner.rs:356-360,797-801,881-884
    • The job log for head aa28578 shows 24 passed, 1 failed. The panic is "normal absent external endpoint is not a noisy warning".
    • Likely cause, traced but not run on Windows: the collector has a 1500 ms total timeout. On Windows, a connection to a closed localhost port is retried for about 2 s rather than refused at once. So reqwest reports a timeout, not is_connect(). send_error then maps it to Transport, absent is false, and a warning is raised.
    • The test does not exist on prw-base.
    • Confidence 100 (CI run) · logic
    • Fix: native Windows has no live dashboard per the README, so cfg-gate these daemon contract tests to unix. Alternatively, set a short connect_timeout and classify connect timeouts explicitly.
  • [new] A managed Lemonade service with an API key shows a permanent "lemonade health … 401" warning and never shows throughput — crates/rocm-dash-daemon/src/runner.rs:817-822,887, crates/rocm-dash-collectors/src/lemonade.rs:258-311
    • This PR newly scrapes managed Lemonade endpoints, and the collector sends no credentials.
    • A public-bind serve gets a minted key: engines/lemonade/src/process.rs:263-270 sets LEMONADE_API_KEY, which gates /api and /v1.
    • A managed record run against a 401 fixture produced, on every snapshot: lemonade health 127.0.0.1:PORT [managed-lemonade]: transport: GET …/api/v1/health: status 401 Unauthorized. That real lemond answers 401 here is taken from the engine's own comment, not from a live server.
    • Confidence 85 · logic
    • Fix: present the CLI-owned key for registry targets, or treat 401/403 on a managed target as "throughput unavailable" with no standing warning. Add a regression test.
  • [new] A stale partial download in an older snapshot revision makes a complete split model permanently unservable — engines/lemonade/src/direct_llama.rs around 599-616 (hf_checkpoint_gguf_matches_in_roots), 639-677 (first_shard_of_complete_set / is_incomplete_shard_set), 359 (ensure_model_with_download)
    • Matches are de-duplicated by file name across all snapshot revisions, in sorted path order. So revision aaa/…-00001-of-00002 shadows the bbb/…-00001 of a complete set. The kept shards then span two directories, the set reads as incomplete, and resolution returns Ok(None).
    • The re-pull resolves to None again, and the serve fails with "did not download".
    • The shard logic is new in this PR.
    • A probe test reproduced it: aaa holds shard 1 only, bbb holds both shards, and the result is Ok(None).
    • Confidence 100 · logic
    • Fix: group matches per snapshot directory, and choose a complete set within one directory.
  • [new] The "previously seen external endpoint went away" warning has no test — crates/rocm-dash-daemon/src/runner.rs:881
    • Mutating absent to drop its endpoint.rows.is_empty() half silences that warning, and all 16 Lemonade telemetry_contract tests still pass. dash-27 only checks that the rows disappear.
    • The code comment promises the code "reports why they disappeared". AGENTS.md §3 requires a message to be asserted together with its behaviour.
    • Confidence 100 (mutation survived) · mechanical
    • Fix: add a daemon test in which the endpoint refuses connections after rows were seen, and expect the warning.
  • [standing since 788a1a3, not re-found blind; the follow-up check found it holds in part] An idle Lemonade's last-request rate is still presented as a live, fresh rate — crates/rocm-dash-daemon/src/observation.rs (observe_direct, ~L249), crates/rocm-dash-daemon/src/runner.rs (stats scrape), crates/rocm-dash-tui/src/ui/tabs/serving.rs:57
    • Resolved: the comment that claimed the rate expires is gone. The test doc (telemetry_contract.rs ~L971) and the collector doc (lemonade.rs ~L144) now say what the code does: the last request's rate stays shown.
    • Still holds: observe_direct re-stamps last_observed_at on every scrape. An idle server's old tok/s therefore always reads as fresh, and the held/stale legend can never fire for it. The TUI shows it as a plain tok/s figure (e.g. the "… status · tok/s" header), with no "last request" qualifier there, in README.md, or in docs/testing.md. A user watching an idle server still sees what looks like current throughput.
    • Round 6 offered two remedies; the author's reply explains why the first was reverted. The second was half done: the comment is gone, but the value is not labelled.
    • Confidence 80 (anchored check, read not run) · logic
    • Fix: label the value where it is displayed (for example "last req tok/s", or a held/stale marker once no new request has completed), or stop re-stamping when the stats body is unchanged.

Non-blocking

  • [new] Typo in the scenario tag line: @id:runtime-lemonade-backend-alignment-reported@requires-gpu — tests/e2e-cucumber/features/runtime_setup.feature:340
    • The gherkin parser still splits this into two tags, so the GPU gate holds and feature_naming passes (4 tests ran).
    • Whitespace-splitting readers, such as feature_tags in expectation.rs, see a different id.
    • Confidence 85 · mechanical · Fix: insert the missing space.
  • [new] runtime-16's new last step never starts from the missing-binary condition — tests/e2e-cucumber/tests/e2e/runtime_steps.rs (assert_lemonade_backend_server_installed)
    • It proves the binary exists after a reinstall, not that the reinstall restores a missing one. Restoration holds only by reading install.rs remove_dir_all.
    • Confidence 70 · mechanical · Fix: delete the selected backend's binary before the reinstall step.
  • [new] The direct readiness check now matches names exactly, but its healthcheck twin still normalizes them — engines/lemonade/src/process.rs around 762 (models_payload_has_loaded_model) vs around 782 (models_payload_has_ready_model)
    • A direct server can fail the serve-time check yet read as ready in healthcheck.
    • Confidence 70 · logic · Fix: match direct entries (those without recipe_options) exactly in both.
  • [new] The engine picks a nested llama-server build directory in read_dir order, while the e2e copy of the lookup sorts — engines/lemonade/src/direct_llama.rs around 325-337 (find_binary_in), tests/e2e-cucumber/src/lemonade_backend.rs (binary_in)
    • The function predates this PR, but direct serving is now the default Linux route for catalog names.
    • Confidence 70 · logic · Fix: sort, or prefer the newest build.
  • [new] is_connect() is treated as "nothing listening", so DNS failures and connect timeouts are silent — crates/rocm-dash-collectors/src/lemonade.rs:122-129
    • A mistyped lemonade_host never warns.
    • Confidence 70 · logic · Fix: treat only a ConnectionRefused found in the error's source chain as "nothing listening".
  • [new] The fixture's premise "a plain llama-server: neither Lemonade health route exists" looks wrong — crates/rocm-dash-collectors/src/lemonade.rs:416
    • Real llama-server serves /v1/health and returns 503 while loading. So a managed direct server would show a transient health warning that the fixture, which serves only /health, cannot reveal.
    • This rests on recall of llama.cpp, not on a run.
    • Confidence 70 · mechanical · Fix: serve /v1/health in the fixture, including a 503 phase.
  • [new] HTTP status errors are labelled "transport:" in warnings — crates/rocm-dash-collectors/src/lemonade.rs (status mapped to CollectorError::Transport)
    • This is visible in the 401 output above.
    • Confidence 85 · mechanical
  • [new] Stale test name — crates/rocm-dash-collectors/src/lemonade.rs:680 (discover_is_clean_noop_when_endpoint_absent)
    • The discover() it names was removed, and the test now asserts an error.
    • Confidence 85 · mechanical
  • [standing, re-found blind] The IPv6 daemon tests skip with eprintln! + return and report ok — lemonade_dual_stack_* and lemonade_managed_ipv6_record_replaced_* in telemetry_contract.rs
    • They ran on this host. On a host without IPv6 they are a silent false green.
    • Confidence 70 · mechanical · Fix: use #[ignore] with a reason, or fail loudly in CI.
  • [new] The leak this PR fixes for Lemonade is still in vLLM (pre-existing, pattern relevance) — engines/vllm/src/process.rs around 118-120 and 147-148
    • write_running_state(...)? runs after the spawn, and a child.wait() error does not mark the state failed.
    • Confidence 85 · logic · Fix: a follow-up issue or PR.
  • [standing] No self-hosted GPU-lane check ran for aa28578. The removed router-to-direct fallback, Vulkan readiness acceptance, dash-25, serve-08 and runtime-16 have no hardware evidence there. The description says the same. This is missing evidence, not a defect. Confidence 85 · architectural.

Standing objections settled this round without being kept

These checks are anchored: each objection was put to the reviewer, with the author's reply, after the blind pass ended.

  • Direct route leaves a server running after a post-spawn failure: no longer holds. supervise_direct_server puts recording, readiness and the wait under one stop-and-reap guard. Removing the stop on the wait-error path fails direct_server_step_failure_stops_the_child_and_marks_state_failed (mutation run).
  • The split-GGUF suggestion does not resolve: no longer holds. Breaking the shard branch fails split_set_ambiguity_suggestion_resolves_to_the_whole_set (mutation run).
  • Shard set assembled across roots or revisions: no longer holds as written. Dropping the same-directory check fails the cross-root and cross-revision tests (mutation run). The out-of-range case is tested by shard_indexes_outside_the_declared_range_are_not_a_complete_set. The stale-partial-revision case is the separate new Blocking item above.
  • --reinstall advice pinned as wording only: no longer holds as filed. runtime-16 now ends with the requested binary-present step, and the pin names it. The residual is the Non-blocking item above.
  • "Running now — 0" in the Serving preview: no longer holds. It now reads "Managed running — N" / "no managed services running", and the tests are updated.
  • Old --managed text in manual-testing: no longer holds. docs/manual-testing.md:278-280 now matches the CLI.
  • Dead SystemInfo.lemond_version: no longer holds. It was removed, and gpu_system_info_ignores_retired_lemond_version_key covers old snapshots.

Decisions for the author

  • Removing the router-to-direct fallback — tradeoff
    • Failures are now honest, and a failed serve stops its server and marks the state failed.
    • The cost is that a host where the router load fails, but a direct server would have worked, now gets an error on the first serve. The description states this. Confirm it is intended for Linux users on the default model.
  • Scope — non-blocking-improvement
    • The change is about 7.5k lines across engine catalog resolution, router error semantics, daemon telemetry, TUI ownership and docs. AGENTS.md §11 asks for one logical change per PR.
    • The serving half and the dashboard half share little beyond canonical_model_id. They would review and bisect better as two PRs.
  • About 250 lines of Lemonade endpoint logic added inline in run_loop (crates/rocm-dash-daemon/src/runner.rs:709-951) — non-blocking-improvement
    • AGENTS.md §6 prefers a domain module, such as a lemonade_scrape module.

Positive signals

  • Every step after a spawn (router, direct server, download lemond) runs under one guard that stops and reaps the child. Each path has a failure-injection test, and 18 production mutations were each caught by a named test.
  • Ownership defaults to managed: false through serde, so older snapshots and replays cannot grant stop or restart. A staged approval is re-checked against current ownership before any verdict.
  • The dash-26 fixture logs every request that is not a health or stats GET. That proves the dashboard takes no lifecycle action against an external server, rather than only asserting the wording.
  • Throughput is attributed only when exactly one model is loaded, instead of inventing a per-model rate.

Deployment notes

  • Behaviour change: on Linux, a failed Lemonade router load no longer falls back to a direct llama-server. The serve fails instead.
  • rocm dash now probes 127.0.0.1:13305 by default. The new dashboard.daemon.enable_lemonade, lemonade_host and lemonade_port keys in config.json control this.
  • The Services overlay now lists only registry-managed services. Docker-discovered containers can no longer be stopped or restarted from the dashboard.

What this covered

  • Read: every changed file in prw-base...aa28578d (merge base f9a868a), completely, by two parallel read-only workers. One took the Lemonade engine, rocm-core, main.rs and the serving and runtime e2e. The other took the dash crates, dash.rs and the dash e2e. Docs, history and the agent-instruction pass were done by the review driver. git merge-tree against prw-base is clean.
  • Ran:
    • rocm-engine-lemonade --lib (177 passed), with 19 single-branch mutations.
    • rocm-dash-daemon --test telemetry_contract, plus the collector, services_manager, pane and dash unit tests, with 24 mutations.
    • e2e-cucumber --test feature_naming (4 passed).
    • The windows-build-and-test job log was read, and the failing test was checked against main.
  • Did not run:
    • The cucumber e2e suite (dash-03/25/26/27, serve-08/25, runtime-16).
    • A Windows-target cargo check, because no mingw linker is available here. Windows cfg arms were reviewed by reading only.
    • Real lemond or llama-server behaviour, so the 401 and /v1/health claims rest on the engine's comments and on recall.
    • The prior-changes pass.
    • The Step 7 design questions were investigated by the review driver, not by independent workers, because of a 2-worker cap.
  • Caveat: both workers mutated the same checkout concurrently. A small window of cross-interference between their mutation verdicts cannot be excluded.
  • CI rollup at aa28578: every check passes except windows-build-and-test. Skill checks were skipped.
  • Findings were reached without the PR discussion. Only the standing objections were weighed against the author's reply, which was acknowledged before this round.

rominf
rominf previously requested changes Oct 15, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · dd0ac46

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
No ticket named. The description is detailed and accurate except for the reinstall-proof claim (the fourth Blocking item).

Round 8 (at dd0ac46). This round reviewed dd0ac46 in full and blind, because the previous review point, aa28578, is no longer in the branch after the rebase. Each finding is marked [new] or [standing]. The five objections carried from round 7 (posted as an informational comment at aa28578) were not re-found blind, so each was put to the reviewer afterwards, one at a time, with the author's reply where there was one. That check is anchored: the reviewer had been told the objection. Four still hold and are appended under Blocking; one no longer holds and is listed at the end of Non-blocking. This change request replaces the round-6 one as this account's complete position.

The fourth new Blocking item (reinstall "proof") is the residual round 7 rated non-blocking. It is reported as the method found it, but it does not on its own decide this change request; the other blocking items do.

Blocking

  • [new] The new router serve path's real stop and its failed-exit branch are never tested — engines/lemonade/src/router_serve.rs:121-133, engines/lemonade/src/router_serve.rs:220-225, engines/lemonade/src/process.rs:686-698
    I made two changes together: turned LemondRouterSteps::stop_server into a no-op, and made a non-zero wait_server status return Ok(()). cargo test -p rocm-engine-lemonade --lib still passed all 177 tests. FakeSteps::wait_server always succeeds, and no test spawns a real lemond and then stops it. The architecture doc and the PR say this new module stops the server and marks state failed on any error. Neither claim is tested.
    Confidence 100 · logic · Fix: add a failing-exit variant to FakeSteps that asserts both the error and state failed. Add a Linux test that spawns the fake lemond (exec sleep 30), calls the real stop_server, and asserts the process is gone.

  • [new] New test helpers write fake executables with std::fs::write instead of the shared ETXTBSY-safe helper — engines/lemonade/src/test_http.rs:112-116 (write_executable), plus the fake_pull_manifest, fake_llama_server and inline-script helpers in engines/lemonade/src/process.rs (the from_mode(0o700) sites the diff adds)
    crates/rocm-core/src/fake_executable.rs exists to prevent this race. Writing a script in-process and then running it while other test threads fork fails intermittently with Text file busy (os error 26). engines/lemonade/src/install.rs in this same crate already uses that helper. One worker reported 5 such failures in 70 serial runs of cargo test -p rocm-engine-lemonade --lib, but its logs did not contain the error. I got 0 failures in 25 serial and 45 parallel runs, so I am relying on the in-repo explanation of the race, not my own reproduction. CI's per-process nextest would hide it. The local cargo test --workspace gate in AGENTS.md §8 would not. This also breaks AGENTS.md §6 (reuse the shared component).
    Confidence 85 · mechanical · Fix: call rocm_core::fake_executable::write_fake_executable from these helpers.

  • [new] Resetting telemetry when only canonical_model_id changes is never tested — crates/rocm-dash-daemon/src/runner.rs:650
    The worker replaced old.canonical_model_id == new.canonical_model_id with true, and all Lemonade telemetry contract tests stayed green. The model_ref and host comparisons next to it are each caught by a test, so this is the one part of the new rewrite detection that nothing tests.
    Confidence 100 · logic · Fix: add a case to assert_record_rewrite_resets_attributed_rate that rewrites only canonical_model_id.

  • [new] The reinstall advice is said to be "proved" by a step that never creates the condition the advice is for — tests/e2e-cucumber/tests/e2e/runtime_steps.rs:532-575, tests/e2e-cucumber/features/runtime_setup.feature:337-339, docs/testing.md (the runtime-lemonade-backend-alignment-reported paragraph), engines/lemonade/src/direct_llama.rs:775-785, PR description ("Local limits")
    The serve error at direct_llama.rs:74-79 says to run rocm engines install lemonade --reinstall when the selected backend's llama-server is missing. Runtime-16 runs --reinstall on a runtime it never damaged, then checks that the binary exists. So it proves that an install leaves a binary, not that --reinstall restores a missing one. AGENTS.md §3 requires the named command to "actually clear the condition that printed it", with the message and the resulting state asserted together. The step's doc comment, docs/testing.md and the PR body all call this step "the proof", so the description overstates what the step covers.
    Confidence 70 · logic · Fix: before the reinstall, delete <runtime_dir>/bin/llamacpp/<backend>/, ideally also check that rocm serve <owner/repo:variant> prints the advice, then reinstall and assert the binary is back. Otherwise, weaken the claim in all four places.

  • [standing since aa28578, not re-found blind; the follow-up check found it still holds] A managed Lemonade service with an API key shows a permanent "lemonade health … 401" warning and never shows throughput — crates/rocm-dash-daemon/src/runner.rs (managed branch, ~L817-822 and ~L886-888), crates/rocm-dash-collectors/src/lemonade.rs (fetch_health_if_router, ~L335)

    • engines/lemonade/src/process.rs:263-270 sets LEMONADE_API_KEY whenever rocm serve protects the endpoint; its comment says Lemonade then gates /api, /v0 and /v1 on it.
    • Nothing in the collector or daemon sends credentials. A 401/403 from the health check is kept as an error on purpose (and tested as such), so the managed branch pushes a "health" warning and skips the row on every cycle.
    • At aa28578 a managed record against a 401 fixture produced that warning on every snapshot. At dd0ac46 this was re-checked by reading, not re-run. That real lemond answers 401 there rests on the engine's own comment.
    • Confidence 85 · logic · Fix: present the CLI-owned key for registry targets, or treat 401/403 on a managed target as "throughput unavailable" with no standing warning. Add a regression test.
  • [standing since aa28578, not re-found blind; reproduced again by the follow-up check] A stale partial download in an older snapshot revision makes a complete split model permanently unservable — engines/lemonade/src/direct_llama.rs (hf_checkpoint_gguf_matches_in_roots ~L514-534, first_shard_of_complete_set, is_incomplete_shard_set, ensure_model_with_download)

    • Matches are still de-duplicated by file name across every snapshot revision in a root, in path order. So revision aaa/…-00001-of-00002 shadows the bbb/…-00001 of a complete set; the kept shards span two directories, the set reads as incomplete, and resolution returns Ok(None).
    • A probe test at dd0ac46 (aaa holds shard 1 only, bbb holds both) returned None, then ensure_model_with_download failed with "Lemonade did not download req for direct serving".
    • The author's earlier note ("a clean error, not a launch that fails after spawn") is accurate that nothing spawns, but nothing the user is told clears it: the automatic re-pull finds the same cache (inferred, not run against a real pull), and the message points at the download rather than the stale revision. That note also describes one root shadowing another; here both revisions are in the same root. The existing shards_spread_over_snapshot_revisions_are_not_a_complete_set does not cover this case.
    • Confidence 100 · logic · Fix: group matches per snapshot directory, and choose a complete set within one directory.
  • [standing since aa28578, not re-found blind; the mutation survived again] The "previously seen external endpoint went away" warning has no test — crates/rocm-dash-daemon/src/runner.rs (let absent = endpoint.rows.is_empty() && …is_connect_error(&error))

    • Dropping the endpoint.rows.is_empty() half silences that warning, and cargo test -p rocm-dash-daemon stayed green (47 unit, 9 end_to_end, 25 telemetry_contract).
    • lemonade_external_health_failure_warns_and_removes_rows_without_grace uses HTTP 503, which is not a connect error, so absent is false either way. A seen endpoint that stops, the commonest way a server goes away, is never exercised.
    • The code comment promises the code "reports why they disappeared". AGENTS.md §3 requires a message to be asserted together with its behaviour.
    • Confidence 100 (mutation run) · mechanical · Fix: seed rows, close the listener, and expect a warning naming the endpoint.
  • [standing since 788a1a3, not re-found blind; the follow-up check found the remaining part still holds] An idle Lemonade's last-request rate is presented as a live, fresh rate — crates/rocm-dash-core/src/observation.rs (observe_direct, ~L249), crates/rocm-dash-daemon/src/runner.rs (stats scrape), crates/rocm-dash-tui/src/ui/tabs/serving.rs:57, crates/rocm-dash-tui/src/ui/tabs/instances.rs:57

    • observe_direct re-stamps last_observed_at on every scrape, so the held legend can never fire for an idle Lemonade. Serving and Instances show the value as plain "tok/s". No "last request" qualifier appears in README.md, docs/testing.md or the TUI; only the collector's code comment and the PR description explain it.
    • The author's reply explains why the deeper fix (not re-stamping an unchanged report) was reverted, and that is credible. It is also true that external rows already behaved this way on main. But this PR newly scrapes managed Lemonade services through the same path, so a model started with rocm serve now shows its last request's rate as current throughput, which it did not before.
    • The cheap remedy needs no new scenario.
    • Confidence 80 (anchored check, read not run) · logic · Fix: label the figure where it is displayed (for example "last req tok/s"), or document what it means in the user-facing docs.

Non-blocking

  • [standing, re-found blind] A missing space joins two tags — tests/e2e-cucumber/features/runtime_setup.feature:340 (@id:runtime-lemonade-backend-alignment-reported@requires-gpu)
    gherkin 0.16 still reads it as two tags, because tag_char excludes @ and __ allows zero separators. But feature_naming.rs splits on whitespace, so it reads the id as runtime-lemonade-backend-alignment-reported@requires-gpu. The guard and the harness now see different ids for this scenario.
    Confidence 100 · mechanical · Fix: restore the space.

  • [new; the test-name part standing] Some new comments say things the code does not do — engines/lemonade/src/runtime_dir.rs:45-46, engines/lemonade/src/direct_llama.rs:145-147, crates/rocm-dash-collectors/src/lemonade.rs:640-653

    • runtime_dir.rs says the only callers of rooted_at are the Linux tests in process.rs. It is also called from lib.rs:898, router_serve.rs:543 and direct_llama.rs:1482.
    • The isolation comment says tests read no real user directories, but hf_cache_roots still reads HF_HOME, HUGGINGFACE_HUB_CACHE and HOME.
    • discover_is_clean_noop_when_endpoint_absent says it checks "an empty discovery result", but it asserts that fetch_health and fetch_stats return errors.

    Confidence 85 · mechanical · Fix: reword the comments, and rename the test or change what it asserts.

  • [new] Three host normalizers in the registry disagree on case — crates/rocm-dash-daemon/src/registry.rs:67, :105, :301-306
    host_reach lowercases and strips brackets; the other two are case-sensitive. A LOCALHOST record only groups correctly because the grouping step happens to lowercase.
    Confidence 70 · mechanical · Fix: route all three through one normalizer.

  • [new] The docs describe the serve bound imprecisely — docs/testing.md (dash-25 paragraph) and the comment on the new expectations.toml row
    Both say the serve is bounded by serve_timeout_secs (90s). Readiness polling does use that, but the CLI process is only killed after 90 + 300 s (serve_cli_budget).
    Confidence 70 · mechanical · Fix: mention the margin.

Standing objections settled this round without being kept

This check is anchored: the objection was put to the reviewer after the blind pass ended.

  • windows-build-and-test failed on lemonade_never_seen_absent_external_endpoint_is_quiet: no longer holds. send_error now maps connect errors and timeouts alike to "nothing listening", and closed_and_unanswered_endpoints_both_read_as_nothing_listening covers both cases. The test is not cfg-gated, and the windows-build-and-test log for dd0ac46 shows it ran and passed.

The seven other round-6 objections were settled at aa28578 as no longer holding (see the round-7 comment); none was re-found blind here.

Decisions for the author

  • The router→direct fallback after a failed load is removed — tradeoff
    An error now stops the server and fails the serve every time, which is more honest. The cost: a host where the router load fails but a direct server would have started now gets an error on its first serve. The PR states this, so this is only a request to confirm it is deliberate.
  • Vulkan counts as GPU-ready — tradeoff
    service_model_reports_gpu_backend (apps/rocm/src/main.rs) and direct serving now accept whichever GPU backend Lemonade selected. docs/manual-testing.md previously required "does not fall back to CPU or Vulkan". CPU is still refused, so the no-CPU-fallback rule holds. But a ROCm-capable host whose Lemonade picked Vulkan will now serve through Vulkan with no notice.
  • The real-GPU dashboard journey can never block a merge — tradeoff
    The new dash-lemonade-managed-inference-journey row in expectations.toml matches every lane where the scenario can run, so every assertion in it is advisory. This is disclosed. One option is to split the dashboard and instance-detail checks into a scenario that uses a planted managed Lemonade record, so they gate without the flaky serve.
  • The PR combines two independent changes — non-blocking-improvement
    It is about 7.6k lines. Lemonade catalog routing and serve-path rework is one change; dashboard external discovery and ownership is the other. Each commit mixes both. Splitting them would make each easier to review and revert.

Positive signals

  • Lifecycle ownership is a positive managed flag that only the registry sets, with serde defaulting it to false for old snapshots. The Services overlay re-checks that the approval target is still managed before acting on a "yes". Mutation testing showed each of those branches is caught.
  • serving_route decides the route before any download or spawn and refuses CPU and unknown backends. The shard-resolution edge cases are covered by mutation-verified tests.
  • The external-discovery scenarios use an in-process fixture that records any request other than a GET of health or stats. That is a real behavioural check that the dashboard never sends a lifecycle request to a server it does not own. Both scenarios pass locally (2 scenarios, 31 steps).

Deployment notes

  • dashboard.daemon.enable_lemonade defaults to true (new config fields enable_lemonade, lemonade_host, lemonade_port). Every Linux/WSL dashboard now probes 127.0.0.1:13305 read-only on each discovery tick.
  • Behaviour change: a failed router load no longer falls back to a direct llama-server.

What this covered

Every file in the change, 12342b3ac7be… (prw-base, equal to the merge-base) … dd0ac46a7141…. The code was split across three independent workers: Lemonade engine plus rocm-core, dashboard telemetry plus TUI, and the e2e harness plus docs. Some parts of large files were read only as diff hunks plus their surroundings, not end to end:

  • runner.rs lines 1000-1739 (snapshot assembly and existing tests);
  • the rest of registry.rs and of the pane.rs production code;
  • direct_llama.rs lines 1300-1412 and 1810-2001 (test bodies);
  • parts of process.rs and lib.rs;
  • the unchanged parts of serve_log.rs, serving_steps.rs and runtime_steps.rs.

Tests run, all in the checkout's own target/ with RUSTC_WRAPPER= empty:

  • lemonade lib: 177 tests, plus mutations;
  • dash-core, dash-collectors, dash-daemon (including telemetry_contract) and dash-tui: all green except 3 sandbox-only agent::clients failures unrelated to this PR;
  • e2e-cucumber lib (161) and feature_naming (4);
  • the two external Lemonade protocol scenarios.

CI is all green, but no self-hosted GPU or Windows-GPU lane appears in the rollup for this head. That is missing evidence for dash-25, serve-08 and serve-25, not a defect.

Did not run:

  • Agent-instruction adherence, history, and design passes: run by the driver in a single context, so not independent.
  • Prior-changes pass: not run (time).
  • Windows cfg arms: checked by reading only, not compiled locally. CI windows-build-and-test passed.
  • GPU scenarios: not run (no GPU).

Not reconciled against any discussion here. The standing objections above were weighed against the author's reply after the blind pass, and that is the only part of the discussion that reached the reviewer.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from dd0ac46 to a133657 Compare October 16, 2026 12:48
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Addressed in a133657, rebased on current main.

  • A complete split model in one snapshot now wins over an older partial download. The regression test failed before the fix.
  • Router stop and failed-exit paths now have tests, including a real stop of a fake lemond.
  • Test helpers use the shared fake-executable helper.
  • A change of only canonical_model_id now has a rewrite-detection test.
  • The "previously seen external endpoint went away" warning has a test.
  • A managed server that needs an API key (health returns 401 or 403) stays listed with its registry status and no throughput, and no longer shows a standing warning. No credentials are read or sent. External discovery still warns on these statuses.
  • The reinstall claim now says only what the step checks: --reinstall is accepted and a binary exists afterwards.
  • Fixed the missing space between two tags, the serve-timeout wording, and several comments and doc lines. The README and manual-testing notes now say the tok/s figure is the last request's rate.

On the decisions:

  • The idle rate is now labelled last req tok/s in the instance detail, with a legend in the Instances table.
  • The router-to-direct fallback is back on Linux. It runs only for a router load or verification failure, only for a model that resolves to a safe checkpoint, and only through the selected GPU backend's llama-server. A missing or CPU backend, Windows and any other error return the router's error. There is no scenario for it because there is no safe way to force a real router load failure; the unit tests assert the result together with the state or log.
  • A non-ROCm backend such as Vulkan is now named in the serve output and log.
  • The PR stays one change.

Not verified here: no GPU lane or cucumber scenario ran locally. CI covers those.

rominf
rominf previously requested changes Oct 21, 2026

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · a133657

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
No ticket named; the description was read and compared with the change (see the description finding below).

Round 9 (at a133657). This round reviewed a133657 in full and blind, because the previous review point, dd0ac46, is no longer in the branch after the rebase. Each finding is marked [new] or [standing]. Of the round-8 objections, one was re-found blind (the canonical_model_id clause: the added test does not catch its removal) and one came back as non-blocking (the reinstall advice). The other six were not re-found blind, so each was put to the reviewer afterwards, one at a time, with the author's reply (comment 6216331509). That check is anchored, because the reviewer had been told the objection. None of the six still holds as a blocker; they are listed at the end. This change request replaces the round-8 one as this account's complete position.

Two Blocking items do not on their own decide this change request. The merge conflict blocks the merge on GitHub anyway. The description items appear only in the PR description, not in any commit message, so they do not land on main. The fallback-window state, the vacuous leak test, the canonical_model_id clause and the missing route scenario do decide it.

CI: the one failing check, E2E tests (Strix Halo, Windows, current), fails scenario diagnose-38. Main's own run of that lane at 1f19198 fails it the same way, and this PR touches no diagnose or fix files, so the failure is not counted against this PR.

Blocking

  • [new] The PR no longer merges, because its new dash scenario numbers collide with ones added on main — tests/e2e-cucumber/features/dash.feature:560-613

    • The PR adds dash-35, dash-36 and dash-37. Main has since added its own dash-35 (dash-gpu-vram-uses-binary-units) and dash-36 (dash-onboarding-adopt-whitespace-cwd-quotes-approval-preview).
    • git merge-tree --write-tree prw-base HEAD reports CONFLICT (content): Merge conflict in tests/e2e-cucumber/features/dash.feature. This is the only conflicting file.
    • Confidence 100 · mechanical · Fix: rebase, then renumber the three new scenarios to dash-37–dash-39. Main's numbers stay because they are already merged.
  • [new] While the router-to-direct fallback runs, the engine state says failed, and any reader in that window locks the service record at failed for good — engines/lemonade/src/router_serve.rs:127-131, engines/lemonade/src/direct_llama.rs:110-149, crates/rocm-core/src/service_record.rs:224-246

    • serve_router stops lemond and calls mark_json_status(.., "failed"). serve_direct_after_router_failure then runs ensure_model_in, which can be a long download, and spawns llama-server. Nothing rewrites the state until that spawn succeeds.
    • refresh_from_engine_state copies failed into the record. Its callers then persist it: load_managed_service and load_managed_services in main.rs, and providers.rs:529. That covers rocm services, chat and dash.
    • Once the record says failed, refresh_from_engine_state returns early for any status outside starting|running|recovering|ready. A fallback that later succeeds therefore stays recorded as failed.
    • The fallback test (lib.rs:1195) asserts only the final state.
    • Confidence 85 (path traced, not run) · logic · Fix: write starting before the fallback starts its work. Add a test that asserts the state during the fallback.
  • [new] A test that claims to prove the direct server is never leaked stays green when nothing kills it — engines/lemonade/src/process.rs:1009-1016, process.rs:1107-1164 (direct_server_stops_when_state_cannot_be_written_after_spawn)

    • The fake server is #!/bin/sh\nexec sleep 30. After the exec, the process's /proc/<pid>/cmdline is sleep 30, so the test's search for the script path can never match.
    • I removed terminate_pid and child.wait() from stop_and_fail in supervise_direct_server. This test still passed, and a sleep 30 process was left running.
    • The sibling direct_server_step_failure_stops_the_child_and_marks_state_failed did fail under the same mutation, so only this test is vacuous.
    • Confidence 100 · logic · Fix: have the fake write $$ to a file before exec, and check that PID.
  • [standing since dd0ac46, re-found blind] One clause of the runner's same-endpoint check can be deleted without any test failing — crates/rocm-dash-daemon/src/runner.rs:655-656

    • With && old.canonical_model_id == new.canonical_model_id removed, all 19 telemetry_contract Lemonade tests passed.
    • lemonade_managed_canonical_model_rewrite_resets_attributed_rate passes only because the rewritten name stops matching health, and the unmatched-row tracker removal clears the rate anyway.
    • Confidence 100 · mechanical · Fix: add a case where canonical_model_id changes while a name that still matches health remains.
  • [new] No scenario covers the new route choice for registered names: simple ones are now served directly on Linux — tests/e2e-cucumber/features/model_serving.feature (serve-07/serve-08), tests/e2e-cucumber/tests/e2e/serving_steps.rs:746

    • Which runtime a serve uses is user-observable. AGENTS.md §3 says such a change needs a Gherkin scenario, and that "a unit test asserting the internal helper does NOT discharge this".
    • The only route assertion in tests/ is "lemonade-cli", in serve-26 (router route). "lemonade-packaged-llama-server" appears nowhere under tests/.
    • serve-07 stays green whichever route a simple registered name takes.
    • The description admits the gap and points to unit tests, which is the substitution §3 rules out.
    • Confidence 85 (cited rule, verified with grep) · logic · Fix: add a @requires-os:linux Then step that reads the engine state for a simple registered name and asserts the direct method and a non-CPU backend.
  • [new; the reinstall part standing] The description overstates two things

    • The non-ROCm notice. The description says "When the selected backend is not ROCm … the serve output and log name that backend". The direct path writes the notice to the log (process.rs:524-548). The router path only calls eprintln! (router_serve.rs:87-89). A managed serve runs under spawn_serve_http_background, whose stdout and stderr go to Stdio::null() (state.rs:141-143), so a managed Vulkan router serve shows the notice nowhere. a_non_rocm_backend_is_named_in_the_notice_and_matches_the_state tests the helper, not the emit.
    • The reinstall advice. The description says the advice "is proved by a new last step of runtime-lemonade-backend-alignment-reported". Later it says the step "does not prove that the command repairs a deleted one". The second statement is the accurate one.
    • It also names scenarios by stale numbers ("dash-25", "the nightly serve-25"); the PR's scenarios are dash-35 and serve-26.
    • Confidence 85 · logic · Fix: for the notice, either share one emit-and-log helper between the two routes and assert it from the log, or narrow the description. For the rest, correct the wording.

Non-blocking

  • [standing, re-found blind] The new reinstall advice is not shown to fix the condition it reports — engines/lemonade/src/direct_llama.rs (the error added by this PR), tests/e2e-cucumber/tests/e2e/runtime_steps.rs (assert_lemonade_backend_server_installed)

    • The error added by this PR tells the user to run rocm engines install lemonade --reinstall. AGENTS.md §3 requires that a named command actually clears the condition.
    • The step never deletes the binary first, so it passes even if the reinstall does nothing. It also runs only on Windows nightly, while the advice is printed on Linux too.
    • The PR states this gap openly.
    • Confidence 70 · logic · Fix: delete the selected backend's llama-server, reinstall, then assert it is back.
  • [new] Comments this change made stale

    • router_serve.rs:91-92 says a load failure is preserved "rather than masking it with direct serving". The next call tags it RouterLoadFailure, which is what turns on direct fallback.
    • crates/rocm-dash-tui/src/ui/mod.rs:258-260 still refers to the "Running now" listing, which this PR renamed to "Managed running".
    • ui/mod.rs:353-357 says service_rows can include an unmanaged Docker/Lemonade entry. It no longer can.
    • In each case the code governs, because the comment describes behaviour the code no longer has.
    • Confidence 85 · mechanical.
  • [new] The last req tok/s label appears only in the Instances tab — crates/rocm-dash-tui/src/ui/tabs/pane.rs (OpenServices gen span), ui/services_manager.rs (gen_tps_compact)

    • Managed Lemonade rows on these surfaces show a stale last-request rate as plain gen throughput. Home and Observe also add it into their totals.
    • The README scopes its claim to the Instances table and detail, so this is an inconsistency, not a false claim.
    • Confidence 70 · logic.
  • [new] External discovery can flap and stall the daemon loop

    • One failed or slow /health call removes every external row with no grace period. That behaviour is pinned by lemonade_external_health_failure_warns_and_removes_rows_without_grace.
    • parallel_scrape is awaited inline in the tick loop (runner.rs:813-886). Up to three sequential 1.5 s requests can delay GPU ticks.
    • Both matter more now that discovery is on by default against 127.0.0.1:13305.
    • Confidence 70 · architectural · Fix: put an overall timeout on each endpoint and allow one missed discovery.
  • [standing] The PR does several separable things in about 8,400 lines. These are:

    • Lemonade catalog routing;
    • the router-to-direct fallback;
    • dash external discovery;
    • Services-overlay ownership;
    • rate labelling.

    AGENTS.md §11 asks for one logical change per PR. Splitting engine routing and fallback from the dash work would make each part reviewable.

Standing objections settled this round without being kept

Each was put to the reviewer after the blind pass ended, one at a time, with the author's reply. These checks are anchored, because the reviewer had been told the objection.

  • A managed Lemonade service with an API key shows a permanent 401 warning: no longer holds. A 401 or 403 on a managed health check now maps to "no telemetry" with no warning. External discovery still errors on these statuses. lemonade_protected_managed_endpoint_keeps_registry_status_without_telemetry_warning passes, and fails with that branch disabled.
  • A stale partial download in an older revision makes a split model unservable: no longer holds. Complete sets within one snapshot directory now sort first. complete_snapshot_is_not_shadowed_by_an_older_partial_download passes, and fails with that ordering disabled.
  • The "previously seen external endpoint went away" warning has no test: no longer holds. With the round-8 mutation re-applied, lemonade_seen_external_endpoint_closing_warns_and_removes_rows fails in 3 of 3 runs.
  • The router's real stop and failed-exit paths are untested: no longer holds. Re-applying both mutations fails nonzero_server_exit_returns_error_and_marks_state_failed and real_router_stop_kills_and_reaps_its_server. The second checks the PID it spawned, so it does not share the exec sleep weakness above.
  • New fake-executable helpers skip the shared ETXTBSY-safe helper: no longer holds, by reading only. Every fake executable this PR adds goes through write_fake_executable.
  • An idle Lemonade's last-request rate is shown as a live rate: no longer holds as a blocker. README, docs/testing.md, docs/manual-testing.md and the Instances table and detail now label it. The Serving and Services surfaces still show plain gen, which is the non-blocking finding above.

Decisions for the author

  • External Lemonade discovery is on by default — tradeoff
    • For: zero-config visibility of a local Lemonade server.
    • Against: every Linux/WSL dashboard now probes a third-party port. Any other service there that answers 404, 401 or 503 produces a standing warning, and the probe adds tick latency.
    • The other option is off by default, with the dashboard telling the user how to turn it on.
  • Registered aliases on Linux now go direct, launched with basic flags only — tradeoff
    • Simple catalog entries no longer receive the router's per-model launch defaults.
    • Restricting the route to "simple" entries limits this, but confirm that losing those defaults is acceptable for the default model.

Positive signals

  • serve_router wraps every post-spawn step in one fallible block behind the RouterSteps trait, so any failure stops and reaps lemond. The old path could leak it.
  • Fallback is limited by a typed RouterLoadFailure marker, checked by downcast. It is Linux-only and refuses cpu and unknown backends, which keeps the no-CPU-fallback policy.
  • Ownership is a serde-defaulted-false Instance.managed, set only by registry discovery and checked in one predicate. The Services manager revalidates a pending approval against fresh rows. A mutation run showed the tests catch removing that check.
  • The external-protocol scenarios assert a negative: the fixture logs every non-GET request, and the scenarios assert that no lifecycle call reached the external server.

Deployment notes

  • A new TUI attached to an already-running older daemon, or replaying an old recording, sees managed = false on every row. The Services overlay then says "No managed services" and silently disables stop/restart until the daemon restarts.
  • dashboard.daemon.enable_lemonade defaults to true for existing configs.
  • rocm dash now runs a builtin Detect call at startup.

What this covered

  • Read: every file in prw-base...HEAD (merge-base 1f19198…a1336572, 46 files), in two coupled groups, each run as an independent read-only worker:

    • the Lemonade engine, main.rs, config.rs and the serving/runtime e2e;
    • the dash daemon, collectors, TUI and dash e2e.

    The engine worker split its group across four sub-workers. Their reported gaps:

    • the unchanged parts of runtime_dir.rs;
    • e2e.rs, serving_steps.rs and runtime_steps.rs, read at the changed hunks plus the functions they depend on;
    • only the changed hunks of apps/rocm/src/main.rs.

    The driver read the docs, README, expectations.toml and the feature files.

  • Tests run:

    • cargo test -p rocm-engine-lemonade -p rocm-core: 189 and 748 passed.
    • The dash worker ran the rocm-dash-tui services_manager (23), telemetry_contract lemonade (19) and rocm dash::tests (37) tests.
    • Branch mutations were run on supervise_direct_server, the runner's endpoint check, and the services-manager approval and selection guards.
    • No e2e or GPU lanes were run.
  • CI: the only failing check, Strix Halo Windows diagnose-38, fails the same way on main at 1f19198 and is outside this diff, so it is not counted. Skill checks (skillscope) was skipped.

  • Did not run:

    • The design-review step was done in the driver context, not as independent workers, because of the 2-worker budget.
    • The prior-changes pass did not run: the PR's earlier reviews and comments are out of bounds here.
    • The history pass was limited to checking for drift against main, which found the conflict.
    • ai-instruction-quality was not activated, because no agent-instruction files changed.
  • Not reconciled against the PR discussion, except as stated in the round note: after the blind pass, the standing objections were weighed against the author's reply. No prompt-injection content was found.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 8826df9

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (11,428 insertions / 2,263 deletions across 65 files). No ticket named; this PR does not claim to close any issue.

Blocking

  • PR description cites scenario numbers that no longer match the current head — tests/e2e-cucumber/features/model_serving.feature:869-887, tests/e2e-cucumber/features/dash.feature:878-915
    The description states: "they are now serve-42 and serve-43 ... and dash-42, dash-43 and dash-44 ... Their @id: tags did not change." At this head, the stable @id: tags are indeed unchanged and correctly described (@id:serve-lemonade-complex-registered-inference, @id:serve-lemonade-simple-registered-direct-route, @id:dash-lemonade-managed-inference-journey, @id:dash-lemonade-external-protocol-discovery, @id:dash-lemonade-external-protocol-transitions), but the scenario numbers themselves have drifted again since the description was last written: they are actually serve-55, serve-56, dash-50, dash-51, dash-52. The numbers the description cites (serve-42/serve-43/dash-42/dash-43/dash-44) now belong to unrelated, pre-existing scenarios (e.g. the actual serve-42 is "Serving on a WSL host whose ROCDXG library the loader cannot find is refused with WSL advice"). A maintainer or future contributor following the description's numbers to find these scenarios will land on the wrong ones.
    Confidence 85 · documentation · Fix: re-read the description against the current main once more before merge (per this repo's own §5 rule to re-read the PR body after every push) and either update the cited numbers or reference scenarios only by their stable @id: tag, since the numbers are expected to keep drifting on every rebase.

Non-blocking

None.

Decisions for the author

  • A mixed Instances table (one live-rate row + one last-request-rate row) cannot be told apart per row — answered-by-intent
    Verified directly: crates/rocm-dash-tui/src/ui/format.rs:254 (gen_tps_cell) has no parameter for gen_tps_last_request and never marks a cell for it (only the orthogonal HELD staleness marker); crates/rocm-dash-tui/src/ui/tabs/instances.rs:119 renders the TOK/S cell without it, and the only indicator is the single table-wide legend built at instances.rs:178-186 from any_held/any_last_req booleans — it does not say which row(s) it refers to. I confirmed no test exercises two simultaneous instances with differing gen_tps_last_request values (state_with_snap in instances.rs:568 only ever builds a single-instance snapshot). This would otherwise be a blocking "behaviour the change introduces is tested" gap, but the PR body already states it verbatim under "Not verified": "The Instances table marks a last-request rate only through the legend under the table, not per row, so a table that mixes live and last-request rows does not say which rows the legend refers to. No test covers a mixed table." Confirm this is still the intended trade-off before merge (and consider at least pinning the current, accepted ambiguity with a regression test so a future change doesn't need to rediscover it).
  • Download-only lemond instance and the router's own lemond may run concurrently on different ports during the direct-serve fallback — tradeoff
    direct_llama.rs's ensure_model_in spawns a throwaway lemond purely to run lemonade pull, independent of the main router lemond spawned in lib.rs/router_serve.rs. Worth confirming there's no cache/lock contention in Lemonade's own runtime directory between the two under concurrent serves for the same checkpoint — raised by one of the review passes, not independently verified against Lemonade's internals (out of this repo).
  • --n-gpu-layers 999 is hardcoded in the direct-serve llama-server command — tradeoff
    Under the GPU-required policy this fails hard (OOM) rather than gracefully degrading to a partial CPU/GPU offload on a tight-VRAM host. May be the intended behavior given "no CPU fallback" is itself a hard invariant here; confirm that's deliberate.
  • The router-failure fallback can trigger an unattended, potentially multi-gigabyte download with no user confirmation or size cap — tradeoff
    A flaky or slow router always risks setting this off. Worth confirming this is accepted given the "keep download and load errors" framing in the description.
  • render_service_action_result's three-way .or_else chain over independently-shaped JSON (apps/rocm/src/services_text.rs) — non-blocking-improvement
    Parses three possible result shapes (service / result.service / result.service_id) to find the stop report's service:/status: lines. A fourth shape would silently fall through to printing no service info rather than erroring. A shared tagged type between service_control.rs and services_text.rs would make a new shape a compile error instead of a silent gap.

Positive signals

  • Instance::live_gen_tps()/live_tokens_per_watt() (crates/rocm-dash-core/src/metrics.rs:282-292) is a single, clearly documented accessor that excludes a last-request rate from node-wide aggregates, and both home.rs and observe.rs heroes consistently call through it rather than reading .gen_tps raw.
  • is_listed_service() (crates/rocm-dash-tui/src/ui/services_manager.rs) is genuinely the single shared ownership predicate used by both the quit-confirm shortcut and the Services overlay (not two independently-maintained copies that happen to agree today), with a dedicated test for the "started but the daemon's broadcast hasn't caught up yet" race window.
  • router_serve.rs's state-machine testing is unusually disciplined: a Fail enum drives one fake-steps test double through every failure point, and a dedicated test (only_a_load_or_verification_failure_is_eligible_for_direct_fallback) pins exactly which failures may and may not trigger the direct-serve fallback — a real state-matrix-with-exits treatment.
  • device_policy.rs's GPU index resolution has explicit tests for the no-silent-CPU-fallback invariant this project requires (no usable device → hard error; explicit absent device → hard error; never silently remapped).
  • The split-GGUF-shard resolution and "complete snapshot beats an older partial download" logic in direct_llama.rs has a real regression test that would fail (via an unwanted pull) if the algorithm regressed, not just a trivial-input check.
  • Docs-sync discipline is strong: README.md, docs/testing.md, docs/manual-testing.md and docs/architecture.md were all updated together for the new dashboard.daemon.enable_lemonade config key and the ROCm-or-Vulkan readiness change, with entries present and accurate for every newly-split file (router_serve.rs, device_policy.rs, runner/discovery.rs).

Deployment notes

  • dashboard.daemon.enable_lemonade defaults to true: after this merges, rocm dash will probe 127.0.0.1:13305 read-only by default on Linux/WSL hosts that previously had no such behavior. Operators who don't want this need the documented opt-out.
  • As of this review, the PR shows mergeable_state: CONFLICTING against main — it will need a rebase before merge regardless of review outcome.
  • CI is fully green on this exact head commit (all listed checks, including clippy — which covers this repo's module-size and architecture-doc gates — and the full E2E suite, report SUCCESS).

What this covered

Diffed eb2a8194a2aa4323904b01ab15a573c5b9cb0fa3...8826df9f834f5b182fcea80b0b7335186672ccd1 (the merge-base of refs/remotes/origin/main and the fetched PR head refs/remotes/origin/pr-631-head), fetched read-only and never checked out. All 65 changed files were split into 8 coupling-based groups for fan-out.

Three groups received a genuine independent fresh-context worker pass that read every assigned file completely: lemonade direct-serve (direct_llama.rs, process.rs, state.rs, runtime_dir.rs, test_http.rs, llamacpp_backend.rs), CLI apps/config/docs (cli.rs, dash.rs, engine_runtime_select.rs, services_cmd.rs, services_text.rs, rocm-core/config.rs, aim/docker.rs, README.md, docs/architecture.md, docs/manual-testing.md, docs/testing.md — though for the largest of these files that worker read the full diff plus targeted greps rather than every unchanged line), and a third (TUI tabs/formatting: format.rs, instance_detail.rs, launcher.rs, ui/mod.rs, tabs/home.rs, tabs/instances.rs, tabs/observe.rs, tabs/pane.rs, tabs/serving.rs, dash_characterization.rs) was dispatched independently but had not returned by the time this review was finalized.

For the remaining five groups — lemonade core orchestration (lib.rs, router_serve.rs, backend_alignment.rs, device_policy.rs), dash collectors/core (rocm-dash-collectors/lemonade.rs, docker.rs, rocm-dash-core/efficiency.rs, metrics.rs, observation.rs, traits.rs), dash daemon registry/discovery (registry.rs, runner.rs, runner/discovery.rs, demo.rs, discovery_reconcile.rs, telemetry_contract.rs), TUI app lifecycle (app/quit_confirm.rs, ui/quit_confirm.rs, services_manager.rs, agent/snapshot.rs, app/event_loop.rs, app/mod.rs, app/test_support.rs, app/types.rs), and e2e-cucumber tests — a subagent concurrency pool shared with unrelated sessions on this machine stayed saturated throughout this review (repeated "concurrent subagent limit reached" errors; ListAgents showed 30-42 concurrent subagents from other sessions at every check). I ran these passes myself instead, which is not an independent pass per this method's own rule, and did not read every line of every file in these five groups: in particular crates/rocm-dash-daemon/src/demo.rs and the bulk of telemetry_contract.rs (2,422 lines) and discovery_reconcile.rs were only sampled by test/function name, not read in full; agent/snapshot.rs, app/event_loop.rs, app/mod.rs, app/test_support.rs, app/types.rs, and ui/quit_confirm.rs were diff-reviewed but not read as complete files; and six of the fourteen e2e-cucumber files (dash_screen.rs, lemonade_backend.rs, serve_log.rs, src/lib.rs, service_stop_steps.rs, serving_steps.rs, expectations.toml, tests/e2e.rs) were not reviewed at all. I did read registry.rs, router_serve.rs, device_policy.rs, lemonade.rs (collectors), efficiency.rs, metrics.rs, lemonade_protocol.rs, and the core of runner.rs's tick loop in full, and traced the specific risk areas named in this method's rubric (no-silent-CPU-fallback, the broadcast-race window, the serial-tick/blast-radius question, and the fixture-based discovery assertion's non-vacuity) to a concrete answer in each case.

Did not run: "prior changes" (no accessible review-comment history on earlier changes to these files separate from this PR's own 27-round discussion thread, which this method's rules keep out of scope for a blind pass) — skipped. "History" was applied only partially (diffed runner.rs against its pre-PR shape to confirm the registry-read code path predates this PR, rather than as a full separate pass). No cargo command was run anywhere in this review (build/test execution against this workspace was judged out of proportion given CI is green on this exact head; all scoring above is from static reading, capped accordingly).

The PR description was necessarily read once, incidentally, at the very start of this review (the dedupe-check tool call required by this task returns the full PR body alongside the head SHA) — earlier than this method's intended order. No finding above was derived from or softened by that framing; its only use is the scenario-numbering finding and the retired mixed-table finding, which is exactly where this method says description-vs-code comparison belongs. Not reconciled against this PR's 27 rounds of prior human/automated review discussion — intentionally, per this method.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio
juhovainio self-requested a review November 12, 2026 13:42

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this change and have three things that need fixing before it can merge, plus a few smaller notes. Nothing here questions the approach; the problems are in how the PR is packaged.

Needs fixing

  1. Scenario ids collide with main. dash-50 and dash-51 already exist on main ("The CPU graph stays a line when the host is idle" and "A modest core count puts the CPU graph beside the per-core strip"). The PR is also CONFLICTING right now. After the rebase these need renumbering (serve-55/serve-56 are still free on main, so only the dash ones move), and the @id tags re-checked.
  2. The PR body no longer matches the diff. It names serve-42/serve-43 and dash-42/dash-43/dash-44, and describes a single commit f42771f9; the diff has serve-55/serve-56, dash-50/dash-51/dash-52, and the head is 8826df9f. It also leaves out several changes that are in the diff: the removal of GpuSystemInfo.lemond_version, the removal of the public collector APIs (lemonade_instance, parse_health_model, fetch_health_model, LemonadeCollector::discover), the Engines pane and Instances empty-state wording, dropping the Starting to Ready promotion on a stats scrape, and exact model-name matching on the direct route. Please rewrite it from the final diff.
  3. docs/architecture.md still says Lemonade discovery is off by default and that the daemon rocm dash starts leaves it off. This PR flips dashboard.daemon.enable_lemonade to true, so that sentence is now false. Please update it in this change.

Smaller notes (inline where they anchor to the diff)

  • A TUI attached to a daemon from an older build gets managed=false on every row, so the Services overlay is empty and the quit prompt never offers s.
  • rocm dash startup now runs an engine Detect request synchronously. I couldn't see a deadline on it in the diff.
  • serve_http_resolving marks any existing state file failed on an error before this run has written state, which would clobber a stale file from an earlier run. Low impact.
  • Soft smell: the same group of arguments (paths, request, runtime, process_env, log_path, parent, is_linux) is threaded through several functions with too_many_arguments allowed, and there are paired X / X_with_parent wrappers in process.rs and llamacpp_backend.rs.

I read this from the diff only and did not build or run tests. CI is green apart from the W7900 e2e lane, which was cancelled by the runner outage, and skillscope, which was skipped. The PR body says smoke_local.py and cargo xtask coverage were not re-run on this head, so those still need a run after the rebase.

Then the dashboard exits successfully

@id:dash-lemonade-managed-inference-journey @requires-os:linux @requires-gpu @requires-engine:lemonade
Scenario: dash-50 - A real managed Lemonade GPU model responds to inference and appears in the dashboard

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dash-50 already exists on main ("The CPU graph stays a line when the host is idle"); renumber on rebase.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renumbered in 223933a. This change's dash scenarios are now dash-52, dash-53 and dash-54 (main has dash-50 and dash-51), and its serving scenarios serve-56 and serve-57 (main took serve-55). The @id: tags did not change. The description now names scenarios by @id: only, since the numbers move on every rebase.

And the model responds to inference requests

@id:dash-lemonade-external-protocol-discovery @requires-os:linux
Scenario: dash-51 - External Lemonade protocol discovery displays every loaded model without adopting the service

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dash-51 already exists on main ("A modest core count puts the CPU graph beside the per-core strip"); renumber.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renumbered with the one above (223933a).

pub memory_partition_mode: MemoryPartitionMode,
pub compute_partition_mode: ComputePartitionMode,
pub vram_per_logical_gpu_mb: u64,
pub lemond_version: Option<String>,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing GpuSystemInfo.lemond_version is a struct/wire change that the PR body doesn't mention. Please call it out (old recordings still load per the new test).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is called out in the PR body now, under the user-visible changes as removed API, together with the removed collector functions. Old recordings that still carry the key load; old_recordings_with_a_retired_lemond_version_key_still_load covers it.

/// The one ownership predicate for lifecycle surfaces: only registry-owned
/// instances are listed and counted, because those are what the controls act on.
#[must_use]
pub const fn is_listed_service(i: &Instance) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ownership is managed only, so a TUI attached to a daemon from an older build gets managed=false on every row and an empty Services overlay. Worth a note, or treating a missing field differently from an explicit false.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added as a deployment note in the PR body. Not changed in code: managed is a plain bool with a serde default, so a missing field and an explicit false look the same after deserialization, and telling them apart would mean an Option<bool> through the snapshot, the reducer and every consumer. rocm dash embeds its daemon from the same binary, so the case only arises for a separately started older daemon. I can do the Option if you would rather have it.

Comment thread apps/rocm/src/dash.rs

/// Read-only managed-engine inventory, independent of GPU telemetry and
/// endpoint liveness. An unavailable detect response means unknown, not absent.
fn lemonade_managed_installed(paths: &AppPaths) -> Option<bool> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Detect request this helper triggers runs synchronously during dash startup. Please confirm the stdio client has a deadline, or move it off the startup path.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked. This Detect request does not spawn a child: engine_request hands a lemonade Detect to the built-in in-process handler (builtin_engine_request), which reads the install manifest and runs the local GPU visibility probe that rocm serve and rocm examine already run. There is no stdio client and no network, so there is no stdio deadline to apply, and I did not add a separate one around it. The PR body says so under the deployment notes.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from 8826df9 to e5439f4 Compare November 12, 2026 14:24

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head e5439f48

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (base b109db12…head e5439f48, 65 files, ~11.5k insertions / ~2.4k deletions). Implements the summarized Lemonade GGUF-resolution/fallback and dash-discovery work described in the PR body in full; no ticket is named for the PR as a whole (the diff's own ROCMAI-898/EAI-7423 identifiers are pre-existing bug-tracking IDs on e2e flaky-test rows, cited below per this repo's own established convention, not an external tracker for this PR).

Blocking

  • New expectations.toml row cites the wrong tracking ticket for its failure mode — tests/e2e-cucumber/expectations.toml:182-187
    The new dash-lemonade-managed-inference-journey row's reason ("can reach ready and then shut down, or never become ready") describes the broad Strix-Halo-Ubuntu Lemonade serve flakiness, but cites bug = "EAI-7423". The file's own header comment a few rows above (lines 61-78) explicitly says ROCMAI-898 is the open ticket covering exactly this broad condition and that "EAI-7423 described only the earlier ready-then-unavailable symptom"; a sibling row's comment (~line 163) even states in so many words that it names ROCMAI-898 "rather than EAI-7423" for the identical condition. Every other row for this condition cites ROCMAI-898; this is the only one citing the narrower ticket. No functional effect on the test itself (flaky = true tolerates either outcome), but it misdirects triage and breaks the file's own established convention.
    Confidence 85 · mechanical · Fix: change bug to "ROCMAI-898".

  • docs/architecture.md's registry.rs entry doesn't mention this PR's main new capability in that file — docs/architecture.md:459
    This PR adds substantial new machinery to registry.rs — ManagedLemonadeTarget, host_reach, endpoints_match, scrape_host, cached_collector_reusable (confirmed present at crates/rocm-dash-daemon/src/registry.rs:385-592) — which is exactly the dual-stack endpoint-matching logic the new managed-Lemonade discovery feature depends on. The entry's description still only covers the pre-existing scrape-target/ancestry-PID behavior. The PR did add a correct new row for the sibling runner/discovery.rs (showing the author followed the doc-update convention there), but left this pre-existing row stale despite the size of the addition. cargo xtask check-architecture-doc only checks that an entry exists, not that it stays accurate, so this passed CI.
    Confidence 85 · mechanical · Fix: extend the registry.rs row to mention the new endpoint-matching machinery.

  • docs/testing.md cites the wrong file for a new regression test — docs/testing.md:805-806
    node_efficiency_leaves_out_a_last_request_rate is cited as living in widgets.rs; it actually lives in crates/rocm-dash-core/src/efficiency.rs:328 (confirmed: no such function exists in widgets.rs, and efficiency.rs has it). The other three test citations in the same sentence (instances.rs, instance_detail.rs, home.rs/observe.rs/snapshot.rs) are all correct — this is an isolated mis-citation, the same class of defect the file.rs:42-citation rule in this repo's contributor guide guards against (a citation pointing at the wrong place rather than a broken link).
    Confidence 85 · mechanical · Fix: change the citation to efficiency.rs.

Non-blocking

  • New dashboard config default duplicates an existing port constant with no test tying them together — crates/rocm-core/src/config.rs:360-362 (default_dashboard_lemonade_port() -> 13305) vs. the pre-existing crates/rocm-dash-collectors/src/lemonade.rs:30 (LEMONADE_PORT: u16 = 13305)
    rocm-dash-collectors depends on rocm-core (not the reverse), so a test in rocm-dash-collectors asserting LEMONADE_PORT == DashboardDaemonConfig::default().lemonade_port is straightforward and doesn't exist; the new config test only checks the literal 13305 against itself. The existing engine_registry.rs:40 already references crate::lemonade::LEMONADE_PORT rather than a fresh literal, so the tying pattern is established elsewhere in this same crate, just not followed for the new config default. (A third, pre-existing, independent 13305 literal also lives in crates/rocm-dash-tui/src/llm.rs:221, outside this PR's diff.)
    Confidence 75 · mechanical · Fix: add a cross-crate test in rocm-dash-collectors tying the two together.

  • The instance-detail "recent throughput" panel doesn't account for a last-request rate, unlike every sibling surface this PR touches — crates/rocm-dash-tui/src/ui/instance_detail.rs:158-219 (recent_activity_rows, has_observed_throughput)
    These use inst.gen_tps directly and ActivityRow doesn't carry gen_tps_last_request at all, unlike render_summary in this same file (which correctly swaps the label via format::gen_rate_label), and unlike every other consumer of this flag added in this PR (home/observe heroes, efficiency.rs, the Instances legend, Services overlay/preview). The panel's own doc comment calls it "the last few snapshot ticks where the instance recorded non-zero throughput... actually serving" — a last-request sample is not that, it is one already-completed request's rate re-reported each tick. The existing Held-tick collapsing logic does prevent this from rendering as multiple duplicate rows, so the practical harm is narrower than it first looks: a single row with a bare tok/s number, unlabeled as "last req" the way every sibling surface labels it. No test in this file's mod tests ties recent_activity_rows/has_observed_throughput to gen_tps_last_request (confirmed by inspecting the full test module), despite the PR's otherwise thorough test coverage of this exact flag everywhere else.
    Confidence 70 · logic · Fix: either label/exclude the last-request case here the same way render_summary does, with a test, or confirm the panel is meant to show it unlabeled and say so in the doc comment.

  • The Lemonade scrape now runs before the vLLM Prometheus scrape in the same daemon tick, newly exposing the fast vLLM path to Lemonade's per-endpoint latency — crates/rocm-dash-daemon/src/runner.rs (Lemonade scrape ~line 799 inside parallel_scrape, vLLM scrape ~line 927, with the comment above the vLLM block reading "Lemonade rows are scraped via their endpoint collectors above")
    Both scrape groups are internally parallelized via parallel_scrape, but the two groups themselves run sequentially within one tick, not concurrently with each other. This repo's own contributor guide already accepts "the dash daemon's tick polls its sources serially" as known debt, but this PR's new Lemonade phase is a real new network round trip (health+stats per discovered endpoint) inserted ahead of the previously-fast vLLM scrape, widening that known window rather than just inheriting it unchanged.
    Confidence 70 · architectural · Fix: run the two scrape phases concurrently (e.g. tokio::join!), or put vLLM's scrape first.

Decisions for the author

  • serve-lemonade-simple-registered-direct-route (serve-43) has no expectations.toml row, unlike the sibling scenarios sharing its lane — answered-by-intent
    PR body: "It needs @requires-gpu, @requires-os:linux and @requires-engine:lemonade, and has no flaky expectations row, so a route regression fails the lane." Would otherwise have been a blocking finding (every structurally similar Strix-Halo-Ubuntu Lemonade scenario in this lane carries a ROCMAI-898 row, and this scenario launches the same llama-server backend that ticket blames for backend-level failures). Confirmed deliberate — the author chose to let a regression in the new route-decision logic fail loudly rather than tolerate it as flaky. Worth confirming this tradeoff (clean signal for the new logic vs. possible CI noise from the shared, already-flaky backend) was made with that lane's track record in mind.

Positive signals

  • The router-to-direct-llama-server fallback resets PID/identity state before falling back, with a regression test that reads state during the pull (not just before/after) — a specific, non-obvious correctness property, actually exercised.
  • Instance::live_gen_tps()/live_tokens_per_watt() correctly exclude a last-request rate from every node-wide aggregate, with an exact-value regression test (node_efficiency_leaves_out_a_last_request_rate asserts the quotient is 0.3, not 5.3 — not a loose substring check).
  • The Services overlay's "managed services only" filter is proven by tests using deliberately misleading ids (svc-looks-managed, lemonade-owned) that still key correctly off the managed boolean rather than a naming pattern — exactly the "correlate by id/flag, not by convention" discipline this repo's guide asks for.
  • tests/e2e-cucumber/tests/e2e/service_stop_steps.rs's reports_stopped step genuinely drives the real rocmd binary end-to-end and asserts both the rendered service:/status: output lines and the on-disk record together, satisfying this repo's "message asserted with resulting state" rule at the integration level for the new stop-output fields.

Deployment notes

  • dashboard.daemon.enable_lemonade defaults to true: on Linux/WSL, rocm dash will now probe 127.0.0.1:13305 by default (read-only) unless explicitly disabled. This is called out in the PR body already, but is worth re-flagging for anyone rolling this out to users who run their own local Lemonade server on that port.
  • Recordings made before this change load with no managed services in the Services overlay, because ownership now defaults to unmanaged (also disclosed in the PR body).

What this covered

The whole diff, base b109db12c7a59f5f5c6f38d1e4a2191ff8851b53…head e5439f48a1d6f1054048cbb804fe9b0f17ab89f4: every changed file read in full (not just the diff hunks) across the Lemonade engine (GGUF/registered-model resolution, router/direct-serve fallback, backend/process/device code), the dash daemon (discovery/registry/runner and its tests), dash-collectors and dash-core, the dash TUI (tabs, instance detail, quit-confirm, services manager), the apps/rocm CLI surfaces touched (cli.rs, dash.rs, engine_runtime_select.rs, services_cmd.rs, services_text.rs), the e2e-cucumber fixtures/step-definitions/feature files, and the documentation diff — via independent fan-out workers grouped by coupling, each told to verify its own candidates against the source before reporting. The highest-value candidates from that fan-out were then independently re-traced against the actual PR-head source by this consolidating pass (not taken on trust), which is also what retired two raw candidates as false positives after direct verification (an instances.rs legend-gap claim that doesn't hold once tokens_per_watt's derivation is traced, and a unit-test gap claim in services_cmd.rs that turned out to be covered by a genuine end-to-end test in a file outside that worker's slice) and is why the serve-43 point above moved from a raw finding to a confirmed, deliberate author decision once the PR's own text was read.

Did not run: a dedicated git log/git blame history pass and a dedicated "do prior review comments on these files still apply" pass were not performed as separate steps (budget was spent on direct source verification of candidates instead). cargo test/cargo clippy/cargo xtask could not be run locally (no Rust toolchain in this environment); CI was read instead (gh pr checks) and is green at this head except one still-pending GPU lane (E2E tests (MI300X, current)) and a few lanes that reported skipping, which is consistent with the gating the PR body itself describes. This PR is large (65 files); it is one cohesive feature rather than several unrelated ones, well-organized across the necessary file splits (the runner.rs→runner/discovery.rs extraction exists specifically to respect this repo's 1,500-production-line module cap, confirmed not violated by any touched file), so size alone is not flagged as a blocking concern here.

Not reconciled against any prior review discussion on this PR — this review was produced blind from the code and the PR's own description only, per this review method's independence requirement.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from e5439f4 to 223933a Compare November 13, 2026 08:09
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. The three required points are addressed in 223933a, and the five thread replies cover the inline notes.

Required

  1. Scenario ids: renumbered. The dash scenarios are now dash-52..dash-54 and the serving ones serve-56 and serve-57; the @id: tags are unchanged and the branch merges cleanly with main.
  2. PR body: rewritten from the final diff. It names scenarios by @id: only, carries no commit hash or rebase history, adds the five changes you listed (the lemond_version removal, the removed collector functions, the Instances empty-state and Engines pane wording, dropping the Starting to Ready promotion on a stats scrape, and exact model-name matching on the direct route), adds a deployment note for a daemon from an older build, and limits the verification section to what ran.
  3. docs/architecture.md: the sentence now says discovery follows dashboard.daemon.enable_lemonade, which defaults to true, with RunnerOptions::default() staying off for standalone use and tests. The registry.rs row also covers the endpoint matching, and runner/discovery.rs has its own row.

Smaller notes

  • A TUI attached to an older daemon, and the startup Detect request: see the thread replies.
  • serve_http_resolving marking an existing state file failed: not changed. The path is this serve's own service record, and a failed launch of that service is the case where an earlier running or ready record would mislead; the helper's doc comment says so. Agreed it is low impact.
  • The repeated argument group and the X / X_with_parent wrappers: not changed here. The wrappers keep the confined parent runtime directory explicit for tests; folding them away is a refactor of its own and a reasonable follow-up.
  • cargo xtask smoke was run on Windows after the rebase and passes. cargo xtask coverage was not run locally; the CI coverage job covers it.
  • The W7900 lane was not touched by this change.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Replies to the automated reviews on the earlier heads, against 223933a.

Fixed

  • The description cited scenario numbers that drift on every rebase. It now names scenarios by @id: only and was rewritten from the final diff.
  • The new dash-lemonade-managed-inference-journey expectations row cites ROCMAI-898, the open ticket for the broad condition, like its sibling rows.
  • The registry.rs row in docs/architecture.md now covers the endpoint matching the file gained.
  • docs/testing.md cites efficiency.rs for node_efficiency_leaves_out_a_last_request_rate.
  • A test in rocm-dash-collectors ties LEMONADE_PORT to the dashboard config's default port.
  • The instance detail's recent-throughput panel no longer lists a last-request rate as activity, with a test; this also removes the repeated "active" samples an idle Lemonade server produced there.
  • The new service: and status: lines of rocm services stop are now asserted by the two default-lane stop scenarios, next to the on-disk status, so they are no longer covered only by the GPU scenario.

Decisions, confirmed

  • The mixed Instances table (live and last-request rows under one legend) stays as disclosed under "Not verified".
  • No scenario covers sharded GGUF resolution or the router-to-direct fallback; both are unit-tested and the gap stays disclosed.
  • @id:serve-lemonade-simple-registered-direct-route has no flaky expectations row on purpose, so a route regression fails its lane.
  • The unattended download after a router failure and the hardcoded --n-gpu-layers 999 follow from the GPU-required rule, and both are listed in the description. The throwaway download lemond runs on its own free port; I have not verified its interaction with Lemonade's own runtime directory, which is outside this repo.

Not changed

  • Scrape ordering: the Lemonade phase runs before the vLLM scrape within a daemon tick. Each Lemonade request has a bounded timeout (1.5 s for managed endpoints), endpoints are scraped in parallel, and the phase does work only when discovery is on and an endpoint exists. Running the two phases concurrently is a larger change to the runner and is left for a follow-up.
  • render_service_action_result reads three result shapes; a shared tagged type between service_control.rs and services_text.rs would make a fourth shape a compile error. Left for a follow-up.

CI note: the MI300X e2e lane's failures are vLLM serves dying at startup with RuntimeError: operator torchvision::nms does not exist in the pre-warmed ROCm 10.1.0 runtime. None involve Lemonade or dashboard scenarios.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 13, 2026 08:13

Addressed in 223933a: the scenario ids are renumbered, the description is rewritten from the final diff, and docs/architecture.md is corrected. Every point has a reply (#631 (comment)). Please re-review.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 223933ae

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (base 691834a0 → head 223933ae, 65 files, +11566/-2356). No ticket identifier (e.g. ROCMAI-###) is named in the PR description, so Step 9's ticket check is informational only: this reads as a self-contained fix/feature PR, not tied to an external tracker item.

Blocking

  • No test proves a successful Lemonade scrape leaves a Starting instance Starting — missing coverage, crates/rocm-dash-daemon/src/runner.rs:800-910 (Lemonade scrape block) vs. :989-990 (the vLLM/Docker block, which does promote Starting→Ready on a successful scrape, with a comment explaining why Docker-discovered instances need that promotion). crates/rocm-dash-daemon/tests/telemetry_contract.rs has exactly one test touching InstanceStatus::Starting (lemonade_protected_managed_endpoint_keeps_registry_status_without_telemetry_warning), and its health route always answers 401/403 — a failing health check — so it proves status survives a failed probe, never a successful one. The PR's own description lists this as a named user-visible behavior change ("A successful Lemonade stats scrape no longer promotes a Starting instance to Ready; status comes from the registry and health."), and the two code paths sit a few hundred lines apart in the same file and same function, making a future copy-paste regression (promoting on a successful Lemonade scrape, mirroring the adjacent vLLM pattern) easy to introduce with nothing to catch it.
    Confidence 80 · architectural · Fix: add a test with a managed record status: "starting" whose Lemonade health and stats both succeed, asserting the row's status stays Starting.

  • README.md asserts native Windows has no live dashboard, contradicting two other statements in the same file — README.md:261 (new in this diff): "Native Windows has no live dashboard, so it has no external discovery." This directly contradicts README.md:250-253 (unchanged): "Live mode works on Windows too, but shows only GPU/CPU/memory — the serving-instance registry needs the Unix domain socket telemetry, so that view is Linux/WSL2 only," and README.md:1552 (unchanged): "Live mode works on Windows too, but shows only GPU/CPU/memory — there's no rocmdashd unix socket to connect to there...". Both of the unchanged statements are backed by an actual #[cfg(windows)] client code path that does connect and stream a reduced live view. docs/manual-testing.md has the correctly-scoped version of the same claim ("Native Windows has no live dashboard telemetry"), so the new README line simply dropped the qualifier and became a flat overgeneralization.
    Confidence 88 · mechanical · Fix: scope the new sentence the same way the rest of the file and docs/manual-testing.md already do, e.g. "Native Windows has no live dashboard telemetry [for this feature], so it has no external discovery."

Non-blocking

None

Decisions for the author

  • Last-request vs. live throughput is disclosed as ambiguous only via a legend under the Instances table, not per row — tradeoff. The PR's own "Not verified" section already names this precisely: a table mixing live and last-request rows has no way to tell which rows the legend refers to, and no test covers a mixed table. Confirming this is an accepted, temporary tradeoff rather than an oversight.

Positive signals

  • Every GPU-backend selection path (router, direct, and the Linux router-failure fallback) fails closed on "cpu" and unknown backend names by construction, not by a late check — llamacpp_backend.rs's backend-directory lookup simply has no "cpu" entry, so a mis-selected backend can't silently resolve a binary at all.
  • The router-to-direct fallback's ordering (stop the router's server, write starting with a null server_pid, then start the direct serve) is traced correctly end to end and is itself pinned by a dedicated test, closing a real window where stop could otherwise target a dead process.
  • Catalog/shard/endpoint matching throughout the new code consistently correlates by identifier (shard index, service id, model name) rather than by list position, matching the project's own stated convention against positional pairing.

Deployment notes

  • Ownership of existing managed services defaults to unmanaged after this change: recordings made before it, and a dashboard attached to a separately-started older daemon, will show no managed services in the Services overlay, and the quit prompt will never offer s for them. This is disclosed in the PR body as intentional.
  • dashboard.daemon.enable_lemonade defaults to true, so upgrading causes rocm dash to start probing 127.0.0.1:13305 read-only on Linux/WSL by default unless a user's config already sets it to false.

What this covered

Every file in the diff, read completely at the PR head and judged against the base, split across independent passes by coupling: direct_llama.rs standalone; lib.rs+state.rs+backend_alignment.rs+device_policy.rs; router_serve.rs+process.rs+llamacpp_backend.rs+runtime_dir.rs+test_http.rs; the dash collectors + core metrics group; runner.rs+runner/discovery.rs+registry.rs+demo.rs; the daemon's telemetry/discovery tests; the dash-TUI app layer; the dash-TUI services-manager/tabs layer; the CLI commands + core config group; the e2e feature/fixture files; the e2e step-definition files; the docs/README group; plus a dedicated cross-cutting pass against this repo's AGENTS.md (GPU-required/no-CPU-fallback, subprocess/FFI checklist, module-size and architecture-doc entries, behavior-claim consistency across surfaces, coverage floors, shared-UI-component reuse, and message-and-behavior co-assertion). Reviewed against 691834a04540743d0dced173bd2ef70c1244b351…223933ae7288750d31ebef8e9334726d6ddeaf27. CI was read (not re-derived): all required jobs, including clippy (which enforces the 1,500-line module-size cap and the docs/architecture.md entry requirement), report success as of the reviewed commit; one GPU lane (E2E tests (MI300X, current)) was still queued and two matrix jobs were skipped, so GPU-hardware coverage for the Linux lanes the PR itself flags as "not run locally" remains unconfirmed by this review.

Did not run: a dedicated git-log/blame history pass, and a pass checking whether review comments on earlier changes to these files (from other PRs) apply here — both skipped given the size of this change; nothing in the files read suggested either would have surfaced something the other passes missed. Design review (Step 7) was done by reasoning over the evidence already gathered across the passes above rather than via separate dedicated workers. This review was not reconciled against this PR's own (long) prior review-thread discussion — it was produced blind, from the diff and source only, per its methodology.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this PR and have a few things worth addressing before merge. None of them is a hard blocker on its own, but together they're enough that I'd like to see them handled or answered. Details are inline.

  • dash-41 lost its clean-exit check. The "quit the dashboard / exits successfully" steps were removed from that scenario and the description doesn't say why. If the behavior changed (e.g. a quit prompt with a loading managed service), it should be stated and covered rather than the assertion dropped.
  • Direct fallback hides why it wasn't taken. direct_fallback_route turns a model-resolution error into "no direct route", so the user only sees the router error.
  • Timeouts read as "nothing listening". send_error maps is_timeout() to connection-refused for the Windows closed-port case, but external discovery only runs on Linux/WSL, where a slow-but-alive server will look absent.
  • One failed probe retires every external row. The managed path keeps its hold on failure; the external path drops everything and re-announces on the next good probe, which may flap a loaded server. Fine if intended, but worth a one-probe grace or an explicit note.
  • Scope. This bundles registered-GGUF resolution, router-to-direct fallback, dash external discovery, ownership gating and last-request throughput semantics. The dash discovery/ownership/UI work looks separable from the engine serving work; splitting would make review and bisecting easier.
  • Test evidence. The description lists passing runs only; please note that the new tests were shown to fail with their target code broken.

Also worth a decision from you: external probing of 127.0.0.1:13305 is on by default for every rocm dash run on Linux/WSL.

CI is green apart from the skipped GPU lane.

Scenario: dash-41 - Home counts a still-loading model as running and shows its stage
Given a managed model is still loading
When the user opens the dashboard
Then the Home view counts the managed model as running and shows it as loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This scenario used to end with quitting the dashboard and asserting a clean exit. Those two steps are removed here and the PR description doesn't mention it. Is there a reason the exit check no longer holds for a still-loading managed service? If so, please note it (ideally cover it another way); otherwise please restore the steps.

if !is_linux {
return None;
}
let model = resolve_direct_llama_model(manifest, &request.model_ref)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

resolve_direct_llama_model(..).ok().flatten() drops an Err (for example a variant matching several cached files), so the caller only sees the router error and never learns why the direct fallback wasn't taken. Consider carrying the resolution error into the returned context.

/// request timeout.
fn send_error(url: &str, error: &reqwest::Error) -> CollectorError {
let message = format!("GET {url}: {error}");
if error.is_connect() || error.is_timeout() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_timeout() is treated as connection-refused for the Windows closed-port case, but external discovery only runs on Linux/WSL. There a slow but alive server reads as "nothing listening" (silent on first contact, and rows get dropped when already known). Would limiting this to is_connect(), or excluding already-known rows, be enough?

// with nothing listening stays quiet; one that answers with
// an HTTP or body error warns even on first contact.
// Managed failures retain only the existing hold.
if endpoint.discover_external {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One failed or timed-out health probe retires every external row, purges the rate baselines and announces them gone; the next good probe re-announces them. The managed path keeps its hold on failure. Is the immediate drop intended for a busy server, or would a one-probe grace avoid flapping?

@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from 223933a to 615eb39 Compare November 13, 2026 10:18
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Replies to the automated review on 223933a, fixed in 615eb39.

Fixed

  • A managed Lemonade row whose registry record says starting is no longer assumed to be promoted by a successful scrape. a_successful_lemonade_scrape_leaves_a_starting_managed_row_starting runs a real scrape (health and stats succeed, telemetry reaches the row) and asserts the row stays Starting.
  • The README now says native Windows has no live dashboard telemetry, so it has no external discovery.

Unchanged

  • The mixed Instances table (live and last-request rows under one legend) stays as disclosed under "Not verified".

r0x0r
r0x0r previously requested changes Nov 13, 2026

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through this change in depth — all 65 files, with the surrounding code, plus the project's own AGENTS.md rules, the module history, review threads on earlier PRs that touched these files, and a run of the test suites with targeted mutations to see which new tests actually fail when the code they cover is broken.

First, credit where it's due: the new tests are genuinely strong. I broke roughly a hundred individual branches one at a time and the suite caught nearly all of them — every_path_that_clears_the_rate_also_clears_the_direct_flag alone catches three independent mutations, and assert_record_rewrite_resets_attributed_rate carries its own control so an expiry can't fake a pass. The runner.rs → runner/discovery.rs split really is a pure move (I diffed it function by function). And the PR description is unusually honest — it discloses the idle-rate refresh, the mixed-table legend, the advisory GPU scenario and the uncapped fallback download up front, which retired several things I'd otherwise have raised here.

The issues below are the ones I think need fixing before merge. They're inline. A few headlines:

  • The dashboard misreports numbers at ordinary terminal widths. The new last req label is five characters longer than gen, and the new empty-state sentence is twenty longer. At 98 columns the Services overlay shows last req 1 for a server doing 137.5 tok/s. At 97–102 the instance-detail modal silently drops the line that says how stale the rate is. I confirmed all of these by rendering frames, not by reading. Every new render test uses 120, 160 or 200 columns.
  • A healthy Lemonade server that reports all_models_loaded: null disappears from the dashboard. null isn't an array, so the guard throws the whole payload away — including status — and the rows are retired with no warning. That's the default JSON shape for an unset optional field in FastAPI.
  • Per-instance tok/W is still computed from the last-request rate. live_tokens_per_watt() was added for exactly this and Home and Observe use it, but the daemon computes the field from the raw rate and the Instances table and detail modal render it unqualified — so the node hero reads — while the row beneath it shows a number.
  • Two tests pass without running. The IPv6 tests return when the bind fails, which libtest counts as a pass with no output. They're the only cover for two branches, and CI containers often lack IPv6.
  • discovery_reconcile.rs's one edit made its assertion compare the id to the function that produced it. I broke per-model identity entirely and that file stayed green.

I also chased several things that did not hold up and am not raising them: the branch now merges cleanly, the scenario numbering in the description is fixed, E2E consolidated report and the hosted E2E lanes are green on this head, and the module split dropped no tests. The GPU Linux lanes still haven't reported for this commit, which is worth a re-run since they're what exercise the new @requires-gpu scenarios.

A couple of larger questions I left out of the inline comments because they're your call rather than defects: whether managed: bool and the container_name == "lemonade" check want to be one typed source field while managed is still new and free to change, and whether the "not live" flag belongs on gen_tps_observation (which already travels to every renderer that formats the value) rather than beside it.

};
if value
.get("all_models_loaded")
.is_some_and(|models| !models.is_array())

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A health body with "all_models_loaded": null is thrown away entirely here, including its status and the legacy model_loaded fallback this function documents.

Value::Null.is_array() is false, so is_some_and(|models| !models.is_array()) is true for an explicit null and we return LemonadeHealth::default(). I checked both ways:

  • with the guard: {"status":"ok","model_loaded":"Model-A","all_models_loaded":null} → status=None, loaded=[]
  • without it: status=Some("ok"), loaded=["Model-A"]

An explicit null is what FastAPI/Pydantic emits for an unset Optional, so this isn't hypothetical. The consequence is quiet: the runner takes the Ok path, services_from_health returns nothing, the rows are retired and no warning is raised — identical on screen to an idle router.

For every other non-array shape the guard is a no-op (serde's Option<Vec<Value>> plus unwrap_or_default() already produces the same result), so null is the only input it actually discriminates, and it handles it wrongly. Replacing it with .is_some_and(|_| false) leaves all 18 lemonade tests green, so nothing holds it.

Suggest !models.is_array() && !models.is_null(), with cases for null and {}.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b8dc364. An explicit null model list now keeps the rest of the body (status and the legacy model_loaded fallback); only a present value that is neither an array nor null is still rejected whole. a_null_model_list_keeps_the_rest_of_the_health_body covers null, {} and a string.

format!(
"gen {}",
"{} {}",
format::gen_rate_label(r.gen_tps_last_request),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

last req is eight characters where gen was three, and the preceding spans consume a fixed 65 columns, so the rate gets clipped into a different and entirely plausible number.

Rendered through the real overlay geometry, same row, gen_tps = 137.5:

terminal w=98   gen=[gen 137.5]      last req=[last req 1]
terminal w=100  gen=[gen 137.5 to]   last req=[last req 137]
terminal w=103  gen=[gen 137.5 tok/] last req=[last req 137.5]

At 98 columns a user reads 1 tok/s for a service doing 137.5. Before this change the same row showed the magnitude correctly. Clipping the unit is pre-existing; clipping the digits is new.

snapshot_labels_a_last_request_rate renders at 120×28, above the band where this happens, so it certifies the label and is blind to the value the label displaces.

Either budget this column explicitly and right-align the value, or drop the per-row prefix in favour of format::LAST_REQ_LEGEND in the overlay footer — which the Instances table already does, and which would also fix the fact that this overlay prints last req with no legend explaining it anywhere.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b8dc364. The Services overlay row now uses req, which is as wide as gen, so the rate fits wherever the live rate did, and the overlay footer carries the shared legend (Lemonade: last req tok/s, not live) the Instances table uses. a_last_request_rate_is_not_clipped_where_a_live_rate_fits renders at 98 columns, where last req 1 showed for 137.5. The quick preview keeps last req; it is free text with no column budget.

@@ -362,7 +365,10 @@ fn render_summary(
Style::default().fg(theme.accent),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The code I mean is the last req: label a few lines above at ~362, together with the Constraint::Length(4) that sizes this block at ~76–82 — GitHub would only let me anchor inside a hunk.)

The longer last req: label wraps line 2 of the summary, and because the block is a hardcoded 4 rows, the line that falls off the bottom is the gen freshness: … observed: … line — the one thing on screen that qualifies the rate.

Width sweep, same instance, realistic id and model name, two GPUs:

w=97..102   gen      → partition ✓, freshness ✓
w=97..102   last req → partition ✓, freshness ✗

With a shorter model name the band moves and both lines go. detail_names_a_last_request_rate renders at 160×48 and can't see it.

Suggest sizing the block from Paragraph::line_count at the actual width rather than a fixed 4, or shortening the label.

Separately, the tok/W: on that same line is computed from the last-request rate and rendered with no qualifier at all, right next to the rate that is labelled — see my comment on runner.rs.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b8dc364. The summary block is sized from its wrapped line count at the real width instead of a fixed 4 rows, and the recent-throughput panel is laid out from that height, so the freshness line cannot fall off. detail_keeps_the_freshness_line_at_every_width sweeps 80 to 200 columns with a long id and model name and two GPUs, for both labels; it fails when the block is pinned back to 4 rows. The tok/W: on that line now comes from live_tokens_per_watt().

match &state.conn {
ConnState::Connected { .. } => {
"no instances · start daemon with --enable-docker on a host with vLLM containers"
"no serving models discovered · open Serving to launch a managed model or check an external endpoint"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This message is 99 characters where the one it replaces was 79, and draw_empty builds the Paragraph without .wrap(...), so it's clipped rather than wrapped:

80 cols:  no serving models discovered · open Serving to launch a managed model or ch
100 cols: no serving models discovered · open Serving to launch a managed model or check an external endp
120 cols: no serving models discovered · open Serving to launch a managed model or check an external endpoint

The half that gets cut is the half naming the external-endpoint path this PR adds. The sibling in-process branch just above is 73 characters and still fits, and the detail pane in pane.rs does call .wrap(Wrap { trim: false }).

No test catches it: dash_characterization.rs renders 160×44, and the 80-column test in this file exercises the other branch.

.wrap(Wrap { trim: false }) here, or shorten to one clause.

Minor, while you're here: absent_managed_lemonade_is_reported_only_from_inventory asserts nothing its two siblings don't, and the name claims it rules out a competing source that the fixture never provides.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b8dc364. The empty state wraps instead of clipping (the_connected_empty_state_names_the_external_endpoint_path_at_80_columns). On the minor note: I renamed absent_managed_lemonade_is_reported_only_from_inventory to no_instances_and_no_managed_runtime_reports_lemonade_not_installed so the name says what it asserts; I did not add a competing source to it, since its siblings cover that.

let (table_area, legend_area) = if any_held || any_last_req {
let split = Layout::default()
.direction(Direction::Vertical)
.constraints([Constraint::Min(0), Constraint::Length(1)])

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Two things here; the code is at ~121 for the first and ~178–184 for the second.)

TOK/W renders an efficiency derived from a last-request rate as if it were current. Line ~121 passes inst.tokens_per_watt — the raw field — while home.rs switched to live_tokens_per_watt(). Rendered, one idle Lemonade instance:

Node efficiency:  —          Node throughput:  —
MODEL         TOK/S   TOK/W
gpt-oss-20b   38      0.12 tok/W
Lemonade tok/s = last req tok/s (not live)

The hero says there's no measurable efficiency; the only row on screen says 0.12 tok/W, and the legend covers tok/s only.

Related gap in any_last_req just above: it gates on gen_tps finiteness alone, so a row with gen_tps = None and a finite last-request-derived tokens_per_watt shows the stale efficiency with no legend at all. any_held handles exactly that asymmetry and documents it.

The joined legend clips at 80 columns, losing the clause that carries the whole warning:

* = held (prior scrape window)  ·  Lemonade tok/s = last req tok/s (not liv

The test asserting the joined form picks TestBackend::new(200, 20), while every other render test in this file uses 160 and table_at_80_columns_keeps_tok_w_marker_and_model_readable exists precisely because 80 is the width that matters.

Also worth knowing: breaking either guard inside any_last_req leaves the whole crate green, where each held twin has its own test; and LAST_REQ_LEGEND's wording is pinned nowhere — I replaced it with "XXXX" and the crate stayed green.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b8dc364. The row's tok/W comes from live_tokens_per_watt(), so an idle Lemonade row shows — like the hero (table_leaves_out_an_efficiency_derived_from_a_last_request_rate, with and without a rate). The legend is now Lemonade: last req tok/s, not live, short enough to join the held legend inside 80 columns (the joined-form test renders at 80), and its wording is pinned. table_has_no_last_request_legend_without_a_rate_to_explain holds the finiteness half of any_last_req; table_labels_a_last_request_rate_and_only_then holds the flag half.

Comment thread docs/architecture.md
| [`test_http.rs`](../engines/lemonade/src/test_http.rs) | Test-only: a loopback HTTP fixture that answers each request with a chosen status and JSON body and records what it received, plus (Linux) fake `lemond` and `lemonade` executables that log their arguments. | flat |
| [`test_support.rs`](../engines/lemonade/src/test_support.rs) | Test-only fixtures shared by the crate's module tests. | flat |

Boundaries:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The sentence is at 675 — anchoring to the nearest line in the hunk.)

This change inverts the import direction the bullet describes, so both halves are now false. Verified at both revisions:

  • merge base 691834a04: llamacpp_backend.rs:10 had use crate::direct_llama::find_binary_in;
  • this head: that import is gone (find_binary_in moved into llamacpp_backend.rs), and direct_llama.rs:15 now has use crate::llamacpp_backend::{backend_server_dirs, find_llama_server_binary_for_backend};

So llamacpp_backend.rs no longer uses direct_llama.rs, direct_llama.rs imports it back, and the new router_serve.rs isn't in the chain at all. cargo xtask check-architecture-doc only checks the row exists — AGENTS.md §6 is explicit that "keeping an entry's description true is still on you".

A few other entries went stale in the same way and are worth a sweep while you're in here:

  • :398 efficiency.rs still says node_throughput is "Σ gen_tps"; it's now live_gen_tps(), which is the behaviour change most likely to surprise a reader of that row
  • :436 lemonade.rs still says "discovery of a running server" for a collector that now emits one row per loaded model
  • :555 services_manager.rs still claims "live throughput"
  • crates/rocm-dash-core/src/efficiency.rs:129 and crates/rocm-dash-tui/src/ui/services_manager.rs:8 restate the same stale rules in doc comments

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b8dc364. The layering bullet now matches the imports at this head (backend_alignment to llamacpp_backend to process; direct_llama uses both; router_serve uses process for its types). I also swept the entries you listed: efficiency.rs (live gen_tps, and instance_tokens_per_watt), lemonade.rs (one row per loaded model), services_manager.rs, and the two doc comments.

model_names: vec![target.model_ref.clone(), target.canonical_model_id.clone()],
});
}
managed_lemonade_collectors.retain(|endpoint, _| groups.contains_key(endpoint));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three new branches in this block have no test. I confirmed each by breaking it and running the full daemon suite — all green in every case.

1. This retain (line 776). It's the only bound on managed_lemonade_collectors, which holds a reqwest Client and its connection pool per (host, port) ever seen. Deleting the line leaves 151 tests green. A long-lived daemon whose managed services rebind ports accumulates them for the process lifetime.

2. The scrape_now && half of the stats gate (line 839). Removing it is green, because every Lemonade test sets instance_tick: TICK, making scrape_now always true. It's live under shipped defaults though: discovery_tick 5 s and instance_tick 2 s give discovery_ticks = 5, instance_ticks = 2, so ticks 5, 15, 25 are discovery-only and must not fetch stats. One case with instance_tick = 2 * TICK and discovery_tick = TICK would pin it.

3. The partial-registry filter (line 679). It keeps any previous ManagedLemonadeTarget whose id isn't in disc.seen — but a service this same pass read and found stopped never enters seen, it enters retired. ManagedDiscovery::retired exists for exactly this and keep_through_partial_registry already applies it to managed_roots; the new lemonade vector didn't get the same rule. Adding && !disc.retired.contains(&target.service_id) leaves everything green either way, so neither behaviour is held. Beyond wasted polls: if anything later binds that host:port, the stale row groups with the live one, two rows match the health model, attributed_id collapses to None, and the live service loses its throughput.

For 3, moving the filter into keep_through_partial_registry beside its sibling would both fix it and give it a seam a unit test can reach — right now it's inline in run_loop where there isn't one.

Unrelated but in the same block: attributed_id.expect("stats require attribution") at ~900 is held off only by the condition two statements earlier, which the tuple type can't express. It's unreachable today, but a panic there kills run_loop and every snapshot with it. Returning Option<(String, Result<InstanceSample, _>)> would make it unrepresentable.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In b8dc364: (3) the partial-registry filter moved into DiscoveryState::keep_lemonade_through_partial_registry and now drops a target the pass read and found stopped (a_partial_registry_read_keeps_unseen_lemonade_targets_and_drops_stopped_ones); (2) the scrape_now half of the stats gate is held by lemonade_stats_are_fetched_on_the_instance_cadence_not_every_discovery_tick (discovery every tick, instance scrape every fifth; it fails with the half removed, 14 stats for 15 health requests); the expect is gone, since the stats now travel with the id attribution picked. Not changed: (1) the retain on the collector map. The map lives inside run_loop and holds only HTTP clients, so nothing observable depends on it, and I would have to expose it just to test it; the retain stays untested.

parent: &ParentRuntimeEnvironment,
) -> Result<LemondChild> {
#[cfg(windows)]
if let Some(log_path) = log_path {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The &[] I mean is the env_overrides argument to spawn_hidden_console_with_log at ~86.)

This Windows branch returns before lemonade_process_environment_vars is ever called, and passes an empty env_overrides. It also ignores the parent argument this change added to the signature.

So on Windows lemond starts without LEMONADE_API_KEY, HIP_VISIBLE_DEVICES, ROCM_PATH or the prepended runtime PATH — and log_path is always Some for a managed serve, so this is the normal path, not an edge.

The part that makes this worth raising on a changed function rather than leaving as pre-existing is the comment further down this file, which says translating the key here "both secures the server and keeps our readiness checks authenticated". On Windows only the second half holds: rocm serve --require-api-key would report a protected endpoint while lemond serves it unauthenticated, and the readiness probe still passes because wait_for_lemonade_cli_status_with_parent applies the vars itself. --gpu pinning is dropped the same way.

I couldn't test this — no Windows host — so it's a code reading. Either build the vars in this branch and pass them through (env_overrides needs widening to OsString values first), or refuse to start when a key is configured and can't be delivered. Either way the comment needs correcting.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed here. This Windows branch is unchanged by this PR: origin/main has the same call with an empty env_overrides, so lemond already starts without the translated key, HIP_VISIBLE_DEVICES and the runtime PATH there. You are right that it matters (a configured API key leaves the server open while the readiness probe, which applies the variables itself, still passes). I would rather fix it in its own change, since it widens spawn_hidden_console_with_log's environment type in rocm-core and needs a Windows run to check the key reaches the server. I have not tested it either.

Comment thread engines/lemonade/src/process.rs Outdated
/// non-ROCm backend notice) through another; without `O_APPEND` the child's next
/// write would land at its own offset, on top of what was appended.
fn open_service_log(log_path: &Path) -> Result<fs::File> {
fs::File::create(log_path)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fs::File::create truncates, and this runs for every lemond spawned — including the short-lived one ensure_model_in starts to pull a model, which is handed request.log_path, the same service log.

That's new behaviour on the fallback path. At the merge base a router failure went straight to resolving the GGUF from cache with no spawn; this change inserts ensure_model_in after the router failure, so a fallback that has to download wipes the router's stdout and stderr — exactly the diagnostics explaining why the fallback is happening. The reason line is appended afterwards, so the failure looks explained while the load error that caused it is gone.

A fresh: bool parameter, truncating only for the serve's own server, would do it.

While you're in this file: stop_server in router_serve.rs sends a single SIGTERM to the lemond PID and then waits unbounded — terminate_pid's force parameter is _force and discarded, so the true at the call site does nothing. rocm_core::terminate_process_tree exists for this and its own doc names the consequence ("leaves those workers reparented to init, where they keep the model resident and the device memory pinned"); vLLM and the managed stop path both use it. That's inherited rather than introduced, but this change adds the first path that kills the router and immediately launches a second GPU server on the same port and device — and the likeliest trigger of a RouterLoadFailure is an allocation failure, which is the case where an orphaned backend still holds VRAM.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b8dc364: LogMode. The serve's own server starts the log over; the helper lemond that pulls a model appends, so a fallback that downloads keeps what the router wrote (a_helper_server_keeps_the_log_it_is_given). The router's stop_server now uses terminate_process_tree, as the vLLM and managed stop paths do, so a backend that still holds device memory is signalled with it; on Windows this is the single-process terminate it was. That change has no unit test, since it needs a real process tree. I left terminate_pid's ignored force parameter alone; other callers use it.

Scenario: dash-41 - Home counts a still-loading model as running and shows its stage
Given a managed model is still loading
When the user opens the dashboard
Then the Home view counts the managed model as running and shows it as loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dash-41 lost its two closing steps here and I can't find a reason for it anywhere:

-    When the user quits the dashboard
-    Then the dashboard exits successfully

Every neighbouring dash scenario keeps the pair, the commit message doesn't mention it, and no comment was added beside the scenario. AGENTS.md §8: "Deleting or weakening a test is a change in its own right... say in the commit why."

I checked the obvious explanation and it doesn't hold — the seeded record is starting/loading and InstanceStatus::is_serving() is Ready | Running only, so the new ownership gate can't have opened the quit-confirm prompt here. Which leaves this as the one dash scenario with a live daemon that never quits the TUI, in a change that edits the quit-confirm path.

Restoring the two steps, or a one-line comment saying what made them wrong, would settle it either way.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored in b8dc364. I found no reason for dropping the two closing steps of dash-41; it was a mistake. It passes in WSL with them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on process grounds; I found no correctness defect in the engine-side code I read.

Blocking

  • Merge conflict. The PR is CONFLICTING against main. It needs a rebase, and the current green CI predates that rebase, so it needs to be re-run on the rebased head.
  • Scope. This bundles four-plus logical changes in roughly 16.6k diff lines (engine/CLI routing and GGUF resolution, the direct-llama fallback and download, and the dash Lemonade discovery/telemetry/Services work). AGENTS.md asks for one logical change per PR. Please split, at minimum engine/CLI changes from dash changes, so each can be reviewed and reverted independently.

Non-blocking

  • Two inline comments on the default-on external probe (timeouts read as "absent"; one failed probe retires every external row).
  • The fallback download runs with no size cap or confirmation. The PR body discloses this, but please consider a cap or an explicit prompt.
  • enable_lemonade now defaults to true, so rocm dash probes 127.0.0.1:13305 on Linux/WSL. It is read-only and documented, but it deserves a release-note line.
  • Test gaps: no scenario covers split-GGUF resolution or the router-to-direct fallback; the Linux GPU scenarios did not run here; the managed-inference journey scenario has a flaky expectations row, so it is not coverage. The PR body shows no evidence of the break-the-target mutation checks AGENTS.md asks for.

What I checked

  • The router-to-direct fallback is GPU-safe (no silent CPU fallback) and its state handling holds up.
  • The dash-side claims in the PR body (managed ownership, 401/403 handling for managed vs external endpoints, throughput exclusion from node aggregates) match the diff.
  • CI is green on 615eb39.
  • Not reviewed in depth: the e2e step definitions, pane.rs / serving.rs / instance_detail.rs in the TUI, and the tests in runner/discovery.rs.

/// request timeout.
fn send_error(url: &str, error: &reqwest::Error) -> CollectorError {
let message = format!("GET {url}: {error}");
if error.is_connect() || error.is_timeout() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reqwest timeout is mapped to Io(ConnectionRefused), which is_connect_error reads as "nothing is listening". A live but slow server (e.g. lemond busy loading a model) is therefore treated as absent: silent on first contact, and on a known endpoint it falls into the retire-all-rows path. The comment justifies this by Windows closed-port hangs, but the feature is on by default for Linux/WSL where a refused connection is immediate. Consider scoping the timeout-to-absent mapping to Windows, or keeping rows through one failed probe.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed. A probe that times out reading as absent is the intended rule: on Windows a closed port can hang rather than refuse, and on Linux a busy lemond that does not answer within the bound is also not usable as a serving row. The result is a brief disappearance, not a wrong figure, and the row returns on the next good probe. I did not scope it by platform because the behaviour would then differ between Linux and Windows hosts for the same server. Keeping a known row through one failed probe is the same decision as on the runner thread.

&& rocm_dash_collectors::lemonade::is_connect_error(&error);
lemonade_discovery_warn =
(!absent).then(|| endpoint.warning("health", &error));
(Some(Vec::new()), Vec::new(), None, None)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One failed external discovery probe (including a single timeout) yields an empty service set, which reconcile_lemonade turns into retire() of every row: telemetry is purged, InstanceGone is broadcast, and the rows are re-announced as new on the next good probe. A transient blip therefore flaps rows and resets the rate baseline. Managed targets get a hold on failure; external ones do not. Consider holding rows for N failed discovery passes, or retiring only on connect-refused/auth errors.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed here. A failed discovery pass retiring the external rows is the specified behaviour: @id:dash-lemonade-external-protocol-transitions asserts that models disappear when the server stops responding and return when it answers, and a hold of N passes would change what that scenario says and how long a dead server stays on screen. The cost you describe is real (the rows are announced again and the rate baseline restarts after a blip), and managed targets do hold. If you want external rows held through a pass or two, I would do it as its own change with the scenario updated to match; say so and I will.

…dels in dash

Resolve registered GGUF models from the installed Lemonade catalog instead of
falling back to the default model. Simple entries (one checkpoint with a
variant, no extra files) are served directly by the selected GPU llama-server;
anything else keeps the Lemonade router. On Linux a failed router load or
verification falls back to the selected GPU backend's llama-server, never to a
missing or CPU backend. Split (sharded) GGUF entries resolve to their first
shard, and a complete set wins over an older partial download.

Persist the name that is actually served, route complex catalog entries through
Lemonade, and restrict lifecycle actions to registry-owned services. A
non-ROCm GPU backend (Vulkan) is named in the serve output and log, on both
routes, only once the model is actually serving.

rocm dash discovers loaded models from a local Lemonade server on Linux and
WSL (read-only, configurable host and port, opt-out), groups equivalent
endpoints and never adopts an external service. A Lemonade rate is the last
completed request's rate: it is labelled, left out of the node-wide tok/s and
tok/W figures, and flagged in the agent snapshot.

rocm services stop also prints the service id and status for the in-process
stop report.

Add routing, ownership, telemetry and user-journey regressions, including
scenarios for the registered-name routes, external discovery, and the
reinstall advice for a missing llama-server.

Signed-off-by: Eugene Volen <eugene.volen@amd.com>
@volen-silo
volen-silo force-pushed the worktree-lemonade-serving-discovery branch from 615eb39 to b8dc364 Compare November 16, 2026 08:29
@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review. All twelve inline points have a reply on their threads; the fixes are in b8dc364, rebased onto current main.

Not changed

  • The Windows lemond spawn passing no environment: the branch is the same on main, so I left it for its own change (reply on the thread).
  • The retain on the managed collector map stays untested: nothing observable depends on it.

Your two design questions

  • A typed source field in place of managed: bool plus the container_name == "lemonade" check, and moving the "not live" flag onto gen_tps_observation: both are cleaner while the field is new, but each is a protocol and renderer change across crates and would not fit this change. I am not making either here.

Verification is in the description: Windows and WSL, with the WSL signing workaround and the GPU scenarios that could not run here (the managed GPU journey needs about 25 GiB of disk; this machine has 17 GiB free). The GPU Linux lanes have not reported for this head yet.

@volen-silo
volen-silo dismissed r0x0r’s stale review November 16, 2026 08:30

Addressed in b8dc364 and every inline point has a reply (#631 (comment)). The Windows lemond environment and the untested collector-map retain are deliberately not changed, with reasons on their threads.

@volen-silo

Copy link
Copy Markdown
Collaborator Author

Thanks for the review, answered against b8dc364.

Blocking

  • Merge conflict: rebased onto current main with no remaining conflict; CI is re-running on the new head.
  • Scope: I am keeping this as one PR. The engine routing and the dashboard discovery were reviewed together and share the Lemonade ownership and state model, and splitting them was considered and decided against by the author. The description separates the two halves under its user-visible changes so each can be read on its own.

Non-blocking

  • The two inline points on the external probe have replies on their threads: not changed here, with the reasoning, and an offer to do the hold as its own change.
  • The uncapped fallback download stays disclosed in the description; it follows from the GPU-required rule and I am not adding a cap or a prompt in this change.
  • The default-on probe is the first item under "User-visible changes" for the config key; it is the release-note line.
  • Test gaps: the scenario gaps and the GPU lanes remain disclosed under "Not verified". The description now lists which new tests were mutation-checked and says the rest were not.

@volen-silo
volen-silo dismissed juhovainio’s stale review November 16, 2026 08:32

Rebased onto main with no conflict (CI re-running on b8dc364); both inline points have replies, and the PR stays one change by the author decision, explained at #631 (comment) and in the follow-up comment.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read through the production changes in the Lemonade engine, the dash daemon/collector/core, the TUI quit/services paths and apps/rocm, and found nothing blocking. The behavior described in the PR body (registered GGUF resolution, router-to-direct fallback limits, dash discovery and ownership, quit-prompt gating) matches the diff.

A few non-blocking notes, left inline where they anchor to a line:

  • is_external_lemonade keys on the container name, so an unrelated Docker container named lemonade would be misclassified.
  • run_rocm_bounded duplicates run_rocm and polls with a blocking sleep on the executor thread.
  • The dash-52 row is marked flaky, so that journey can't fail its lane; worth a tracked follow-up to remove the flag once the serve is stable.
  • The PR bundles several fixes plus harness changes (the bounded e2e runner is applied to unrelated serve steps), which makes it large to review; splitting would help next time.

This is a diff-only read: I did not build or run anything. The Windows e2e lane is red on serve-01/02 (vLLM unsupported on native Windows) and serve-52 (runner download failure), which look unrelated to this change since they also fail on other PRs; the MI300X lane is still pending, and the GPU scenarios (serve-57/58) need Linux GPU runners.

/// External Lemonade discovery uses the engine marker, not the instance id.
/// This identifies provenance independently of the current serving status.
#[must_use]
pub fn is_external_lemonade(&self) -> bool {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is_external_lemonade identifies an external Lemonade by container_name == "lemonade" plus !managed. An unmanaged Docker-discovered container that happens to be named lemonade matches and gets reported as an external Lemonade runtime (and drives the Serving engine-status line). An explicit source/engine marker on Instance would be sturdier than a display name, with a test for a Docker row named lemonade.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and it is the same point r0x0r raised as a typed source field in place of managed: bool plus the name check. Not changed in this PR: it is a change to Instance and every renderer, so it should be its own change. No test covers a Docker row named lemonade today.

/// child writes to files, so an inherited handle cannot keep us waiting, and the
/// child is killed once the deadline passes. A timeout is reported as rc `-1`
/// with a note appended to stderr.
pub fn run_rocm_bounded(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_rocm_bounded duplicates run_rocm's spawn/isolation/recording path and polls with a blocking std::thread::sleep on the async executor thread. Sharing a helper would keep the two from drifting (env setup, diagnostics); if the blocking 200ms poll is intentional, a comment saying so would help.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed here. You are right that the two share a spawn, isolation and recording path and could drift; folding them into one helper is worth doing, and I did not attempt it in this PR. I have not checked whether the blocking poll is intentional, so I will not claim it is.

serve_timeout_secs = 90
flaky = true

# The dashboard journey does the same real lemonade serve as serve-lemonade-inference,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dash-52 is flaky = true (ROCMAI-898), so the only real-GPU managed-Lemonade dashboard journey can't fail its lane. Please keep a tracked follow-up so the row is removed once the serve is stable.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The row names ROCMAI-898, the open ticket for the Linux managed-serve instability; the flag should come off when that is fixed. Not changed here.

@volen-silo
volen-silo added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 6f3185d Nov 16, 2026
44 of 46 checks passed
@volen-silo
volen-silo deleted the worktree-lemonade-serving-discovery branch November 16, 2026 09:55
rominf added a commit that referenced this pull request Nov 16, 2026
#963 fixed Lemonade serving on Linux by resolving registered model
names, but the ROCMAI-898 expected-failure rows stayed in
expectations.toml. Every one of them is flaky, so their passes were
tolerated as XPASS and never surfaced as stale rows.

The lanes have been passing these scenarios since. In the main push and
merge-queue E2E runs from #963's merge (2026-11-09) to 2026-11-13, the
Strix Halo Ubuntu jobs passed every removed scenario that ran, MI300X
passed serve-07 and serve-40 (as it also did on this PR's own MI300X
job), and the W7900 jobs in that window passed them too. The
dash-lemonade-managed-inference-journey row, added by #631, has passed
in all seven Strix Halo Ubuntu jobs since that change merged.

Remove twelve rows: the default-engine, readiness-contract, Anthropic
inference/tool-use/streaming, explicit Lemonade inference (serve-07) and
background-helper (serve-40) serving scenarios, the two chat scenarios,
the bench real-serve scenario and the dash managed-inference journey.

Keep serve-50 and serve-52 as flaky rows, re-attributed to ROCMAI-1023:
their serve becomes ready, but `rocm services list` intermittently shows
no `anthropic endpoint:` row within the 30s poll. They set no
serve-timeout override, since the serve itself succeeds.

Add a guard test that resolves each of the twelve scenarios from its real
tag line on the modelled Linux Lemonade hosts (Strix Halo Ubuntu, a WSL2
guest with a usable GPU, gfx1100 bare metal) and, for the ids pinned to
Lemonade, on MI300X, failing if any row matches; the pin is held per id
and the failure message says how to proceed. Tests that used the deleted
rows as fixtures now parse inline matrices. Update docs/testing.md and
a model_serving.feature comment that still described the old rows.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants