Repository navigation
Capture backend identity for Lemonade managed-serve failures - #912
Conversation
Strix Halo Ubuntu managed serves can return success without exposing the model, with logs showing argument rejection, backend-ready/router-unavailable, or download failure. The existing xfails attribute all three to an immediate shutdown. Capture the runtime version, configured pins, candidate backend paths, SHA-256 fingerprints and version/help output alongside the archived service logs before cleanup. Probe duration and retained output are bounded, hashing is capped and runs off the async executor, and timed-out children are killed and reaped. The engine is read from both `--engine X` and `--engine=X`, falls back to the printed serve plan, and the failure report says so when no engine can be determined. Probe capture and kill/reap results are printed only when they fail. Replace the shutdown-only xfail attribution with the observed failure signatures and cover serve-40's shared serve precondition. Intermittent-failure handling and platform scope are unchanged. This is diagnostic coverage, not a confirmed backend fix: configured pins and candidate binaries are evidence of what could have run, not proof of what did. Refs #897 Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
The regression test uses a non-Linux mock capability, so it skips the relevant xfail guards and needs Linux/no-GPU cases, including one with a reported gfx target, to protect MockServer-backed chat scenarios.
1 open issue
What changed in this PR
This PR improves E2E diagnostics for Lemonade managed-serve failures tracked in #897 without changing runtime pins or the CLI’s asynchronous launch behavior.
Changes:
- Captures candidate backend identities, configured pins, and bounded version/help output before service cleanup.
- Updates expected-failure attribution to cover observed failure signatures and serve-40’s shared precondition.
- Adds diagnostic and expectation tests, plus documentation of evidence limits.
| File | Description |
|---|---|
tests/e2e-cucumber/tests/e2e/serving_steps.rs |
Collects backend evidence before stopping failed services. |
tests/e2e-cucumber/src/serve_log.rs |
Adds engine detection and backend evidence to reports; bounds reads. |
tests/e2e-cucumber/src/lib.rs |
Exposes the diagnostics module. |
tests/e2e-cucumber/src/lemonade_diagnostics.rs |
Implements backend inventory, fingerprints, bounded probes, and tests. |
tests/e2e-cucumber/src/expectation.rs |
Tests updated failure attribution and platform scope. |
tests/e2e-cucumber/features/model_serving.feature |
Updates a comment’s xfail reference to #897. |
tests/e2e-cucumber/expectations.toml |
Revises failure descriptions and adds serve-40 coverage. |
tests/e2e-cucumber/Cargo.toml |
Adds hashing and asynchronous process/I/O dependencies. |
docs/testing.md |
Documents collected evidence, limitations, and focused tests. |
Cargo.lock |
Records the E2E crate’s hashing dependency. |
Review effort: Balanced
| for platform in ["mock", "strix-windows"] { | ||
| assert_eq!( | ||
| resolve(&scenario, &cap(platform), &matrix, Included::default()), | ||
| Expectation::ExpectPass, | ||
| "{id} on {platform}", | ||
| ); | ||
| } |
There was a problem hiding this comment.
Fixed in 2649611: a new test covers Linux hosts with no GPU, with a gfx target only, and with a GPU only, and still expects the xfail for the full GPU host. Removing has_amd_gpu from bench-load-real-serve turns only the new test red, and so does removing therock_family from chat-end-to-end-local-model; the old test stays green in both cases, which confirms your point.
juhovainio
left a comment
There was a problem hiding this comment.
I went through the whole diff. I found no correctness problems in the new diagnostics; the hash worker's own deadline inside spawn_blocking, the kill-and-reap on the probes, and the boundary tests around the size cap all look right. Two small things are left as inline comments, and I'm requesting changes so they get a look before merge.
Questions for the author:
- serve-40 is now xfail+flaky on lemonade/linux. The comment in the PR admits its assertion carries no signal on that lane, so a real background-helper regression there stays hidden until #897 closes. Is that the intended trade-off?
- The
llama-server --version/--helpprobes run while the failed service is still alive. On a shared-memory APU, could a probe that initialises the GPU backend contend with the still-running router? I couldn't verify this either way. - A
llama-serverbinary larger than the 32 MiB hash cap gets no fingerprint. That is probably fine for current builds, but I haven't confirmed it.
The branch is also CONFLICTING with main (Cargo.lock, tests/e2e-cucumber/Cargo.toml, tests/e2e-cucumber/src/lib.rs). It needs a rebase, so the green CI so far is on the pre-rebase head.
| let value = if let Some(inline) = arg.strip_prefix("--engine=") { | ||
| Some(inline) | ||
| } else if arg == "--engine" { | ||
| args.next() |
There was a problem hiding this comment.
--engine followed by another flag (e.g. serve m --engine --managed) returns --managed as the engine here. The report then neither captures backend evidence nor prints the "engine unknown" line, which breaks the promise that a skipped inventory is visible. Treating a value that starts with -- as absent would fix it; a test case for it would be good too.
There was a problem hiding this comment.
Fixed in f475592: a value starting with -- after --engine (and --engine=--managed) now counts as no engine named, so the report prints the "engine unknown" line. Both forms are in the test table, and with the guard removed the test fails with left: Some("--managed").
| # the model appearing on :11435 (41984503055), and backend download failure | ||
| # (41986772728). A backend ready line does not prove a later shutdown. | ||
| # EAI-7423 described only the earlier ready-then-unavailable symptom. | ||
| # Retain flaky=true because historical runs also passed intermittently; do not |
There was a problem hiding this comment.
The header this replaces also said that flaky is per-row and not derived from bug, so a new row has to set it explicitly. That rule is gone now. Worth keeping that sentence for the next person adding a row.
…reports A serve invocation such as 'serve m --engine --managed' made the engine lookup return '--managed'. The failure report then skipped the backend inventory without printing the 'engine unknown' line that makes a skipped inventory visible. A value that starts with '--' is another flag, not an engine, so it now counts as absent, for both the separate and the '--engine=' forms. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The header comment replaced in this PR also said that flaky is set per row and is not derived from the bug reference, so a new row for the same bug has to set it explicitly. Restore that sentence next to the rule that keeps flaky=true for the #897 rows. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…tion rows The existing regression test resolves the real-GPU rows against the mock capability, whose os_family is 'other', so the os=linux condition excluded it before the therock_family and has_amd_gpu guards were consulted. Removing either guard left the test green even though Linux MockServer-backed bench and chat scenarios would then be expected to fail. Add Linux hosts with no GPU, with a gfx target but no usable GPU, and with a GPU but no gfx target, and assert that a full real-GPU host is still expected to fail. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…gerprint The size-cap tests only covered the fingerprint helper. Cover the report itself: a llama-server larger than the hash cap is still listed and probed, and the report prints the reason it has no sha256 line, so the missing fingerprint is visible in the failure output. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Cargo.lock conflicted in the e2e-cucumber dependency list: this branch adds sha2 and main adds shlex. Both are kept, in sorted order, and 'cargo metadata --locked' accepts the result. tests/e2e-cucumber/Cargo.toml and tests/e2e-cucumber/src/lib.rs merged automatically with both sides' additions. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…obe-timing behaviour Document that a value starting with -- after --engine counts as no engine named, that a llama-server over the hash cap is listed and probed with an explicit 'sha256 unavailable' reason, and that the version/help probes run before the failed service is stopped, with the GPU contention question marked as unchecked. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
|
Pushed 5f37981 with fixes for the inline notes (see the replies on each) and a merge of On your three questions:
|
The oversized-executable test asserted that the report contained the display form of a path it built with join("bin/llamacpp/rocm-stable"), while the inventory joins "bin/llamacpp" first and the backend directory second. On Windows the two spell the same path with different separators, so the substring check failed even though the report was right. Parse the candidate lines back into paths and compare those; Path equality is component-wise, so both spellings agree.
The same helper replaces a negative check in the candidate-cap test that could never fail on Windows: asserting the report did not contain the display form of the skipped candidate passed vacuously there. It now asserts the listed candidates exactly (count, the skipped one absent, the first one present).
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
|
The Windows lane found a bug in a test I added in the last push, and c05b62c fixes it. |
tests/e2e-cucumber/expectations.toml conflicted in two places. Main had removed the runtime-path-not-nested row, so it stays removed. Main also added its own serve-stop-leaves-no-background-helper row (lemonade, linux, #897, serve_timeout_secs = 90, flaky = true) with its own explanation; this branch had added an identical-condition row for the same scenario. The two are reconciled to the single row from main, whose fields match what this branch set, so the scenario is not listed twice. The header sentence that flaky is set per row and is not derived from the bug reference is kept, and the bench-load-real-serve row keeps its #897 attribution. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
|
Merged |
juhovainio
left a comment
There was a problem hiding this comment.
Overall: no blocking problems found; the scope is honest (diagnostics for #897, not a fix) and CI is green. Requesting changes on the points below.
Should address
- The
llama-server --version/--helpprobes run before the service is stopped and may initialise a GPU backend alongside the live router, perturbing the state being diagnosed. Move the probes after the stop, or confirm on the lane that they are harmless and replace the "has not been checked" note in docs/testing.md. - No test ties the new call site to the documented behaviour: the
runtime.jsonmanifest path matches what the CLI writes, and evidence is taken before the stop. A wrong path degrades silently tounavailable:on the real lane.
Smaller points
backend_candidatessearches only one level under four fixed backend dirs, and its empty-result message hints at a download problem.- 32 MiB hash cap may be below real ROCm server binaries.
- The "revisit before deleting" caveat was dropped from the
expectations.tomlheader; it is still true until the new evidence lands. - Please confirm a repo-wide
grep EAI-7423is clean. - The Windows timeout/kill path of the probe has no test and the PR text does not say so.
- The strix lane that would exercise the new report was not among the checks; please note which run confirms the report is populated.
| &crate::results_path(), | ||
| world.current_scenario.as_deref().unwrap_or("scenario"), | ||
| ); | ||
| let backend_evidence = match engine { |
There was a problem hiding this comment.
Evidence, including the llama-server --version/--help probes, is collected here, before stop_scenario_services below. llama.cpp ROCm/Vulkan builds typically initialise the backend on --version, so the probes may touch the GPU next to the still-running router and perturb the state being diagnosed. Could the probes (not the file inventory/hashes) run after the stop, or has the lane shown they are harmless? docs/testing.md currently says this is unchecked.
There was a problem hiding this comment.
Fixed in dfb844f. The probes now run after the stop: the manifest, pins, candidate paths, fingerprints and version.txt are still read here, before the stop, and the --version and --help probes execute only after stop_scenario_services returns, with their own 20 second budget. The order lives in one function, backend_evidence_around_stop, so this call site cannot reorder it, and docs/testing.md now describes what the code does instead of saying it was unchecked. I could not show on a lane whether a probe initialises a GPU backend, so I kept them off the router's side of the stop rather than call them harmless. At the previous head the Strix Ubuntu lanes ran the probes before the stop and both exited 0, which says nothing about contention either way.
| Some("lemonade") => Some( | ||
| e2e_cucumber::lemonade_diagnostics::backend_evidence( | ||
| &world | ||
| .app_paths() |
There was a problem hiding this comment.
Nothing asserts that this is the file the CLI actually writes, nor that this call precedes the stop. A wrong filename would only show up as unavailable: ... on the real lane, which defeats the feature. Consider sharing the filename constant with the producer or adding a test for the path.
There was a problem hiding this comment.
Fixed in 05bcc1c and dfb844f. The filename is now one definition, AppPaths::engine_runtime_manifest_path, which the Lemonade installer's manifest_path and this call site both use, and an engine test writes a manifest and asserts it lands at that path with the field names the report reads (it goes red when the engine builds a different path). For the order, backend_evidence_around_stop takes the stop as a closure, and its test makes the stop delete the candidate: the fingerprint in the report shows the inventory came before the stop and a recording probe shows the probes came after it. Moving either step to the other side of the stop turns that test red. On the lane, the manifest was read correctly at the previous head: the Strix Ubuntu nightly and release jobs both printed the manifest, pins and a fingerprinted llama-server.
There was a problem hiding this comment.
Correction to my earlier reply: the shared helper is AppPaths::engine_install_manifest_path, not engine_runtime_manifest_path. It was renamed in d4ca134 because the registry writer guard on main (a text match for runtime_manifest_path( in any file that also writes a file) matched the old name inside the Lemonade installer, and the new name also says what it is, the engine's own install manifest and not a runtime registry entry. The path it returns and the engine and harness call sites that share it are unchanged.
| report | ||
| } | ||
|
|
||
| fn backend_candidates(runtime: &Path) -> Vec<PathBuf> { |
There was a problem hiding this comment.
backend_candidates only looks at depth 0 and 1 under four hard-coded backend dirs, and when nothing is found the report hints at an incomplete download. Consider wording the empty case neutrally and listing the searched locations.
There was a problem hiding this comment.
Fixed in 59d7891 (with a test-order correction in f5bff5f). The search now goes three directory levels below each of the four backend directories, bounded by an entry cap per directory, and the cpu backend is still left out on purpose. An empty result no longer hints at a download: it lists every location searched, the depth, and what bin/llamacpp actually contains (or that it cannot be read), because an interrupted download, an unsearched layout and a build that was never installed look the same. A test with literal layouts shows a build three levels down is found and one four levels down is not, and it goes red when the depth is lowered. For reference, on the Strix Ubuntu lanes the b12109 build sits one level below rocm-stable, so the old depth already found it there.
| candidates | ||
| } | ||
|
|
||
| fn fingerprint(server: &Path, deadline: Instant) -> std::io::Result<String> { |
There was a problem hiding this comment.
32 MiB hash cap: please confirm real ROCm llama-server builds fit under it, otherwise the fingerprint is absent on exactly the lane this is for.
There was a problem hiding this comment.
Checked, and changed anyway as margin, in b064945 and ee08e62. The Strix Halo Ubuntu nightly and release jobs at the previous head both fingerprinted the rocm-stable b12109 llama-server under the old 32 MiB cap, so a current ROCm build fits. A statically linked multi-target build could be larger, so the cap is now 512 MiB; hashing still streams 64 KiB at a time under the inventory time budget, so memory and time stay bounded, and the over-cap line is worded from the constant. The remaining assumption, about builds larger than the observed one, is stated in docs/testing.md and the PR body.
…ine writes it to The serve failure report reads the installed runtime's manifest to find the backend directory, and built the path with its own "runtime.json" literal next to the engine's. A rename on either side would have degraded the report to "unavailable:" on the one lane it exists for, with nothing failing before then. AppPaths::engine_runtime_manifest_path is now the one definition. The engine's manifest_path and the harness both call it, and an engine test writes a manifest and asserts it lands at that path with the field names the report reads. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…pped The version and help probes executed a backend binary while the failed serve's router could still be running. llama.cpp builds commonly initialise a GPU backend even for --version, so on a shared-memory device a probe could contend with the router and change the state the report exists to describe. docs/testing.md said this had not been checked. The report now reads files first and executes second: manifest, pins, candidate paths, fingerprints and version.txt are taken before the stop, and the probes run after it, with their own 20 second budget so a slow stop cannot use it up. The order lives in one function, backend_evidence_around_stop, which also runs the stop exactly once for every engine, so the serve step cannot reorder or skip it. Its tests make the stop delete the candidate, which shows the fingerprint came before the stop and, through a recording probe, that the probes came after it. docs/testing.md and the expectations header now state what the code does. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…empty search looked The inventory looked only at depth 0 and 1 under four backend directories, while the router chooses how a llama.cpp build is unpacked, so a build one level deeper would have produced no candidates and no probe. When nothing was found, the report also suggested the backend download may not have completed, which an unsearched layout or a never-installed build would equally explain. The search now goes three directory levels below each backend directory, bounded by an entry cap per directory. An empty result names every location searched, the depth, that the cpu backend is deliberately left out, and what bin/llamacpp actually contains, without naming a cause. docs/testing.md states the search. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…sorts them The previous commit's depth test listed its two expected candidates in the wrong order: the search sorts sibling directories by path, so a/b comes before a/build. The test therefore failed even with the depth limit in place. With the order corrected it passes, and with the limit lowered to two it fails because the three-level candidate is no longer found. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…r guard does not match it The registry guard in runtime_registry.rs reads each production file as text and flags one that both writes a file and spells runtime_manifest_path(. The shared helper added for the Lemonade install manifest was called engine_runtime_manifest_path, which contains that spelling, and the Lemonade installer both calls it and writes its manifest, so the guard reported a second registry writer. It only showed once the guard reached main. The helper is now AppPaths::engine_install_manifest_path, which also says what it is: the engine's own install manifest, not an entry in the runtime registry. The doc comment records why the names must stay apart. The path it returns is unchanged, and so is the guard. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🟡 Changes recommended
The diagnostics report can read and embed an outside file through a symlinked version.txt despite reporting the link as skipped, so the marker needs a non-symlink regular-file check and regression coverage.
2 open issues
Review effort: Balanced
… directories The directory walk counted a symlinked version.txt as a link it did not follow, but the report then read version.txt through read_clamped, which follows it, so a real llama-server beside a marker linking outside the backend directory put the outside file's content into the report while the report said the link was skipped. Every file the report reads from the tree now goes through the same rule: the manifest, the pins file, version.txt and a candidate's fingerprint target are refused when the last path component is a symlink, and the report says "symlink not followed" in place of the content. The symlink regression fixture gains the real server with a version marker, pins file and manifest that link outside the tree; the test asserts the outside content is absent and each refusal is named. Letting the link be followed again turns it red. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because FIFO reads can hang inventory and prevent service cleanup, while directory-read failures can misleadingly report that no candidates were found.
1 open issue
1 resolved since last review
Review effort: Balanced
juhovainio
left a comment
There was a problem hiding this comment.
No blocking issues found. Approving; the points below are non-blocking.
- A Lemonade serve with no isolated root panics in
serve_failure_evidencebefore the stop runs, losing the failure report (inline). read_clampednow truncates a grown log silently for existing callers too (inline).- #897 is closed but is still cited by all eight xfail rows and the docs link (inline).
- Small
is_symlink-then-open race against the docs' "symlinks refused" claim (inline). - Cosmetic: "truncated ... more may exist" prints when there are exactly 8 candidates, and "remaining evidence skipped" can print twice when the search ends on the deadline.
- The 5 s child-pid wait in the kill/reap test could flake on a loaded runner.
- The PR-body checklist boxes are still unchecked.
The stop-exactly-once and read-before/probe-after ordering is well covered by tests that assert behavior rather than wording, and CI is green across lanes.
| e2e_cucumber::lemonade_diagnostics::backend_evidence_around_stop( | ||
| move || { | ||
| app_paths | ||
| .expect("a Lemonade serve runs in an isolated scenario root") |
There was a problem hiding this comment.
This expect runs before the stop, so a Lemonade serve without an isolated root panics here, never stops the service, and drops the failure report. The previous code reported "not attempted: scenario has no isolated root". Could this degrade to an "unavailable" evidence string and still run the stop?
There was a problem hiding this comment.
Confirmed and fixed in 96f67f0. The expect ran in the manifest lookup before the stop, so a Lemonade serve without an isolated root panicked and the stop and the whole failure report were lost. The lookup now returns Err(reason) when there is nothing to look for, the report says backend evidence unavailable: the scenario has no isolated root, the stop's own status still says it was not attempted, and the stop runs exactly once with the read-before and probe-after order untouched. A panic inside the lookup is caught and reported the same way, and a_failing_manifest_lookup_still_runs_the_stop_once_and_reports_why covers both the error and the panic; removing the panic guard turns it red. For the class sweep I checked the rest of the pre-stop path: the only other expect is on a child pipe that is configured to be piped and runs after the stop, and the scan and fingerprints run in spawn_blocking with their join errors handled.
| if len <= max { | ||
| let mut body = Vec::new(); | ||
| file.read_to_end(&mut body)?; | ||
| file.take(max).read_to_end(&mut body)?; |
There was a problem hiding this comment.
take(max) now silently truncates a log that grew after metadata(), for existing callers too, with no truncation note (the note only exists on the len > max branch). Worth adding a note or a comment on why this is acceptable.
There was a problem hiding this comment.
Confirmed and fixed in 2c5a106. read_clamped now asks for one byte past the cap on the fits branch, so a log that grew after its size was read is clamped to its head and tail like an oversized one and the note says the file grew while it was read; a file that really fits is still returned verbatim with no note. All callers consume the note: the archived-log line, the pins and version.txt in the Lemonade report, and the manifest read, which discards it but fails to parse a cut JSON document and says so. a_log_that_grew_after_its_size_was_read_is_clamped_and_says_so hands the inner function a stale size, and not requesting the extra byte turns it red. The branch for a file that shrank during the read is not tested, because it needs the size to change between two reads inside one call.
| when = { effective_engine = "lemonade", os = "linux" } | ||
| bug = "EAI-7423" | ||
| reason = "Lemonade managed serve reaches ready then shuts down immediately, so the endpoint never becomes reachable." | ||
| bug = "#897" |
There was a problem hiding this comment.
#897 is closed (2026-11-05, by the author), but this row, its siblings and the docs link cite it as the live tracker. Reopen it or repoint to the open ticket.
There was a problem hiding this comment.
Done in 2840f96, and #897 stays closed. Every bug field, the comments around the rows, the harness tests that pin them and the docs now name the open ticket ROCMAI-898, with one header note that these failures were first tracked as #897, since closed. The stale-xfail hint reads the ids out of the matrix and hard-codes none, so it follows the new id; I ran it on the real file (a body mentioning ROCMAI-898 lists all eight rows, one mentioning #897 lists none) and added that case to its self-test. The rows' conditions, timeouts and flaky = true are unchanged, and the PR body now refers to the open ticket without naming its key, so merging this PR cannot close it through the key-in-body automation.
| if is_symlink(server) { | ||
| return Err(std::io::Error::other(SYMLINK_NOT_FOLLOWED)); | ||
| } | ||
| let mut file = std::fs::File::open(server)?; |
There was a problem hiding this comment.
The is_symlink check followed by File::open leaves a window where the path can become a link, while the docs say links are refused. Consider a no-follow open on unix, or soften the claim.
There was a problem hiding this comment.
Confirmed and fixed in 4293a30. On unix every read of a file in the backend tree (manifest, pins, version.txt, fingerprint target) now goes through one helper that opens with O_NOFOLLOW through the libc the crate already depends on, so there is no window between a check and the open and the refusal message is unchanged; the_open_itself_refuses_a_symlink calls the helper directly on a link, which is what a swap after the check looks like, and removing the flag turns it red. On Windows the behaviour is unchanged (a metadata check, then an open), so a swap between the two is not caught there; docs/testing.md now says so, and that only the last path component is checked.
Two further points from the review body, which fit this thread. The "truncated ... more may exist" note printed for exactly eight candidates; in 198e385 the search keeps eight and stops on finding a ninth, so the note appears only when something is not listed, and "remaining evidence skipped" is now printed once when the search ends on the clock (a test for each, red when the old behaviour returns). And the kill/reap test's fixed five-second wait for the child's pid could flake on a loaded runner; in 2e39826 an attempt whose child printed no pid is retried with a 20 then 60 second timeout, so the generous waits matter only on a slow start and a healthy run still takes five seconds (with a child that starts in seven seconds it passes on the second attempt, and with the retries removed it fails).
…fest lookup fails The serve step looked the runtime manifest up with an expect that runs before the service is stopped. For a Lemonade serve without an isolated scenario root it panicked, so the stop never ran and the whole failure report was lost, where the code before the evidence change reported "not attempted: scenario has no isolated root". The lookup now returns Err(reason) when there is nothing to look for, and a panic inside it is caught and treated the same way. The report says "backend evidence unavailable: <reason>", the stop still runs exactly once, and nothing is probed. The only other expect on that path is on a child pipe that is configured to be piped and runs after the stop. The test covers both the Err and the panic and goes red when the panic guard is removed. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
read_clamped read a file whose size fit the cap with take(max), so a log that grew after the size was taken was cut at the cap with no note, for the archived service log and for the pins and version.txt in the Lemonade report alike. The note existed only on the branch for a file that already exceeded the cap. The read now asks for one byte past the cap. If it is there, the file grew, and it is clamped the way an oversized file is, head and tail with the gap marked, and the note says the file grew while it was read. A file that fits is still returned verbatim with no note. The size is a parameter of an inner function so a test can hand it a stale one; with the extra byte not requested the test is red. All callers consume the note (the manifest read discards it, but a cut JSON manifest fails to parse and the report says so). Signed-off-by: Jussi Elo <jussi.elo@amd.com>
GitHub issue #897 is closed, but the xfail rows, their header comments, the harness tests that pin them and the docs still named it as the live tracker. The open ticket is ROCMAI-898, so every row, comment, test and doc line that named #897 as the tracker now names ROCMAI-898, with one note in the header that it was first tracked as #897 and that issue is closed. The issue stays closed. The stale-xfail hint reads the ticket ids out of the matrix and hard-codes none, so it follows the new id on its own; its self-test now covers that (a PR that mentions ROCMAI-898 is told about the rows, one that mentions #897 is not). The rows' conditions, timeouts and flaky handling are unchanged. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The report refused symlinked files by reading the path's metadata and then opening it, which left a window in which the path could become a link while the docs said links are refused. On unix the open now passes O_NOFOLLOW (through the libc the crate already depends on, so no new dependency), so a link is refused whenever it is there at the moment of the open, with the same "symlink not followed" message. Manifest, pins, version.txt and fingerprint reads all go through that one helper. On Windows the behaviour is unchanged: a metadata check followed by an open. A swap between the two is not caught there, and docs/testing.md says so, and that only the last path component is checked. A unix test calls the open helper directly on a link, which is what a swap after the check looks like to it, and goes red when O_NOFOLLOW is removed. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…, and say the budget is out once Two cosmetic lies in the Lemonade report. The "truncated after 8 candidates" note was printed when exactly eight candidates existed, because the search stopped on reaching the cap and could not tell whether a ninth existed. And "remaining evidence skipped: evidence budget exhausted" was printed twice when the search ended on the clock and the candidate loop then found the budget gone too. The search now keeps eight and stops on finding a ninth, so the note appears only when there is something not listed, and exactly eight is a complete list. The budget line is said once. Both have tests, and each turns red when the old behaviour returns. docs/testing.md states the ninth-candidate rule. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ts slowly The kill and reap test lets a helper child run for a fixed 5 seconds and reads its pid from the probe's report. On a loaded runner a cold test binary can take longer than that to start and print the pid, which failed the test for a slow start and not for anything the probe did. An attempt whose child printed no pid before the timeout is now retried with a longer timeout (20 seconds, then 60), and only an attempt that did report a pid is asserted on, so the generous waits matter only when the start is slow and a healthy run still takes 5 seconds. With a child that takes 7 seconds to start the test passes on the second attempt, and with the retries removed it fails. Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The kill and reap test retries with timeouts of 5, 20 and 60 seconds, but the helper child exited by itself after 30. If the first two attempts timed out before the child printed its pid and the last one started it, the child exited normally before the 60 second timeout, so the probe saw a normal exit, not a timeout, and the test failed for a correct probe. Reproduced with a child that takes 25 seconds to start: the third attempt reports "exit status: 0". The helper's lifetime is now derived from the attempt timeouts (the longest plus a 60 second margin), so they cannot drift apart, and a test pins that the child outlives every attempt by at least 30 seconds. With the same 25 second start the test now passes on the third attempt; with a literal lifetime or no margin the pinning test fails. A child that takes 7 seconds to start still passes on the second attempt and fails with the retries removed. The other helper lifetimes in the file (30 second sleeps against 1 second timeouts, an immediate exit against a 60 second timeout) have no such pair. Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Strix Halo Ubuntu managed serves return success without exposing the model, with logs showing argument rejection, backend-ready/router-unavailable, or download failure. Existing xfails attribute all three to an immediate shutdown.
version.txtare read before the service is stopped, and thellama-server --versionand--helpprobes run after the stop returns, with their own 20 second budget. Both steps and the stop sit in one function,backend_evidence_around_stop, which runs the stop exactly once for every engine, and nothing before the stop can keep it from running: a Lemonade serve with no isolated scenario root reportsbackend evidence unavailable: the scenario has no isolated rootinstead of panicking, and a panic while looking the manifest up is caught and reported the same way. The runtime manifest is read throughAppPaths::engine_install_manifest_path, the same helper the Lemonade installer writes it with, so the two cannot drift. Candidates are searched underrocm-stable,rocm-nightly,rocmandvulkandown to three directory levels (the cpu backend is deliberately left out). The search is bounded as a whole: the inventory's 20 second deadline starts at entry and is checked at every directory and entry, at most 128 directories and 2048 entries are read across all backend directories, and the walk ends as soon as it finds a ninth candidate, in sorted order (it lists the first eight and says the list is cut only when a ninth exists, so exactly eight is not reported as truncated). Symlinks are never followed (a link cycle ends, and a link out of the backend directory is not read; the report counts them, and a symlinked manifest, pins file,version.txtor fingerprint target is refused withsymlink not followedinstead of being read; on unix the refusal is made by the open itself withO_NOFOLLOW, so a path swapped for a link after an earlier check is still refused, while on Windows it is a metadata check followed by an open, so a swap between the two is not caught, and only the last path component is checked), and a search cut short by the clock or a limit says which and never emits the completed-empty-search text, which is reserved for a search that finished and found nothing; the budget-exhausted line is printed once. The manifest and pin reads, the search and each fingerprint run on blocking threads with that in-traversal deadline. An empty search lists the locations it covered and whatbin/llamacppholds (16 names out of at most 64 entries read, with the omitted count), and does not guess at a cause. Bound probe duration and retained output; kill and reap timed-out children. A log that grows past the read cap after its size was taken is clamped to its head and tail with a note that it grew, instead of being cut silently. The engine is read from both--engine Xand--engine=X, falls back to the printed serve plan, and the failure report says so when no engine can be determined. A value that starts with--after--engine(for exampleserve m --engine --managed) counts as no engine named, so the report still prints theengine unknownline instead of silently skipping the inventory. Executables up to 512 MiB are fingerprinted (streamed 64 KiB at a time under the inventory's time budget); a larger one is still listed and probed, and its entry sayssha256 unavailable: executable exceeds 512 MiB hash limit.bugfield and the comments, harness tests and docs around them name the open tracking ticket recorded inexpectations.tomlinstead of the closed GitHub issue E2E: lemonade serving fails on the Strix Halo Ubuntu lanes, with a different log signature depending on the run #897 that first tracked these failures (that issue stays closed); the stale-xfail hint reads ids out of the matrix, so it follows the new id, and its self-test now covers that. The header keeps both the rule thatflakyis set per row and is not derived frombug, and the caveat to revisit before deleting any of these rows until a failed run on the lane has carried the new backend evidence. The regression tests include Linux hosts with no GPU, with a gfx target but no usable GPU, and with a GPU but no gfx target, so removing either thetherock_familyor thehas_amd_gpuguard from the bench and chat rows turns a test red; the earlier mock-capability case could not, because itsos_familyisotherand theos = "linux"condition excluded it first.AppPaths::engine_install_manifest_pathhelper inrocm-core, which the Lemonade engine's ownmanifest_pathnow calls; it returns the same path as before.Verification. The search bounds are pinned by unit tests that go red when each guard is removed: the clock read during the walk (driven by an injected clock, no sleeping), the whole-search entry and directory limits, the cap ending the walk when reached, symlink handling on Unix with a real link cycle, the bounded listing, and the scan running off the async executor. The ordering is pinned by unit tests in
lemonade_diagnostics.rs: the stop step deletes the candidate, so a fingerprint in the report shows the file was read before the stop, and a recording probe shows the probes ran after it; moving either step to the other side of the stop turns the test red. An engine test writes a manifest and asserts it lands at the shared path with the field names the report reads, and it goes red when the engine builds a different path. The new fixes each have a test that goes red when the fix is removed: the manifest lookup returning an error or panicking still runs the stop once (removing the panic guard fails it), a log that grew after its size was read is clamped with a note (a read that does not ask for the extra byte fails it), the open refuses a symlink even when it is called directly on one, as after a swap (removingO_NOFOLLOWfails it), exactly eight candidates are not reported as truncated and the budget line is printed once, and the rows' ticket id is asserted by the expectations tests. The probe's timeout, kill and reap path now runs on every platform: the unit tests re-run the test binary itself as the child (an ignored helper test selected by an environment variable), let the probe time out, and require the child to be gone afterwards (run with the child on one test thread and on several, because libtest labels a single-threaded child's output), and leaking the child instead of killing it turns that test red; a child that is slow to start is retried with a longer timeout rather than failing the test. The shell-scriptllama-serverstand-ins used by the other probe tests remain unix-only. Not covered anywhere is thekill/reap failedbranch, which needs an operating system that refuses to kill a child it just started. I could not run the Windows lane locally;clippyfor the Windows target passes for both the library and thee2etest target, and the Windows lane on the push is what runs the new child-process tests there. The e2e scenarios themselves were not run locally (no GPU).Lane evidence. At the previous head, the Strix Halo Ubuntu nightly and release jobs both printed the new report five times each (
E2E tests (Strix Halo, Ubuntu, nightly, current)andE2E tests (Strix Halo, Ubuntu, release, current)in the E2E self-hosted run for that head): the manifest was read from the data dir, therocm-stableb12109llama-serverwas found one directory below its backend directory and fingerprinted under the old 32 MiB cap, and--versionand--helpexited 0. That run used the old probe order (probes before the stop), so it shows the report is populated and the manifest path is right, not that the new order changes anything. The same two jobs on this push are what confirm the new order: the report'sprobes, run after the service stop was attemptedsection should follow itsstopsection. The W7900 and MI300X lane jobs had not reported at the time of writing.Known trade-offs, stated so they are not read as more than they are. serve-40 (
serve-stop-leaves-no-background-helper) is xfail and flaky on lemonade/linux because it needs the same real serve, so until the tracking ticket for these failures closes its "no background helper" assertion carries no signal on that lane: a pass there does not show the scenario's behaviour is healthy and a failure is not attributed to it. The guard itself stays covered by the unit testbackground_helper_not_spawned_when_no_automation_is_enabledinapps/rocm/src/service_launch.rs(which checks the gate function, not theservecall site) and by scenarioautomations-11on the mock lane (the helper's own lifecycle). Whether allama-server --versionor--helprun initialises a GPU backend has not been observed on a lane, which is why the probes now run after the stop and not next to the router; a stop that fails or finds no record can still leave the router running, and the report'sstopsection says so. The 512 MiB hash cap is an assumption about builds larger than the one observed (a statically linked multi-target build), not a measured size.Supersedes #899, which had the same content under a bot author; this is the same change started from
mainat the time, plus one commit per review fix and merges ofmain(the branch is merged rather than rebased so the existing review context keeps matching).flaky = truebecause the failure is still open.tests/e2e-cucumber/src/lemonade_diagnostics.rs, and the only change outside the e2e harness is the small path helper inrocm-core.symlink not followedtogether with the outside content being absent, and the truncation note together with the candidate count).Refs #897 (closed; the xfail rows now name the open follow-up ticket recorded in
tests/e2e-cucumber/expectations.toml)