Skip to content

Capture backend identity for Lemonade managed-serve failures - #912

Merged
jussielo-amd merged 32 commits into
mainfrom
e2e/lemonade-serve-failure-evidence
Nov 9, 2026
Merged

jussielo-amd merged 32 commits into
mainfrom
e2e/lemonade-serve-failure-evidence

Conversation

@jussielo-amd

@jussielo-amd jussielo-amd commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Strix Halo Ubuntu managed serves return success without exposing the model, with logs showing argument rejection, backend-ready/router-unavailable, or download failure. Existing xfails attribute all three to an immediate shutdown.

  • Diagnostics: Capture runtime version, configured pins, candidate backend paths, SHA-256 fingerprints, and version/help output alongside archived service logs. The report reads files first and executes a backend second: the manifest, pins, candidate paths, fingerprints and version.txt are read before the service is stopped, and the llama-server --version and --help probes run after the stop returns, with their own 20 second budget. Both steps and the stop sit in one function, backend_evidence_around_stop, which runs the stop exactly once for every engine, and nothing before the stop can keep it from running: a Lemonade serve with no isolated scenario root reports backend evidence unavailable: the scenario has no isolated root instead of panicking, and a panic while looking the manifest up is caught and reported the same way. The runtime manifest is read through AppPaths::engine_install_manifest_path, the same helper the Lemonade installer writes it with, so the two cannot drift. Candidates are searched under rocm-stable, rocm-nightly, rocm and vulkan down to three directory levels (the cpu backend is deliberately left out). The search is bounded as a whole: the inventory's 20 second deadline starts at entry and is checked at every directory and entry, at most 128 directories and 2048 entries are read across all backend directories, and the walk ends as soon as it finds a ninth candidate, in sorted order (it lists the first eight and says the list is cut only when a ninth exists, so exactly eight is not reported as truncated). Symlinks are never followed (a link cycle ends, and a link out of the backend directory is not read; the report counts them, and a symlinked manifest, pins file, version.txt or fingerprint target is refused with symlink not followed instead of being read; on unix the refusal is made by the open itself with O_NOFOLLOW, so a path swapped for a link after an earlier check is still refused, while on Windows it is a metadata check followed by an open, so a swap between the two is not caught, and only the last path component is checked), and a search cut short by the clock or a limit says which and never emits the completed-empty-search text, which is reserved for a search that finished and found nothing; the budget-exhausted line is printed once. The manifest and pin reads, the search and each fingerprint run on blocking threads with that in-traversal deadline. An empty search lists the locations it covered and what bin/llamacpp holds (16 names out of at most 64 entries read, with the omitted count), and does not guess at a cause. Bound probe duration and retained output; kill and reap timed-out children. A log that grows past the read cap after its size was taken is clamped to its head and tail with a note that it grew, instead of being cut silently. The engine is read from both --engine X and --engine=X, falls back to the printed serve plan, and the failure report says so when no engine can be determined. A value that starts with -- after --engine (for example serve m --engine --managed) counts as no engine named, so the report still prints the engine unknown line instead of silently skipping the inventory. Executables up to 512 MiB are fingerprinted (streamed 64 KiB at a time under the inventory's time budget); a larger one is still listed and probed, and its entry says sha256 unavailable: executable exceeds 512 MiB hash limit.
  • Expectations: Replace shutdown-only attribution with the observed failure signatures and cover serve-40's shared serve precondition (main now carries the serve-40 row itself, so this PR keeps that single row and only the other rows' attribution changes). Retain intermittent-failure handling and existing platform scope. The rows' bug field and the comments, harness tests and docs around them name the open tracking ticket recorded in expectations.toml instead of the closed GitHub issue E2E: lemonade serving fails on the Strix Halo Ubuntu lanes, with a different log signature depending on the run #897 that first tracked these failures (that issue stays closed); the stale-xfail hint reads ids out of the matrix, so it follows the new id, and its self-test now covers that. The header keeps both the rule that flaky is set per row and is not derived from bug, and the caveat to revisit before deleting any of these rows until a failed run on the lane has carried the new backend evidence. The regression tests include Linux hosts with no GPU, with a gfx target but no usable GPU, and with a GPU but no gfx target, so removing either the therock_family or the has_amd_gpu guard from the bench and chat rows turns a test red; the earlier mock-capability case could not, because its os_family is other and the os = "linux" condition excluded it first.
  • Scope: Diagnostic coverage, not a confirmed backend fix. Preserve the runtime pin and asynchronous managed-launch contract; distinguish configured pins and candidate binaries from proof of what actually ran. The only change outside the e2e harness is the small AppPaths::engine_install_manifest_path helper in rocm-core, which the Lemonade engine's own manifest_path now calls; it returns the same path as before.
  • Scenario coverage: This changes the e2e harness's failure reporting and xfail expectations only, not anything the CLI prints or does, so there is no new user-observable behavior to cover with a scenario. The existing serving scenarios keep running; their xfail attribution is what changes.

Verification. The search bounds are pinned by unit tests that go red when each guard is removed: the clock read during the walk (driven by an injected clock, no sleeping), the whole-search entry and directory limits, the cap ending the walk when reached, symlink handling on Unix with a real link cycle, the bounded listing, and the scan running off the async executor. The ordering is pinned by unit tests in lemonade_diagnostics.rs: the stop step deletes the candidate, so a fingerprint in the report shows the file was read before the stop, and a recording probe shows the probes ran after it; moving either step to the other side of the stop turns the test red. An engine test writes a manifest and asserts it lands at the shared path with the field names the report reads, and it goes red when the engine builds a different path. The new fixes each have a test that goes red when the fix is removed: the manifest lookup returning an error or panicking still runs the stop once (removing the panic guard fails it), a log that grew after its size was read is clamped with a note (a read that does not ask for the extra byte fails it), the open refuses a symlink even when it is called directly on one, as after a swap (removing O_NOFOLLOW fails it), exactly eight candidates are not reported as truncated and the budget line is printed once, and the rows' ticket id is asserted by the expectations tests. The probe's timeout, kill and reap path now runs on every platform: the unit tests re-run the test binary itself as the child (an ignored helper test selected by an environment variable), let the probe time out, and require the child to be gone afterwards (run with the child on one test thread and on several, because libtest labels a single-threaded child's output), and leaking the child instead of killing it turns that test red; a child that is slow to start is retried with a longer timeout rather than failing the test. The shell-script llama-server stand-ins used by the other probe tests remain unix-only. Not covered anywhere is the kill/reap failed branch, which needs an operating system that refuses to kill a child it just started. I could not run the Windows lane locally; clippy for the Windows target passes for both the library and the e2e test target, and the Windows lane on the push is what runs the new child-process tests there. The e2e scenarios themselves were not run locally (no GPU).

Lane evidence. At the previous head, the Strix Halo Ubuntu nightly and release jobs both printed the new report five times each (E2E tests (Strix Halo, Ubuntu, nightly, current) and E2E tests (Strix Halo, Ubuntu, release, current) in the E2E self-hosted run for that head): the manifest was read from the data dir, the rocm-stable b12109 llama-server was found one directory below its backend directory and fingerprinted under the old 32 MiB cap, and --version and --help exited 0. That run used the old probe order (probes before the stop), so it shows the report is populated and the manifest path is right, not that the new order changes anything. The same two jobs on this push are what confirm the new order: the report's probes, run after the service stop was attempted section should follow its stop section. The W7900 and MI300X lane jobs had not reported at the time of writing.

Known trade-offs, stated so they are not read as more than they are. serve-40 (serve-stop-leaves-no-background-helper) is xfail and flaky on lemonade/linux because it needs the same real serve, so until the tracking ticket for these failures closes its "no background helper" assertion carries no signal on that lane: a pass there does not show the scenario's behaviour is healthy and a failure is not attributed to it. The guard itself stays covered by the unit test background_helper_not_spawned_when_no_automation_is_enabled in apps/rocm/src/service_launch.rs (which checks the gate function, not the serve call site) and by scenario automations-11 on the mock lane (the helper's own lifecycle). Whether a llama-server --version or --help run initialises a GPU backend has not been observed on a lane, which is why the probes now run after the stop and not next to the router; a stop that fails or finds no record can still leave the router running, and the report's stop section says so. The 512 MiB hash cap is an assumption about builds larger than the one observed (a statically linked multi-target build), not a measured size.

Supersedes #899, which had the same content under a bot author; this is the same change started from main at the time, plus one commit per review fix and merges of main (the branch is merged rather than rebased so the existing review context keeps matching).

  • N/A (not a bug fix): this PR fixes no bug. It adds diagnostics and re-attributes the existing xfail rows; no row is removed or narrowed, and they stay flaky = true because the failure is still open.
  • N/A (no new subcommand or subsystem): nothing is added to the CLI. The new harness module is in its own file, tests/e2e-cucumber/src/lemonade_diagnostics.rs, and the only change outside the e2e harness is the small path helper in rocm-core.
  • N/A (no CLI message changed): nothing the CLI prints is touched. The harness's own report lines were each read against the code that produces them and are asserted together with the state they describe (for example symlink not followed together with the outside content being absent, and the truncation note together with the candidate count).

Refs #897 (closed; the xfail rows now name the open follow-up ticket recorded in tests/e2e-cucumber/expectations.toml)

Strix Halo Ubuntu managed serves can return success without exposing the
model, with logs showing argument rejection, backend-ready/router-unavailable,
or download failure. The existing xfails attribute all three to an immediate
shutdown.

Capture the runtime version, configured pins, candidate backend paths,
SHA-256 fingerprints and version/help output alongside the archived service
logs before cleanup. Probe duration and retained output are bounded, hashing
is capped and runs off the async executor, and timed-out children are killed
and reaped. The engine is read from both `--engine X` and `--engine=X`, falls
back to the printed serve plan, and the failure report says so when no engine
can be determined. Probe capture and kill/reap results are printed only when
they fail.

Replace the shutdown-only xfail attribution with the observed failure
signatures and cover serve-40's shared serve precondition. Intermittent-failure
handling and platform scope are unchanged.

This is diagnostic coverage, not a confirmed backend fix: configured pins and
candidate binaries are evidence of what could have run, not proof of what did.

Refs #897

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd requested a balanced review from Copilot November 5, 2026 11:28
@jussielo-amd
jussielo-amd marked this pull request as ready for review November 5, 2026 11:28
@jussielo-amd
jussielo-amd requested a review from a team as a code owner November 5, 2026 11:28
@jussielo-amd
jussielo-amd requested a review from rominf November 5, 2026 11:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression test uses a non-Linux mock capability, so it skips the relevant xfail guards and needs Linux/no-GPU cases, including one with a reported gfx target, to protect MockServer-backed chat scenarios.

1 open issue
What changed in this PR

This PR improves E2E diagnostics for Lemonade managed-serve failures tracked in #897 without changing runtime pins or the CLI’s asynchronous launch behavior.

Changes:

  • Captures candidate backend identities, configured pins, and bounded version/help output before service cleanup.
  • Updates expected-failure attribution to cover observed failure signatures and serve-40’s shared precondition.
  • Adds diagnostic and expectation tests, plus documentation of evidence limits.
File Description
tests/​e2e-cucumber/​tests/​e2e/​serving_steps.rs Collects backend evidence before stopping failed services.
tests/​e2e-cucumber/​src/​serve_log.rs Adds engine detection and backend evidence to reports; bounds reads.
tests/​e2e-cucumber/​src/​lib.rs Exposes the diagnostics module.
tests/​e2e-cucumber/​src/​lemonade_diagnostics.rs Implements backend inventory, fingerprints, bounded probes, and tests.
tests/​e2e-cucumber/​src/​expectation.rs Tests updated failure attribution and platform scope.
tests/​e2e-cucumber/​features/​model_serving.feature Updates a comment’s xfail reference to #897.
tests/​e2e-cucumber/​expectations.toml Revises failure descriptions and adds serve-40 coverage.
tests/​e2e-cucumber/​Cargo.toml Adds hashing and asynchronous process/I/O dependencies.
docs/​testing.md Documents collected evidence, limitations, and focused tests.
Cargo.lock Records the E2E crate’s hashing dependency.

Review effort: Balanced

Comment on lines +852 to +858
for platform in ["mock", "strix-windows"] {
assert_eq!(
resolve(&scenario, &cap(platform), &matrix, Included::default()),
Expectation::ExpectPass,
"{id} on {platform}",
);
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 2649611: a new test covers Linux hosts with no GPU, with a gfx target only, and with a GPU only, and still expects the xfail for the full GPU host. Removing has_amd_gpu from bench-load-real-serve turns only the new test red, and so does removing therock_family from chat-end-to-end-local-model; the old test stays green in both cases, which confirms your point.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the whole diff. I found no correctness problems in the new diagnostics; the hash worker's own deadline inside spawn_blocking, the kill-and-reap on the probes, and the boundary tests around the size cap all look right. Two small things are left as inline comments, and I'm requesting changes so they get a look before merge.

Questions for the author:

  • serve-40 is now xfail+flaky on lemonade/linux. The comment in the PR admits its assertion carries no signal on that lane, so a real background-helper regression there stays hidden until #897 closes. Is that the intended trade-off?
  • The llama-server --version/--help probes run while the failed service is still alive. On a shared-memory APU, could a probe that initialises the GPU backend contend with the still-running router? I couldn't verify this either way.
  • A llama-server binary larger than the 32 MiB hash cap gets no fingerprint. That is probably fine for current builds, but I haven't confirmed it.

The branch is also CONFLICTING with main (Cargo.lock, tests/e2e-cucumber/Cargo.toml, tests/e2e-cucumber/src/lib.rs). It needs a rebase, so the green CI so far is on the pre-rebase head.

let value = if let Some(inline) = arg.strip_prefix("--engine=") {
Some(inline)
} else if arg == "--engine" {
args.next()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--engine followed by another flag (e.g. serve m --engine --managed) returns --managed as the engine here. The report then neither captures backend evidence nor prints the "engine unknown" line, which breaks the promise that a skipped inventory is visible. Treating a value that starts with -- as absent would fix it; a test case for it would be good too.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f475592: a value starting with -- after --engine (and --engine=--managed) now counts as no engine named, so the report prints the "engine unknown" line. Both forms are in the test table, and with the guard removed the test fails with left: Some("--managed").

# the model appearing on :11435 (41984503055), and backend download failure
# (41986772728). A backend ready line does not prove a later shutdown.
# EAI-7423 described only the earlier ready-then-unavailable symptom.
# Retain flaky=true because historical runs also passed intermittently; do not

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The header this replaces also said that flaky is per-row and not derived from bug, so a new row has to set it explicitly. That rule is gone now. Worth keeping that sentence for the next person adding a row.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored in f237d2d: the header says again that flaky is per-row and not derived from bug, so a new #897 row has to set flaky = true explicitly.

…reports

A serve invocation such as 'serve m --engine --managed' made the engine lookup return '--managed'. The failure report then skipped the backend inventory without printing the 'engine unknown' line that makes a skipped inventory visible. A value that starts with '--' is another flag, not an engine, so it now counts as absent, for both the separate and the '--engine=' forms.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The header comment replaced in this PR also said that flaky is set per row and is not derived from the bug reference, so a new row for the same bug has to set it explicitly. Restore that sentence next to the rule that keeps flaky=true for the #897 rows.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…tion rows

The existing regression test resolves the real-GPU rows against the mock capability, whose os_family is 'other', so the os=linux condition excluded it before the therock_family and has_amd_gpu guards were consulted. Removing either guard left the test green even though Linux MockServer-backed bench and chat scenarios would then be expected to fail. Add Linux hosts with no GPU, with a gfx target but no usable GPU, and with a GPU but no gfx target, and assert that a full real-GPU host is still expected to fail.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…gerprint

The size-cap tests only covered the fingerprint helper. Cover the report itself: a llama-server larger than the hash cap is still listed and probed, and the report prints the reason it has no sha256 line, so the missing fingerprint is visible in the failure output.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Cargo.lock conflicted in the e2e-cucumber dependency list: this branch adds sha2 and main adds shlex. Both are kept, in sorted order, and 'cargo metadata --locked' accepts the result. tests/e2e-cucumber/Cargo.toml and tests/e2e-cucumber/src/lib.rs merged automatically with both sides' additions.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…obe-timing behaviour

Document that a value starting with -- after --engine counts as no engine named, that a llama-server over the hash cap is listed and probed with an explicit 'sha256 unavailable' reason, and that the version/help probes run before the failed service is stopped, with the GPU contention question marked as unchecked.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd

Copy link
Copy Markdown
Collaborator Author

Pushed 5f37981 with fixes for the inline notes (see the replies on each) and a merge of main instead of a rebase, so your review context stays valid. The only conflict was Cargo.lock in the e2e-cucumber dependency list: I kept both sha2 from this branch and shlex from main, in sorted order, and cargo metadata --locked accepts it; tests/e2e-cucumber/Cargo.toml and src/lib.rs merged cleanly. The PR body is updated, including dropping the stale "squashed to one commit" claim.

On your three questions:

  1. serve-40 is xfail and flaky on lemonade/linux on purpose: until E2E: lemonade serving fails on the Strix Halo Ubuntu lanes, with a different log signature depending on the run #897 closes the assertion carries no signal on that lane, and the row is to be removed when it does (the expectations.toml comment says so, and the PR body now does too). What covers the guard elsewhere, verified in the code: the unit test background_helper_not_spawned_when_no_automation_is_enabled in service_launch.rs covers the gate function and scenario automations-11 covers the helper's lifecycle on the mock lane. The gap is that neither covers the serve call site itself.
  2. Probe contention: every call site in serving_steps.rs runs the evidence collection only after a non-zero exit or a timed-out readiness poll, and only after the log has been read and archived, so the failing outcome is already decided. The router can still be alive during the probes, which run one at a time (5 s each, 20 s total) in the harness environment, not the engine's. What I cannot verify here (no GPU, no llama-server) is whether --version or --help initialises a GPU backend, so that part is unchecked, and docs/testing.md now says so.
  3. A binary over the 32 MiB cap is still listed and probed, and the report prints sha256 unavailable: executable exceeds 32 MiB hash limit; no partial hash is ever labelled SHA-256. 25e11c5 adds a report-level test for it, which goes red when that line is suppressed.

cargo test -p e2e-cucumber passes (226 lib tests plus the extra targets), and clippy, fmt and cargo xtask manifest --check are clean. Could you take another look?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new llama-server probes run before a potentially live router is stopped, and their GPU initialization and contention risks remain unvalidated on Strix Halo hardware.

1 open issue

Review effort: Balanced

The oversized-executable test asserted that the report contained the display form of a path it built with join("bin/llamacpp/rocm-stable"), while the inventory joins "bin/llamacpp" first and the backend directory second. On Windows the two spell the same path with different separators, so the substring check failed even though the report was right. Parse the candidate lines back into paths and compare those; Path equality is component-wise, so both spellings agree.

The same helper replaces a negative check in the candidate-cap test that could never fail on Windows: asserting the report did not contain the display form of the skipped candidate passed vacuously there. It now asserts the listed candidates exactly (count, the skipped one absent, the first one present).

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd

Copy link
Copy Markdown
Collaborator Author

The Windows lane found a bug in a test I added in the last push, and c05b62c fixes it. oversized_executable_reports_why_it_has_no_fingerprint built its expected candidate path with / separators and compared printed text, while the report on Windows prints the same path with mixed separators (...\bin/llamacpp\rocm-stable\llama-server.exe); the report itself was correct. The test now parses the candidate lines back into paths and compares them component-wise. Auditing the same class in this PR found one more instance: evidence_truncates_candidates_without_hashing_or_probing_the_rest asserted that the skipped candidate's printed path was absent, which could never match on Windows and so passed vacuously. It now asserts the exact set of listed candidates (the count, the absent skipped one and the present first one), and a mutation that lists the wrong eight turns it red. The other new tests use no paths. One gap remains and is not new: five tests in lemonade_diagnostics.rs are unix-only because they use shell-script stand-ins, so the probe timeout and kill path does not run on the Windows lane. I could not run a Windows check locally (the C toolchain for the cross build is missing), so the Windows lane on this push is the confirmation.

tests/e2e-cucumber/expectations.toml conflicted in two places. Main had removed the runtime-path-not-nested row, so it stays removed. Main also added its own serve-stop-leaves-no-background-helper row (lemonade, linux, #897, serve_timeout_secs = 90, flaky = true) with its own explanation; this branch had added an identical-condition row for the same scenario. The two are reconciled to the single row from main, whose fields match what this branch set, so the scenario is not listed twice. The header sentence that flaky is set per row and is not derived from the bug reference is kept, and the bench-load-real-serve row keeps its #897 attribution.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd

Copy link
Copy Markdown
Collaborator Author

Merged main into the branch again (5463b83, a plain merge commit) because main moved, including #900, which adds its own serve-stop-leaves-no-background-helper row to expectations.toml. The only conflict was in that file: the merge left two identical rows for that scenario (same lemonade/linux condition, #897, 90 s timeout, flaky = true), so I kept main's row with its fuller comment (it already says that while the row is in place the scenario's own helper check is not exercised, and that the row has to be deleted by hand when #897 closes) and dropped this branch's copy, so the scenario is listed once. The sentence saying flaky is per-row and not derived from bug is kept in the header, runtime-path-not-nested, which main removed, stays removed, and the regression test for the mock scope still passes against the merged row. cargo test -p e2e-cucumber --lib passes (238), clippy and fmt are clean, and the PR body is updated to match.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall: no blocking problems found; the scope is honest (diagnostics for #897, not a fix) and CI is green. Requesting changes on the points below.

Should address

  • The llama-server --version/--help probes run before the service is stopped and may initialise a GPU backend alongside the live router, perturbing the state being diagnosed. Move the probes after the stop, or confirm on the lane that they are harmless and replace the "has not been checked" note in docs/testing.md.
  • No test ties the new call site to the documented behaviour: the runtime.json manifest path matches what the CLI writes, and evidence is taken before the stop. A wrong path degrades silently to unavailable: on the real lane.

Smaller points

  • backend_candidates searches only one level under four fixed backend dirs, and its empty-result message hints at a download problem.
  • 32 MiB hash cap may be below real ROCm server binaries.
  • The "revisit before deleting" caveat was dropped from the expectations.toml header; it is still true until the new evidence lands.
  • Please confirm a repo-wide grep EAI-7423 is clean.
  • The Windows timeout/kill path of the probe has no test and the PR text does not say so.
  • The strix lane that would exercise the new report was not among the checks; please note which run confirms the report is populated.

&crate::results_path(),
world.current_scenario.as_deref().unwrap_or("scenario"),
);
let backend_evidence = match engine {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Evidence, including the llama-server --version/--help probes, is collected here, before stop_scenario_services below. llama.cpp ROCm/Vulkan builds typically initialise the backend on --version, so the probes may touch the GPU next to the still-running router and perturb the state being diagnosed. Could the probes (not the file inventory/hashes) run after the stop, or has the lane shown they are harmless? docs/testing.md currently says this is unchecked.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in dfb844f. The probes now run after the stop: the manifest, pins, candidate paths, fingerprints and version.txt are still read here, before the stop, and the --version and --help probes execute only after stop_scenario_services returns, with their own 20 second budget. The order lives in one function, backend_evidence_around_stop, so this call site cannot reorder it, and docs/testing.md now describes what the code does instead of saying it was unchecked. I could not show on a lane whether a probe initialises a GPU backend, so I kept them off the router's side of the stop rather than call them harmless. At the previous head the Strix Ubuntu lanes ran the probes before the stop and both exited 0, which says nothing about contention either way.

Some("lemonade") => Some(
e2e_cucumber::lemonade_diagnostics::backend_evidence(
&world
.app_paths()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing asserts that this is the file the CLI actually writes, nor that this call precedes the stop. A wrong filename would only show up as unavailable: ... on the real lane, which defeats the feature. Consider sharing the filename constant with the producer or adding a test for the path.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05bcc1c and dfb844f. The filename is now one definition, AppPaths::engine_runtime_manifest_path, which the Lemonade installer's manifest_path and this call site both use, and an engine test writes a manifest and asserts it lands at that path with the field names the report reads (it goes red when the engine builds a different path). For the order, backend_evidence_around_stop takes the stop as a closure, and its test makes the stop delete the candidate: the fingerprint in the report shows the inventory came before the stop and a recording probe shows the probes came after it. Moving either step to the other side of the stop turns that test red. On the lane, the manifest was read correctly at the previous head: the Strix Ubuntu nightly and release jobs both printed the manifest, pins and a fingerprinted llama-server.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my earlier reply: the shared helper is AppPaths::engine_install_manifest_path, not engine_runtime_manifest_path. It was renamed in d4ca134 because the registry writer guard on main (a text match for runtime_manifest_path( in any file that also writes a file) matched the old name inside the Lemonade installer, and the new name also says what it is, the engine's own install manifest and not a runtime registry entry. The path it returns and the engine and harness call sites that share it are unchanged.

report
}

fn backend_candidates(runtime: &Path) -> Vec<PathBuf> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

backend_candidates only looks at depth 0 and 1 under four hard-coded backend dirs, and when nothing is found the report hints at an incomplete download. Consider wording the empty case neutrally and listing the searched locations.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 59d7891 (with a test-order correction in f5bff5f). The search now goes three directory levels below each of the four backend directories, bounded by an entry cap per directory, and the cpu backend is still left out on purpose. An empty result no longer hints at a download: it lists every location searched, the depth, and what bin/llamacpp actually contains (or that it cannot be read), because an interrupted download, an unsearched layout and a build that was never installed look the same. A test with literal layouts shows a build three levels down is found and one four levels down is not, and it goes red when the depth is lowered. For reference, on the Strix Ubuntu lanes the b12109 build sits one level below rocm-stable, so the old depth already found it there.

candidates
}

fn fingerprint(server: &Path, deadline: Instant) -> std::io::Result<String> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

32 MiB hash cap: please confirm real ROCm llama-server builds fit under it, otherwise the fingerprint is absent on exactly the lane this is for.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked, and changed anyway as margin, in b064945 and ee08e62. The Strix Halo Ubuntu nightly and release jobs at the previous head both fingerprinted the rocm-stable b12109 llama-server under the old 32 MiB cap, so a current ROCm build fits. A statically linked multi-target build could be larger, so the cap is now 512 MiB; hashing still streams 64 KiB at a time under the inventory time budget, so memory and time stay bounded, and the over-cap line is worded from the constant. The remaining assumption, about builds larger than the observed one, is stated in docs/testing.md and the PR body.

…ine writes it to

The serve failure report reads the installed runtime's manifest to find the
backend directory, and built the path with its own "runtime.json" literal next
to the engine's. A rename on either side would have degraded the report to
"unavailable:" on the one lane it exists for, with nothing failing before then.

AppPaths::engine_runtime_manifest_path is now the one definition. The engine's
manifest_path and the harness both call it, and an engine test writes a manifest
and asserts it lands at that path with the field names the report reads.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…pped

The version and help probes executed a backend binary while the failed serve's
router could still be running. llama.cpp builds commonly initialise a GPU backend
even for --version, so on a shared-memory device a probe could contend with the
router and change the state the report exists to describe. docs/testing.md said
this had not been checked.

The report now reads files first and executes second: manifest, pins, candidate
paths, fingerprints and version.txt are taken before the stop, and the probes run
after it, with their own 20 second budget so a slow stop cannot use it up. The
order lives in one function, backend_evidence_around_stop, which also runs the
stop exactly once for every engine, so the serve step cannot reorder or skip it.
Its tests make the stop delete the candidate, which shows the fingerprint came
before the stop and, through a recording probe, that the probes came after it.
docs/testing.md and the expectations header now state what the code does.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…empty search looked

The inventory looked only at depth 0 and 1 under four backend directories, while
the router chooses how a llama.cpp build is unpacked, so a build one level
deeper would have produced no candidates and no probe. When nothing was found,
the report also suggested the backend download may not have completed, which an
unsearched layout or a never-installed build would equally explain.

The search now goes three directory levels below each backend directory, bounded
by an entry cap per directory. An empty result names every location searched, the
depth, that the cpu backend is deliberately left out, and what bin/llamacpp
actually contains, without naming a cause. docs/testing.md states the search.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…sorts them

The previous commit's depth test listed its two expected candidates in the wrong
order: the search sorts sibling directories by path, so a/b comes before a/build.
The test therefore failed even with the depth limit in place. With the order
corrected it passes, and with the limit lowered to two it fails because the
three-level candidate is no longer found.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…r guard does not match it

The registry guard in runtime_registry.rs reads each production file as text and
flags one that both writes a file and spells runtime_manifest_path(. The shared
helper added for the Lemonade install manifest was called
engine_runtime_manifest_path, which contains that spelling, and the Lemonade
installer both calls it and writes its manifest, so the guard reported a second
registry writer. It only showed once the guard reached main.

The helper is now AppPaths::engine_install_manifest_path, which also says what it
is: the engine's own install manifest, not an entry in the runtime registry. The
doc comment records why the names must stay apart. The path it returns is
unchanged, and so is the guard.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The diagnostics report can read and embed an outside file through a symlinked version.txt despite reporting the link as skipped, so the marker needs a non-symlink regular-file check and regression coverage.

2 open issues

Review effort: Balanced

… directories

The directory walk counted a symlinked version.txt as a link it did not follow,
but the report then read version.txt through read_clamped, which follows it, so a
real llama-server beside a marker linking outside the backend directory put the
outside file's content into the report while the report said the link was skipped.

Every file the report reads from the tree now goes through the same rule: the
manifest, the pins file, version.txt and a candidate's fingerprint target are
refused when the last path component is a symlink, and the report says "symlink
not followed" in place of the content. The symlink regression fixture gains the
real server with a version marker, pins file and manifest that link outside the
tree; the test asserts the outside content is absent and each refusal is named.
Letting the link be followed again turns it red.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because FIFO reads can hang inventory and prevent service cleanup, while directory-read failures can misleadingly report that no candidates were found.

1 open issue
1 resolved since last review

Review effort: Balanced

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking issues found. Approving; the points below are non-blocking.

  • A Lemonade serve with no isolated root panics in serve_failure_evidence before the stop runs, losing the failure report (inline).
  • read_clamped now truncates a grown log silently for existing callers too (inline).
  • #897 is closed but is still cited by all eight xfail rows and the docs link (inline).
  • Small is_symlink-then-open race against the docs' "symlinks refused" claim (inline).
  • Cosmetic: "truncated ... more may exist" prints when there are exactly 8 candidates, and "remaining evidence skipped" can print twice when the search ends on the deadline.
  • The 5 s child-pid wait in the kill/reap test could flake on a loaded runner.
  • The PR-body checklist boxes are still unchecked.

The stop-exactly-once and read-before/probe-after ordering is well covered by tests that assert behavior rather than wording, and CI is green across lanes.

e2e_cucumber::lemonade_diagnostics::backend_evidence_around_stop(
move || {
app_paths
.expect("a Lemonade serve runs in an isolated scenario root")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This expect runs before the stop, so a Lemonade serve without an isolated root panics here, never stops the service, and drops the failure report. The previous code reported "not attempted: scenario has no isolated root". Could this degrade to an "unavailable" evidence string and still run the stop?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 96f67f0. The expect ran in the manifest lookup before the stop, so a Lemonade serve without an isolated root panicked and the stop and the whole failure report were lost. The lookup now returns Err(reason) when there is nothing to look for, the report says backend evidence unavailable: the scenario has no isolated root, the stop's own status still says it was not attempted, and the stop runs exactly once with the read-before and probe-after order untouched. A panic inside the lookup is caught and reported the same way, and a_failing_manifest_lookup_still_runs_the_stop_once_and_reports_why covers both the error and the panic; removing the panic guard turns it red. For the class sweep I checked the rest of the pre-stop path: the only other expect is on a child pipe that is configured to be piped and runs after the stop, and the scan and fingerprints run in spawn_blocking with their join errors handled.

Comment thread tests/e2e-cucumber/src/serve_log.rs Outdated
if len <= max {
let mut body = Vec::new();
file.read_to_end(&mut body)?;
file.take(max).read_to_end(&mut body)?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

take(max) now silently truncates a log that grew after metadata(), for existing callers too, with no truncation note (the note only exists on the len > max branch). Worth adding a note or a comment on why this is acceptable.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 2c5a106. read_clamped now asks for one byte past the cap on the fits branch, so a log that grew after its size was read is clamped to its head and tail like an oversized one and the note says the file grew while it was read; a file that really fits is still returned verbatim with no note. All callers consume the note: the archived-log line, the pins and version.txt in the Lemonade report, and the manifest read, which discards it but fails to parse a cut JSON document and says so. a_log_that_grew_after_its_size_was_read_is_clamped_and_says_so hands the inner function a stale size, and not requesting the extra byte turns it red. The branch for a file that shrank during the read is not tested, because it needs the size to change between two reads inside one call.

Comment thread tests/e2e-cucumber/expectations.toml Outdated
when = { effective_engine = "lemonade", os = "linux" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the endpoint never becomes reachable."
bug = "#897"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#897 is closed (2026-11-05, by the author), but this row, its siblings and the docs link cite it as the live tracker. Reopen it or repoint to the open ticket.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2840f96, and #897 stays closed. Every bug field, the comments around the rows, the harness tests that pin them and the docs now name the open ticket ROCMAI-898, with one header note that these failures were first tracked as #897, since closed. The stale-xfail hint reads the ids out of the matrix and hard-codes none, so it follows the new id; I ran it on the real file (a body mentioning ROCMAI-898 lists all eight rows, one mentioning #897 lists none) and added that case to its self-test. The rows' conditions, timeouts and flaky = true are unchanged, and the PR body now refers to the open ticket without naming its key, so merging this PR cannot close it through the key-in-body automation.

if is_symlink(server) {
return Err(std::io::Error::other(SYMLINK_NOT_FOLLOWED));
}
let mut file = std::fs::File::open(server)?;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The is_symlink check followed by File::open leaves a window where the path can become a link, while the docs say links are refused. Consider a no-follow open on unix, or soften the claim.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in 4293a30. On unix every read of a file in the backend tree (manifest, pins, version.txt, fingerprint target) now goes through one helper that opens with O_NOFOLLOW through the libc the crate already depends on, so there is no window between a check and the open and the refusal message is unchanged; the_open_itself_refuses_a_symlink calls the helper directly on a link, which is what a swap after the check looks like, and removing the flag turns it red. On Windows the behaviour is unchanged (a metadata check, then an open), so a swap between the two is not caught there; docs/testing.md now says so, and that only the last path component is checked.

Two further points from the review body, which fit this thread. The "truncated ... more may exist" note printed for exactly eight candidates; in 198e385 the search keeps eight and stops on finding a ninth, so the note appears only when something is not listed, and "remaining evidence skipped" is now printed once when the search ends on the clock (a test for each, red when the old behaviour returns). And the kill/reap test's fixed five-second wait for the child's pid could flake on a loaded runner; in 2e39826 an attempt whose child printed no pid is retried with a 20 then 60 second timeout, so the generous waits matter only on a slow start and a healthy run still takes five seconds (with a child that starts in seven seconds it passes on the second attempt, and with the retries removed it fails).

…fest lookup fails

The serve step looked the runtime manifest up with an expect that runs before the
service is stopped. For a Lemonade serve without an isolated scenario root it
panicked, so the stop never ran and the whole failure report was lost, where the
code before the evidence change reported "not attempted: scenario has no isolated
root".

The lookup now returns Err(reason) when there is nothing to look for, and a panic
inside it is caught and treated the same way. The report says "backend evidence
unavailable: <reason>", the stop still runs exactly once, and nothing is probed.
The only other expect on that path is on a child pipe that is configured to be
piped and runs after the stop. The test covers both the Err and the panic and goes
red when the panic guard is removed.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
read_clamped read a file whose size fit the cap with take(max), so a log that grew
after the size was taken was cut at the cap with no note, for the archived service
log and for the pins and version.txt in the Lemonade report alike. The note existed
only on the branch for a file that already exceeded the cap.

The read now asks for one byte past the cap. If it is there, the file grew, and it
is clamped the way an oversized file is, head and tail with the gap marked, and
the note says the file grew while it was read. A file that fits is still returned
verbatim with no note. The size is a parameter of an inner function so a test can
hand it a stale one; with the extra byte not requested the test is red. All
callers consume the note (the manifest read discards it, but a cut JSON manifest
fails to parse and the report says so).

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
GitHub issue #897 is closed, but the xfail rows, their header comments, the harness
tests that pin them and the docs still named it as the live tracker. The open
ticket is ROCMAI-898, so every row, comment, test and doc line that named #897 as
the tracker now names ROCMAI-898, with one note in the header that it was first
tracked as #897 and that issue is closed. The issue stays closed.

The stale-xfail hint reads the ticket ids out of the matrix and hard-codes none,
so it follows the new id on its own; its self-test now covers that (a PR that
mentions ROCMAI-898 is told about the rows, one that mentions #897 is not). The
rows' conditions, timeouts and flaky handling are unchanged.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
The report refused symlinked files by reading the path's metadata and then opening
it, which left a window in which the path could become a link while the docs said
links are refused. On unix the open now passes O_NOFOLLOW (through the libc the
crate already depends on, so no new dependency), so a link is refused whenever it
is there at the moment of the open, with the same "symlink not followed" message.
Manifest, pins, version.txt and fingerprint reads all go through that one helper.

On Windows the behaviour is unchanged: a metadata check followed by an open. A swap
between the two is not caught there, and docs/testing.md says so, and that only the
last path component is checked. A unix test calls the open helper directly on a
link, which is what a swap after the check looks like to it, and goes red when
O_NOFOLLOW is removed.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…, and say the budget is out once

Two cosmetic lies in the Lemonade report. The "truncated after 8 candidates" note
was printed when exactly eight candidates existed, because the search stopped on
reaching the cap and could not tell whether a ninth existed. And "remaining
evidence skipped: evidence budget exhausted" was printed twice when the search
ended on the clock and the candidate loop then found the budget gone too.

The search now keeps eight and stops on finding a ninth, so the note appears only
when there is something not listed, and exactly eight is a complete list. The
budget line is said once. Both have tests, and each turns red when the old
behaviour returns. docs/testing.md states the ninth-candidate rule.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
…ts slowly

The kill and reap test lets a helper child run for a fixed 5 seconds and reads its
pid from the probe's report. On a loaded runner a cold test binary can take longer
than that to start and print the pid, which failed the test for a slow start and
not for anything the probe did.

An attempt whose child printed no pid before the timeout is now retried with a
longer timeout (20 seconds, then 60), and only an attempt that did report a pid is
asserted on, so the generous waits matter only when the start is slow and a
healthy run still takes 5 seconds. With a child that takes 7 seconds to start the
test passes on the second attempt, and with the retries removed it fails.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The helper exits after 30 seconds despite a 60-second final-attempt timeout, which can falsely fail the timeout/kill/reap assertion; it must stay alive until killed.

2 open issues

Review effort: Balanced

The kill and reap test retries with timeouts of 5, 20 and 60 seconds, but the
helper child exited by itself after 30. If the first two attempts timed out before
the child printed its pid and the last one started it, the child exited normally
before the 60 second timeout, so the probe saw a normal exit, not a timeout, and
the test failed for a correct probe. Reproduced with a child that takes 25 seconds
to start: the third attempt reports "exit status: 0".

The helper's lifetime is now derived from the attempt timeouts (the longest plus a
60 second margin), so they cannot drift apart, and a test pins that the child
outlives every attempt by at least 30 seconds. With the same 25 second start the
test now passes on the third attempt; with a literal lifetime or no margin the
pinning test fails. A child that takes 7 seconds to start still passes on the
second attempt and fails with the retries removed. The other helper lifetimes in
the file (30 second sleeps against 1 second timeouts, an immediate exit against a
60 second timeout) have no such pair.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed for a critical filesystem edge case that can hang diagnostics indefinitely and prevent service cleanup.

1 open issue
1 resolved since last review

Review effort: Balanced

@jussielo-amd
jussielo-amd added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit a3b4014 Nov 9, 2026
32 checks passed
@jussielo-amd
jussielo-amd deleted the e2e/lemonade-serve-failure-evidence branch November 9, 2026 12:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants