Repository navigation
Conversation
Add one behavioural scenario per defect, each asserting the CORRECT behaviour and registered as an xfail in expectations.toml citing its EAI id, so the mock lane stays green now and the row goes stale (XPASS) the day the bug is fixed. No product behaviour changed. - EAI-8046: `rocm chat` ignores a piped prompt (reads no stdin, sends no request, exits 0 with a status summary). New run_rocm_with_stdin helper since Command::output gives the child a null stdin; the load-bearing assertion is that the mock received the prompt, not stdout. - EAI-8047: automations list publishes no usable identifier, so the checks it lists cannot be enabled from what it shows. The scenario derives each id from the listing rather than hard-coding the real ids. - EAI-8048: `dash --replay <missing>` opens the interactive view instead of refusing; the step imposes its own deadline so the known hang xfails promptly rather than burning the CI budget. - EAI-8049: the engine shell's activation hint names a path that does not exist, because the recorded env_path already includes the interpreter directory. Phrased as path-exists so the row goes stale on any fix. All four re-confirmed in a plain Linux container against this branch's binary before writing the scenarios. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/expectations.toml # tests/e2e-cucumber/tests/e2e.rs
Address pre-PR review: each of the three step assertions pinned the bug's present symptom rather than the behaviour that must hold, so a valid fix could stay xfailed and the row would never go stale. - EAI-8047: parse the identifier the listing explicitly exposes (inline [id]/(id) on the header, or an `id:`/`identifier:` detail line), keeping the display-name slug only as the current-broken-output fallback. A fix that publishes the id now flips the row to XPASS. - EAI-8048: also assert the interactive view never opened, not just a prompt non-zero exit. Adds a latched entered_alternate_screen() to the PTY driver so an "open the TUI, then exit non-zero" impl no longer satisfies the contract. - EAI-8046: assert the user-role message content rather than messages[0], so a fix that prepends a system message still XPASSes. Mock lane unchanged: still 6 xfail / 0 XPASS; each scenario still fails on its assertion step alone. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Round-2 review: the latch checked the parser's alternate_screen() state only after processing a whole PTY read, so a chunk carrying both the enter (?1049h) and leave (?1049l) sequences left it false; and wait_for_any_exit returned before the reader drained, so a still-buffered enter could be missed. Either produced a false negative that would let a fixed dashboard open the TUI and still pass the "must not open the interactive view" half. - Detect the ?1049h enter sequence in the RAW byte stream, with a one-short-of-marker carry-over so a boundary-split sequence is still found. Independent of what follows it in the same read. - Drain the reader (bounded by DRAIN_TIMEOUT) after the child exits, before the caller reads the latch — same post-exit drain wait_for_screen uses. Mock lane unchanged: 6 xfail / 0 XPASS. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
450d4c8 to
dd733fe
Compare
The CI Clippy (e2e harness + steps) job runs clippy -p e2e-cucumber --test e2e -D warnings, which the local container gate did not (its clippy step excludes e2e-cucumber). Four lints surfaced: - future-not-send: drain_reader took &self, and a shared &TuiSession held across .await is not Send (the struct holds !Sync PTY handles), poisoning every step awaiting it. Take &mut self like the sibling waits. - collapsible-if, or-fun-call (automations_steps) and map-unwrap-or (engines_steps): mechanical rewrites, no behaviour change. Verified with the exact CI commands in a Linux container: both clippy invocations clean, e2e mock lane unchanged (6 xfail / 0 XPASS). Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/features/automations.feature # tests/e2e-cucumber/tests/e2e/automations_steps.rs
Upstream (EAI-8072) independently added a run_rocm_with_stdin helper that also takes an env slice, and the merge left both definitions in e2e.rs (E0428), with my chat step calling the 3-arg form (E0061). Keep upstream's more general helper and delete mine; the chat stdin step now passes an empty env slice. Behaviour is identical - both pipe stdin and close the handle before waiting, so a command reading to EOF still terminates. Verified with the exact CI commands in a Linux container: both clippy invocations clean, e2e mock lane 6 xfail / 0 XPASS. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
michaelroy-amd
left a comment
There was a problem hiding this comment.
Reviewed at 1452ee174a87cf05693284b32a8b16fce0cc094a. The four scenarios pin the intended user-visible contracts rather than implementation details: piped chat reaches the model, listed automation identifiers are usable, a missing replay file fails without opening the TUI, and the engine-shell activation path exists. Each expectation is scoped to its ticket and removal condition; helper changes have bounded PTY drains and preserve command environment isolation. All 17 required checks pass, including the mock E2E gate and DCO/signoff. No blocking finding. Approving.
|
I read through this PR's diff (scenarios, step definitions, expectations.toml, and the tui_driver additions) and traced each of the four claimed defects against the actual source:
I also checked the One thing worth flagging, though it's not a code issue: GitHub currently reports this PR as having a merge conflict. That's because Nothing else stood out. This looks safe to approve after a rebase and once CI (mock e2e lane, clippy) is green — this is a diff-only read, not a substitute for CI or a maintainer's own pass. |
|
🔴 Automated review · pr-review-watcher · 1452ee1 SummaryTest-only PR pinning four README-walkthrough defects as cucumber scenarios that assert the correct behaviour, plus four 🚫 Blocking (must fix before merge)None. Non-blocking
|
…-for-the-second Absorbs 38 commits of main and reconciles the four pinned scenarios with the scenario-numbering convention main adopted repo-wide in the meantime: - dash.feature: main's launcher scenario took the dash-10 slot, so the replay rejection scenario becomes dash-11 (both kept). - chat.feature / automations.feature / engine_shell.feature: renumbered onto the <feature>-NN form (chat-09, automations-04, engine-shell-02). main independently added its own automations.feature and automations_steps.rs; the listing-discoverability scenario folds in alongside them, with no duplicate step phrases and no dead expectation rows. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Addresses the non-blocking review points on #292. Each one was a way a CORRECT fix could leave the expectation row xfailed instead of going stale, which is the failure mode these scenarios exist to avoid. - chat: compare the user message trimmed. The step pipes a trailing newline, so forwarding stdin verbatim -- a perfectly good fix -- would otherwise not XPASS. - automations: recognise the identifier in the shapes a fix might publish it -- a detail line (id/identifier/slug/key/name), bracketed, parenthesised, dash-separated, or as a trailing column. A parenthesised or dash/column token must look like an identifier (lowercase with a separator, as every real watcher id is), so a display-name qualifier such as "Server recovery (beta)" is not mistaken for one. The slug-of-display-name fallback is unchanged, so today's listing still fails the contract. - engine shell: accept ". <path>" and a quoted path as well as "source <path>", so the existence check fails only when the CLI really named a missing file. - dash: replace the bare 2s replay deadline with a documented REPLAY_REJECTION_TIMEOUT. Still bounded (today's behaviour is a hang, and that bound is paid every run until it is fixed), but with enough headroom that a cold start on a loaded runner will not flake once the row is deleted. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
Thanks both — all five non-blocking points are addressed in Rebase / 38 commits behind (both reviewers): merged The four durability notes — each was a way a correct fix could leave the row xfailed instead of going stale, which is precisely what these scenarios exist to prevent, so all four are fixed:
Verification (Linux container, native-Linux host matching CI): clippy Two failures in that run are pre-existing environment gaps, not regressions, and I verified each by running the identical gate against a pristine |
|
🔴 Automated review · pr-review-watcher · 3490bf6 SummaryTest-only PR adding four cucumber scenarios (plus supporting steps and a PTY-driver capability) that pin correct behaviour for four open CLI defects as documented 🚫 Blocking (must fix before merge)None. Discrimination check (each row fails only against a binary lacking the behaviour)
On the 2s → 30s question: keep the dedicated bound, do not alignKeep
Non-blocking
|
…(PR review) Second round of pr-review-watcher notes, all about a row staying xfailed or failing for the wrong reason after a valid fix: - chat-09 now waits for the request that carries the piped prompt rather than for whichever chat request lands first, so a future readiness probe on this path cannot satisfy the wait. Its bound is named and the reason for keeping it short is recorded at the constant. - automations-04 sweeps the unambiguous identifier keys (id/identifier/ slug) across the whole block before the ambiguous ones (key/name), which must additionally look like an identifier, so a listing that prints a display name above its id no longer yields the name. - The engine-shell expectation reason now states that the fixture pins the manifest shape an install already recorded, so the row clears on the hint composition rather than on what a fresh install writes. Verified in the Linux container: 6 xfail, 0 XPASS, 0 stale. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
🔴 Automated review · pr-review-watcher · 2c4b600 SummaryTest-only PR adding four e2e scenarios ( 🚫 Blocking (must fix before merge)None. Non-blocking
|
siloteemu
left a comment
There was a problem hiding this comment.
Reviewed at 2c4b6002: no blocking issues, third consecutive clean pass.
Verified this round: merge and renumber integrity after absorbing the base's repo-wide scenario renumbering — every expectation row names a scenario that exists, all scenario tags unique, per-file indexes sequential, no ambiguous or duplicate step definitions, and every step resolves. Staleness checked directly: all four pinned defects were traced in the current base and each is still live for its stated reason, so none of the rows has quietly become an unexpected pass.
Each scenario discriminates in both directions — it fails today for the reason stated, and would go stale on a correct fix rather than silently continuing to pass. Three minor notes in the review comment; none needs action before merge.
…-for-the-second Resolves a semantic conflict in tui_driver.rs: main (#354) and this branch independently added the same capability under different names — a poll-until- exit that returns the raw code instead of asserting zero. Converged on main's wait_for_exit_code, since it is already published and has a caller there (diagnose-16), and folded in this branch's bounded drain_reader() wait, which the dash-11 alternate-screen latch needs to see the child's final bytes. dash-11 now calls wait_for_exit_code. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…ine's interpreter The doubled env_path also sets a wrong VIRTUAL_ENV and a nonexistent bin/bin on PATH, so a fix to the hint text alone would flip this row while the shell still runs the wrong python. Reuse the existing interpreter step so the row clears only once the user's actual problem is fixed. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
Took your recommendation: pushed Native-Linux container on the merged tree: 4 xfail, 0 XPASS, 0 stale, 0 unexpected; |
|
Pushed 485d697: engine-shell-02 now also asserts the engine interpreter runs, as you recommended; main merged, Linux gate 4 xfail / 0 XPASS / 0 stale. |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 485d697
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Round outcome: this round files a change request, for one objection only: the first item under Blocking. The engine-shell-02 interpreter step that 485d6974 added to stop a hint-only fix from flipping the row cannot fail. A mutation shows a hint-only fix still flips it. The head commit's message, which lands on main in the squash, says the opposite. The other two Blocking items are published as found, but neither gates. The internal-environment comment was already on the previous head, and the name is already public in this repo's workflows. The description mismatch is description-only. Items are marked new (first raised this round) or standing (raised in an earlier round).
The failing "E2E tests (MI300X, current)" check does not bear on this. Main's latest MI300X runs fail on the same scenarios (chat-06, serve-01, serve-02, serve-07 with a Lemonade 404, runtime-02, therock-next-09). This head adds only its own two expected xfails, automations-06 and engine-shell-02. W7900 has not run yet.
Review — needs work
Full review of the whole change.
Implements the regression pins for EAI-8047 and EAI-8049, both still open, and fixes neither defect. This matches the description's "test-only" framing.
Blocking
-
new — engine-shell-02's interpreter step can never fail, so a hint-only fix flips the row while the shell still runs the wrong python —
tests/e2e-cucumber/features/engine_shell.feature:39-42,tests/e2e-cucumber/tests/e2e/engines_steps.rs:144-161
The feature comment, and the head commit, say this step is there so that "a fix to the hint text alone" cannot flip the row. It cannot do that job.assert_engine_interpretersendscommand -v pythonand then waits for<env>/bin/pythonanywhere on screen.engine_shellprints that same string in its banner (python: {python_executable},apps/rocm/src/engines_cmd.rs) before bash starts. The wait therefore returns at once, whatevercommand -vprints.
Confirmed by mutation. I changed only the hint (hint_root = env_path.parent()whenenv_pathends inbin) and leftPATH=<env>/bin/bin:…andVIRTUAL_ENVwrong. Withcargo xtask e2e -- -i features/engine_shell.feature, both scenarios passed and the run reported1 XPASS (… 1 stale)forengine-shell-activation-hint-is-usable.
The step was already this weak in engine-shell-01. What this change adds is a scenario whose stated contract depends on the step. That fix is also the one the EAI-8049 ticket proposes ("build the hint from the environment root"), so it is the likely path.
Confidence 100 · logic · Fix: wait for output that only the command can produce. For example, runpythonand wait forplanted-engine-python(the planted interpreter's output, which the banner never prints), or echo a sentinel such aspy=$(command -v python)and wait forpy=<expected>. -
new (does not gate) — A comment names an internal cluster environment —
tests/e2e-cucumber/expectations.toml:185
The comment says "walking the CLI on an MI300X app-dev pod". AGENTS.md §2 forbids cluster names in code comments on upstream surfaces. The name already appears in a few workflow and docs files on the base branch. That reduces the extra exposure but does not exempt a new occurrence.
Confidence 70 · mechanical · Fix: "on an MI300X host". -
standing (description-only, does not gate) — The PR description misstates the change — PR description, "What it pins"
- It credits
automations-05with deriving identifiers from the listing. The new scenario isautomations-06.automations-05is the existing binary-units scenario. - It says the parser "recognises an id in any shape a fix might plausibly print it in".
exposed_identifier(automations_steps.rs:88) recognises onlyid:/identifier:/slug:detail lines and a trailing[id]on the header. Its own doc comment says "Any other shape a fix might pick leaves the row xfailed".
Later readers trust the description as the map of the change.
Confidence 85 · mechanical · Fix: narrow the description. Name automations-06 and state the two recognised shapes. - It credits
Non-blocking
-
standing — automations-06 stays xfailed under plausible fixes, so a fixed bug would go unnoticed —
tests/e2e-cucumber/tests/e2e/automations_steps.rs:88-145
Two plausible fixes leave the row xfailed instead of making it go stale:enableaccepts the display name exactly as listed. The step only ever sends the slug (server-recovery) or an exposed id, never the listed name.- The id is shown in parentheses beside the label, e.g.
Server recovery (server-recover) (off). The ticket suggests showing the id "alongside its label". The parser would then slug the whole string.
This fails safe: the run stays green. The cost is that the row never reports the fix.
Confidence 75 · logic · Fix: when no identifier is exposed, try the verbatim display name before the slug, and consider recognising a parenthesised id. -
new — "Lemonade reports the root" is inaccurate —
tests/e2e-cucumber/expectations.toml:208,tests/e2e-cucumber/tests/e2e/engines_steps.rs:41
Lemonade's install returnsenv_path: manifest.runtime_dir(engines/lemonade/src/lib.rs). That is the directory holdinglemond/lemonade, not a Python environment root withbin/activate. The vLLM-versus-Lemonade contrast in the comment makes the defect look narrower than it is.
Confidence 75 · mechanical · Fix: drop the Lemonade contrast, or describe what Lemonade actually reports. -
standing — The soft-wrap branch of
screen_logical_linesis not exercised on a short-TMPDIRhost —tests/e2e-cucumber/tests/e2e/tui_driver.rs:374-395
With a/tmp/rocm-e2e-XXXXXXroot, the hint line is about 72 columns and does not wrap at 80. A break in therow_wrappedrejoin alone would stay green here. Only a lane with a long temp path reaches that branch.
Confidence 70 · logic · Fix: a small unit test that feeds the vt100 parser an over-width line.
Decisions for the author
- standing — engine-shell-02 plants a manifest shape no install writes — tradeoff
- For: it reaches the shell-launch path without a multi-GiB install, and the comments explain the stand-in clearly.
- Against: a vLLM-side fix (reporting the env root) closes EAI-8049 without flipping the row, so the row then needs a manual rewrite or delete. The comments already document this. Confirm that is the intended maintenance burden.
Positive signals
assert_listed_checks_enableablederives identifiers from the listing instead of hard-coding them, and guards against an empty parse. I confirmed that it fails today for the right reason (all six checks:unknown watcher). Anid:detail line added torender_automations_textflips it to a stale XPASS.- The new hint step is solid. It reads logical lines and checks that the path exists, so it fails today on
…/bin/bin/activateand passes once the hint names the real file.
Deployment notes
None
What this covered
- Read: all six changed files in full, plus the product paths they exercise:
engine_shell/resolve_engine_env,runtime_python_activation_hint, the vLLM and Lemonade installenv_path,render_automations_text,builtin_watcher, and harness resolution and reconciliation. Range63ecab2d…485d6974, diffed againstprw-base. - Run, in this checkout's own
target/:cargo xtask e2e -- -i features/engine_shell.featureat the PR head: engine-shell-02 xfails as expected and engine-shell-01 passes.- Two product mutations, described above.
- The automations feature was run only with the mutation applied. At the PR head it was run only with
-n, which showed automations-06 failing for the stated reason.
- Harness note: a
-nname filter bypasses the harness'sfilter_runclosure, because cucumber skips the closure when a name regex is set. Every xfail row then reports as an "unexpected failure", so-iwas used for every reconciliation result above. - CI: the only failing check is "E2E tests (MI300X, current)". It fails the same way on main, apart from this PR's two expected xfails. W7900 is still queued.
- Sign-off: all 9 commits carry a DCO sign-off.
- Fan-out: two independent workers ran. One did a code pass over the whole change. The other did the agent-instruction, history and code-comment passes.
- Not run:
- The design-question fan-out (Step 7) did not run as separate workers because of the session's two-worker cap. I asked the single design question inline.
- The prior-changes pass did not run: prior review discussion is out of bounds for this run.
- Tickets: both named tickets were retrieved by ID; both are open.
- Reconciliation: not reconciled against any PR discussion here.
The step waited for the interpreter's path on screen, but engine_shell prints that path in its banner before bash starts, so the wait returned at once and a hint-only fix still flipped engine-shell-02. Run python and wait for the planted interpreter's own output, which the banner never prints. Verified by applying a hint-only fix to the product: the hint step passes and the interpreter step now fails, so the row stays xfailed. Also drop an inaccurate Lemonade contrast and an internal host name from the expectation comments. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
Verified your blocking point against the code before changing anything, and it holds. Fixed in Also done:
Left as is:
|
Review - no blocking findingsReviewed Looks safe to approve after final validation. This is a code-review comment, not an approval or independent runtime validation; I did not run tests. The mock E2E check is green, while hardware lanes are still pending. |
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 20fcc4a
This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.
Round outcome: this round files a change request, for one objection only: the first item under Blocking (new). 20fcc4a4 makes the engine-shell interpreter step able to fail, but when it fails, the e2e run hangs instead of reporting the failure. This also hits engine-shell-01, which uses the same step. The second Blocking item is published as found but does not gate: it only bites on a partial fix (hint and PATH fixed, VIRTUAL_ENV left wrong), and the usual fix, normalising the env root, corrects all three. It is cheap to close in the same push. Items are marked new (first raised this round) or standing (raised in an earlier round).
The standing objection from the previous round is lifted from the gate: the interpreter step "can never fail". This round's blind pass did not re-find it. It also re-ran the hint-only mutation independently: the hint step passed, the interpreter step timed out on planted-engine-python, and the row stayed xfail with 0 XPASS. When the objection was then put to the reviewer, it confirmed the objection no longer holds at 20fcc4a4. That confirmation is an anchored check, because the reviewer was told what to look for. The internal-environment comment, the Lemonade wording and the wrong scenario number in the description are all gone too.
The author's reply was read. Of the items it left as is, the display-name and parenthesised-id shapes were re-found blind below as non-blocking. The soft-wrap unit test is deferred to a follow-up, as the reply says.
Review — needs work
Full review of the whole change.
Pins EAI-8047 and EAI-8049 as xfail scenarios. It does not fix either bug and does not claim to close either ticket. Both tickets were retrieved and are in Backlog.
Blocking
-
new — When the interpreter step fails, it hangs the whole e2e run instead of failing —
tests/e2e-cucumber/tests/e2e/engines_steps.rs:160,tests/e2e-cucumber/tests/e2e/tui_driver.rs:1380- The step now types a bare
python\ninto the shell. Ifpythonon PATH is not the planted stub, a host with a systempythonstarts an interactive REPL, and that REPL becomes the terminal's foreground job. - When the step times out,
TuiSession::dropkills only the direct child (rocm). The kernel's SIGHUP then reaches the REPL's process group and not bash, so bash survives, orphaned, and keeps the PTY slave open. The reader thread never sees EOF, andhandle.join()blocks forever. - Before this change,
command -v pythonnever started a foreground process, so a failure stayed a clean failure. - Reproduced: I applied a hint-only product fix as a mutation. The interpreter step failed as designed (
timed out after 30s waiting for "planted-engine-python"). The e2e binary then sat infutex_do_waitfor 13 minutes beside an orphaned/bin/bash --rcfile … -i, and finished only after Ikill -KILLed that bash. - The same hang will hit engine-shell-01 the day a regression stops PATH being prepended. In CI it would turn a red scenario into a job timeout and lose every scenario after it.
20fcc4a4's message says the step "now fails". That is true only up to the panic: the run itself never completes, and the message lands on main in the squash. - Confidence 100 · logic · Fix: run the interpreter non-interactively, so a wrong
pythonexits on its own. For examplepython </dev/null\norpython -c pass\n: the planted stub ignores its arguments and still prints its marker.
- The step now types a bare
-
new (does not gate) — engine-shell-02's comment says the row flips only once the user's problem is fixed, but it flips while VIRTUAL_ENV is still wrong —
tests/e2e-cucumber/features/engine_shell.feature:38-42. Commit485d6974's message ("clears only once the user's actual problem is fixed", namingVIRTUAL_ENV) and the PR description ("the same doubled path also breaksPATHandVIRTUAL_ENV") make the same claim.engine_shellsets.env("VIRTUAL_ENV", &resolved.env_path)(apps/rocm/src/engines_cmd.rs:576), and no step checks it.- Reproduced: I mutated
engine_shellto normalise the root for the hint and forvenv_bin, and left VIRTUAL_ENV as<env>/bin. The run reported1 XPASS (0 flaky, 1 stale), so the row would be deleted with VIRTUAL_ENV still wrong. - The code governs here: VIRTUAL_ENV is what the shell actually exports, so the comment's claim is the side that is wrong.
- Confidence 100 · logic · Fix: either assert VIRTUAL_ENV inside the shell (for example,
echo "$VIRTUAL_ENV", then check the printed directory holdsbin/activate), or narrow the comment and the description to "the hint and PATH".
Non-blocking
-
standing — A fix that makes
enableaccept the check names the listing prints does not flip automations-06 —tests/e2e-cucumber/tests/e2e/automations_steps.rs:143,:374- For each check, the step tries exactly one candidate: an identifier the listing exposes, or else a slug of the display name.
- Reproduced: I mutated
automations enableto accept the display name exactly as listed ("Server recovery"). The scenario still failed: it triedserver-recovery,gpu-status-checksand so on. The row stayed xfail with 0 stale. - That fix satisfies the scenario's own wording ("can be turned on from what the listing shows"), and it is the one route that keeps the product's existing unit tests green (see Decisions).
- The description discloses that id shapes the parser does not know leave the row xfailed. This is a different kind of fix, though, not a new shape.
- Confidence 100 · logic · Fix: also try the display name verbatim, and count the check as reachable if any candidate succeeds.
-
standing — The parser does not recognise
command id: <id>, or an id in parentheses beside the label —tests/e2e-cucumber/tests/e2e/automations_steps.rs:88-105strip_prefix("id:")missescommand id: server-recover. That is the one id format the product's own tests name (apps/rocm/src/main.rs:19292,:19316,:19454assert!rendered.contains("command id:")), so it is the likeliest format to come back.- The ticket also asks for the id "alongside its label". A form like
Server recovery (server-recover) (off)falls back to a slug. - Confidence 75 · logic · Fix: accept any
… id:key, and a single whitespace-free parenthesised token before the on/off state.
Decisions for the author
- new — automations-06 pins as "correct" a listing that the product's unit tests currently forbid — tradeoff
render_automations_text_*asserts the listing does not containgpu-metrics,gpu-thermal-protectorcommand id:(apps/rocm/src/main.rs:19292-19294,:19316-19318,:19353,:19454). So hiding the ids was a tested, deliberate choice.- The ticket (EAI-8047, now keyed ROCMAI-379) asks for the opposite: show the ids, or list them in
--helpand in the "unknown watcher" error. - Either is defensible. Whoever fixes it has to delete those unit assertions or take the display-name route. The scenario or the row comment should say which is intended, so the fixer does not get contradictory signals.
Positive signals
- The interpreter step now runs
pythonand waits for the stub's own output, rather than waiting for the path. The banner already prints the path, so the old wait could be satisfied without the interpreter ever running. The PR's own hint-only mutation claim held when I re-ran it: the hint step passed, the interpreter step failed, and the row stayed xfail. screen_logical_linescorrectly rejoins soft-wrapped rows using vt100'srow_wrapped, so a long path is read whole. That avoids the truncated-path false failure.
Deployment notes
None
What this covered
- Read: all six changed files in full at HEAD, plus the product code they exercise:
automations.rs,render_automations_textand its tests,engine_shell,resolve_engine_env,runtime_python_activation_hint, and the vLLMinstall.rsenv_path. Also the harness pieces (TuiSessiondrop and reader, reconciliation). Range:fe1dbabec0…20fcc4a452. - Ran locally (each report.json mtime from this run, named scenarios executed):
cargo xtask e2e -- -i features/automations.feature: 6 scenarios, automations-06 xfail.cargo xtask e2e -- -i features/engine_shell.feature: engine-shell-02 xfail atbin/bin/activate.- Three product mutations, described in the findings above.
- CI at the head: all reported checks green. Skillscope and Sphinx were skipped. No GPU lane is listed for this test-only change.
- Sign-off: the check script reported 10 commits checked, none missing.
- Fan-out: one independent worker did the code pass. I did the agent-instruction pass, the history pass and the verification runs myself, so those were not independent. Step 7 design questions were answered in this context rather than by separate workers.
- Did not run: the prior-changes pass, because earlier reviews are out of bounds for this run.
- Not reconciled against any PR discussion here.
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 20fcc4a
Review — needs work
Test-only PR pinning two README-walkthrough defects (automations-06, engine-shell-02) as xfail cucumber scenarios, plus the TUI-driver support they need. I read all six changed files at 20fcc4a4 against the main merge-base in full, and traced the product code they exercise (apps/rocm/src/engines_cmd.rs, apps/rocm/src/main.rs's render_automations_text and its unit tests). One of the two new/changed assertions in engines_steps.rs/tui_driver.rs introduces a real hang risk in test teardown; everything else is sound test design with a couple of smaller gaps.
Blocking
- The new interpreter check can hang the whole e2e binary on failure instead of failing cleanly —
tests/e2e-cucumber/tests/e2e/engines_steps.rs:160,tests/e2e-cucumber/tests/e2e/tui_driver.rs:1340-1370,tests/e2e-cucumber/tests/e2e/tui_driver.rs:1406-1441
assert_engine_interpreternow types a barepython\ninto the shell and waits for the planted stub's own output (replacing the oldcommand -v pythonpath check).engine_shell(apps/rocm/src/engines_cmd.rs:645) launches the user's shell viacommand.status(), i.e. as a child process ofrocm, not of the test harness directly — the harness's trackedself.childis only therocmprocess. If the shell resolves a real, non-stubpython(wrongPATHcomposition, or any regression in the fix this scenario is meant to pin), thatpythonbecomes bash's interactive foreground job and keeps running past the 30swait_for_screentimeout.TuiSession::drop(tui_driver.rs:1340-1357) only callsself.child.kill()/wait()on therocmprocess — it does not reach bash or python as separate PIDs — andspawn_reader's read loop (tui_driver.rs:1412-1439) only checks thestopflag between reads; it is blocked inside aread()syscall that won't return until the pty slave's last holder closes it. With bash/python still alive and holding the slave open,reader.read()never returns, sohandle.join()inDrop(line 1368) blocks indefinitely. The scenario's own assertion still fails correctly, but the test process itself never returns — in CI this turns a red scenario into a job timeout and takes every scenario queued after it down too. Confidence 85 · logic · Fix: run the interpreter non-interactively so a wrongpythonexits on its own (e.g.python </dev/null\norpython -c pass\n— the planted stub ignores arguments and still prints its marker).
Non-blocking
- The row/scenario comments claim the row only flips once
VIRTUAL_ENVis also fixed, but nothing assertsVIRTUAL_ENV—tests/e2e-cucumber/features/engine_shell.feature:38-42,tests/e2e-cucumber/expectations.toml(EAI-8049 block)
engine_shellsetsVIRTUAL_ENVto the unmodifiedresolved.env_path(apps/rocm/src/engines_cmd.rs:586), and the newassert_engine_interpreterstep only checks that runningpythonexecutes the planted stub — it never reads or asserts theVIRTUAL_ENVvalue. A fix that corrects the activation hint andPATHbut leavesVIRTUAL_ENVdoubled would still make this scenario pass (XPASS), contradicting the comment. Fix: either assertVIRTUAL_ENVinside the shell (e.g.echo "$VIRTUAL_ENV") or narrow the comment/description to "the hint and PATH". - The id-parsing fallback misses two plausible real-fix shapes —
tests/e2e-cucumber/tests/e2e/automations_steps.rs:88-105
exposed_identifieronly recognizes a detail line that starts withid:/identifier:/slug:, so a rendering likecommand id: server-recover(a form the product's own tests explicitly name, see below) is missed. It also only recognizes a bracketed id on the header (Server recovery [server-recover]); a parenthesised form (Server recovery (server-recover) (off)) falls through to the display-name slug, which does not reduce to the real id. Either shape would leave the row needlessly xfailed after a reasonable fix. Not a correctness bug — the scenario is still sound today — but worth tightening so the row doesn't outlive the behaviour it pins.
Decisions for the author
automations-06pins a listing contract that the product's own unit tests currently forbid — tradeoff
render_automations_text's existing tests assert the rendered listing does not containgpu-metrics,gpu-thermal-protect, orcommand id:(apps/rocm/src/main.rs:19292-19294,:19310+), i.e. hiding raw ids was a deliberate, tested choice. This scenario asserts the opposite: every listed check must be enableable from what the listing shows. Either a display-name-based fix toautomations enableor an id-exposing fix to the listing resolves it, but whoever picks up EAI-8047 has to either delete those unit assertions or take the display-name route (which, per the non-blocking item above, the current parser wouldn't even recognize fully). Worth a one-line note in the row/scenario saying which direction is intended, so the fixer isn't working against contradictory signals.
Positive signals
listed_checks/exposed_identifierinautomations_steps.rsgenuinely derives ids from the rendered listing rather than hard-coding them — verified directly againstrender_automations_text's indentation (" {name} ({state})"header," "-indented details), so the scenario can't pass against a listing that exposes nothing.screen_logical_linescorrectly rejoins soft-wrapped rows viavt100'srow_wrapped, avoiding a false failure from a long activation path splitting across two terminal rows.
Deployment notes
CI at this head is green apart from one failing check, E2E tests (Strix Halo, Windows, current), and a few hardware lanes still pending (W7900, MI300X, Strix Halo nightly/release); I could not open the failing job's log from this environment to tell whether it's related to this change (the only new OS-scoped scenario here is tagged @requires-os:linux, and automations-06 carries no GPU/OS tag). gh pr view reports mergeable: MERGEABLE but mergeStateStatus: BLOCKED, consistent with the outstanding change-request review rather than a merge conflict. This is test-only — no migration or runtime compatibility concerns.
What this covered
Read all six changed files in full (tests/e2e-cucumber/expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs) against the merge-base 63ecab2d..20fcc4a4, plus the product code they exercise (apps/rocm/src/engines_cmd.rs's engine_shell/resolve_engine_env, apps/rocm/src/main.rs's render_automations_text and its unit tests). Did not run cargo xtask e2e or any test/lint locally — no Rust toolchain or network access for cargo in this sandbox, so the hang finding above is a traced-through-the-code argument, not a reproduced run. Read the PR description and the existing review threads for context, including the prior automated review at this same head; where its findings overlapped mine I re-derived them independently from the source before including them here, and I did not treat that review's conclusions as given.
The interpreter step typed a bare python into the shell. When PATH is wrong that can start a real REPL as the terminal's foreground job; it outlives the kill of rocm, keeps the PTY open and hangs teardown instead of failing the scenario. Run python with stdin at EOF so a wrong interpreter exits on its own. The same step now also checks that VIRTUAL_ENV names an environment holding bin/activate, so engine-shell-02 flips only once the hint, PATH and VIRTUAL_ENV are all right. Verified with two product mutations: a hint-only fix and a hint+PATH fix each leave the row xfailed, and the run completes. automations-06 now tries every plausible identifier per check (exposed ids, including a 'command id:' key or a parenthesised token, then the label as listed, then its slug) and counts the check reachable if any works. The row comment says which fix directions are accepted. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
|
Thanks both. The hang is real and now fixed in Blocking: the interpreter step could hang the run. A wrong VIRTUAL_ENV. You were right that nothing asserted it. The step now also runs automations-06 candidates. The step now tries every plausible identifier per check and counts it reachable if any works: exposed ids (an Decision (the unit tests forbid the ids). The row comment now says either direction closes it and the scenario accepts both, and that Still deferred: the soft-wrap unit test, with the alt-screen follow-up. The Strix Halo Windows failure is unrelated: |
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 9f4bc82
Review
Full review of the whole change, against the current head (9f4bc826), a fresh pass independent of any earlier round's conclusions. Test-only PR pinning two README-walkthrough defects (automations-06 for EAI-8047, engine-shell-02 for EAI-8049) as xfail cucumber scenarios, plus the TUI-driver/parsing support they need. I read all six changed files in full at this head against the origin/main merge-base (63ecab2d), traced the product code both scenarios exercise (apps/rocm/src/main.rs's render_automations_text and apps/rocm/src/engines_cmd.rs's engine shell/resolve_engine_env, plus crates/rocm-core/src/runtime.rs), and verified vt100 0.16's row_wrapped/contents_between semantics against the crate itself. Two independent fan-out passes (one per coupled file group: the automations-06 slice, and the engine-shell-02 slice) additionally traced the same code blind and corroborate: no confirmed code-level defect survived scrutiny in either slice. This implements the two remaining xfail pins from a four-defect batch; the PR description says the other two (EAI-8046, EAI-8048) were dropped because main fixed them independently while this branch was open — consistent with the current diffstat, which touches neither chat nor dash files. No ticket tracker is reachable from here (EAI-xxxx is not a GitHub issue), so that claim is taken on the diff's own evidence, not verified against a tracker.
Blocking
- The PR description's account of the id-matching logic is inaccurate — it says "two shapes" and claims other shapes stay unrecognized, but the code recognizes more and would not stay xfailed on those shapes —
tests/e2e-cucumber/tests/e2e/automations_steps.rs:59-105(exposed_identifiers), PR description "What it pins" bullet for EAI-8047
The description: "It recognises an id in two shapes: anid:/identifier:/slug:detail line, or a bracketed token on the header... A fix that prints it some other way leaves the row xfailed until the parser learns that shape." The actualexposed_identifiersalso matches (a) any detail-line key ending in" id"— socommand id: <x>matches too, which is exactly the shape the product's own unit tests spell out literally (apps/rocm/src/main.rs:assert!(!rendered.contains("command id:")), three occurrences) — and (b) a parenthesised token on the header (Server recovery (server-recover)), not just a bracketed one; both are explicitly named in the function's own doc comment just above it. So the description's closing sentence is false for exactly the two shapes it omits: acommand id:-style fix, or a parenthesised-id fix, would not leave the row xfailed, contrary to what the description tells a reader to expect. No functional risk — the code is strictly more permissive than described, so nothing is silently missed by the test itself — but it misstates what was built, which is unreviewed work by the description's own logic. This is also the exact category of description/code mismatch earlier rounds on this PR treated as blocking.
Confidence 85 · logic · Fix: update the description to list all four matching strategies (two detail-line forms, two header forms), or drop the itemization and point at the code comment instead. One-line/paragraph fix, not a rework.
Non-blocking
- A header-embedded id placed after the on/off marker would silently drop that check from the test rather than making it fail —
tests/e2e-cucumber/tests/e2e/automations_steps.rs:110-147(listed_checks)
listed_checksonly recognizes a header as a check at all if the trimmed line's literal suffix is"(on)"or"(off)"(strip_suffix). Bracket/paren extraction then runs on the header text with that suffix already stripped, so an id token placed before the state marker (Server recovery [server-recover] (off)) is found correctly — but a hypothetical fix that instead appends it after (Server recovery (off) [server-recover]) would make the line's literal suffix]instead of(off), fail thestrip_suffixcheck, and cause the whole check to vanish fromcheckssilently — not "tried and failed," just absent — rather than being caught byassert!(!checks.is_empty(), ...)(which only guards a totally empty listing, not a partially-dropped one). All of the candidate shapes the code's own comments enumerate place the id before the marker, so this is speculative rather than demonstrated, which is why it is non-blocking.
Confidence 70 · logic · Fix: strip"(on)"/"(off)"viacontains(matching anywhere in the line) rather than requiring it as a literal suffix, so an id appended after the marker is still recognized as the same header.
Decisions for the author
- The parsing helpers (
slug_of,exposed_identifiers,listed_checks) have no isolated Rust unit tests — non-blocking-improvement
This logic has gone through several rounds of tightening across this PR's history and is non-trivial (multi-shape candidate extraction, header/detail boundary detection). It is exercised end-to-end by theautomations-06scenario against the real CLI's current output, which is real coverage, but a focused#[test]table over synthetic listing text would catch a regression in the parser itself faster and more precisely than an e2e failure would. Current approach is not wrong, just a cheaper alternative exists.
Positive signals
- The EAI-8049 fixture (
plant_engine_env_recorded_as, planting a bin-shapedenv_path) is traced precisely against the real bug mechanism inengines/vllm's install response andresolve_engine_env's unnormalized use of it — this is a fixture that reproduces the actual reported defect, not a stand-in that merely looks similar. python </dev/nullfor the interpreter check is a well-targeted fix for the PTY-hang risk a prior round found: it makes a wrong interpreter exit on EOF instead of becoming bash's foreground job, while leaving the planted stub (which ignores stdin) unaffected.screen_logical_lines's use ofrow_wrappedis correct in both directions checked (continuation set from the current row and consulted on the next one), and concatenates wrapped rows with no inserted separator — correct for a raw-terminal hard wrap, as opposed to the codebase's existing word-wrap reconstruction helper for ratatuiWrap, which deliberately does insert one. Good instinct not to reuse logic built for a different wrapping model.- The
VIRTUAL_ENVcheck added in this head's own commit closes a real, previously-identified gap (a hint-only fix could otherwise flip the row whileVIRTUAL_ENVstayed wrong) with a minimal, non-echoable assertion (venv-ok-$((6*7)), computed by the shell, not just echoed input).
Deployment notes
None. Test-only change (tests/e2e-cucumber/**); no product crate touched, no new dependency, no migration.
What this covered
Every line of all six changed files (tests/e2e-cucumber/expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs) at 63ecab2d…9f4bc826, plus the product code both new scenarios exercise (apps/rocm/src/main.rs's render_automations_text and its unit tests, apps/rocm/src/engines_cmd.rs's engine-shell command, crates/rocm-core/src/runtime.rs's activation-hint helpers, and engines/vllm's install response shape) and AGENTS.md for applicable conventions (internal-content leak scan: clean; DCO sign-off present; CI fully green including the required e2e/mock lanes). Two independent background passes additionally reviewed the automations-06 and engine-shell-02 file groups blind and corroborated no code-level defect in either.
Did not run: history (git log/git blame on the modified regions) and prior-changes passes were not run as separate dedicated passes — this PR's own long review history was read for context and for Step 9 framing, which means this round is not a strictly blind pass in the sense the method otherwise requires; every finding above was nonetheless independently re-derived and verified directly against the current code rather than carried over from that history. No ticket-tracker tool is available here to retrieve EAI-8047/EAI-8049 directly; their status is taken from the PR description's own account. Not reconciled against unresolved discussion on this PR beyond what's stated above.
|
Fixed the description: it now lists all four id shapes the parser accepts (id:/identifier:/slug: or any key ending in ' id' detail lines; bracketed or parenthesised header token). No code change, head stays 9f4bc82. Not taking the non-blocking items (id-after-marker, parser unit tests) in this PR; the parser is exercised end to end by automations-06. Re-review of 9f4bc82 requested. |
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/features/automations.feature # tests/e2e-cucumber/tests/e2e/automations_steps.rs
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/features/automations.feature # tests/e2e-cucumber/tests/e2e/automations_steps.rs
nowycondro
left a comment
There was a problem hiding this comment.
🔴 Automated review · pr-review-watcher · 0d286fd
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (test-only; no product code touched).
Implements regression-pinning test coverage for two open, Backlog-status Jira tickets — EAI-8047 (resolved via lookup to ROCMAI-379, "automations enable/disable ids are not discoverable from the CLI") and EAI-8049 (resolved via lookup to ROCMAI-377, "rocm engines shell prints an activate_hint with a doubled bin segment") — retrieved independently by identifier; both tickets' repro details match this PR's scenarios almost verbatim (e.g. ROCMAI-379's exact "Server recovery → server-recover, not server-recovery" slug mismatch). The PR does not claim to close either ticket, only to pin them as xfail, which matches what the diff does.
Blocking
None
Non-blocking
None
Decisions for the author
plant_engine_env_as_installed's bin-shapedenv_pathmeans engine-shell-02 cannot go stale from anengines/vllm-only fix — answered-by-intent. Traced: the fixture always writespython/activateat<root>/engine-env/bin/*, but for scenario 02 the planted manifest recordsenv_pathas that samebindirectory (mimickingengines/vllm/src/install.rs'senv_path = command.parent()).resolve_engine_env's manifest branch passes it through unchanged, andruntime_python_activation_hintappendsbin/activateagain, so the hint resolves to abin/bin/activatethat was never created — reproducing EAI-8049 exactly, which is why the row isxfailtoday. One fan-out worker initially flagged this as a defect ("the fixture can't pass"), but failing today is the intended state for a pinned defect; the real question is whether it can ever flip. The PR's ownexpectations.tomlcomment already discloses the narrowed scope: "A fix in engines/vllm that reports the environment root... does NOT flip this row... Only a fix to the hint composition itself flips it." ROCMAI-377's own "Expected" field ("build the hint from the environment root rather thanenv_path") points at exactly that hint-composition fix, so the scenario is well-targeted at the most likely real fix location, not a dead end. Confirmed independently that no code inapps/rocm,crates/, orengines/ever writes a<env_id>.jsonmanifest to disk (onlyload_engine_env_manifestreads one) — so the PR's claim that "no install writes such a manifest to disk" is accurate, not a cover for an untested path. Nothing to fix; flagging only so the narrowed-flip-condition tradeoff is visible and confirmed as still the intended design.user_lists_checksassertsrc == 0inline instead of deferring to aThenstep viarecord()— non-blocking-improvement,tests/e2e-cucumber/tests/e2e/automations_steps.rs:526-534. Every otherWhenstep in the file (enable_observe,enable_propose,disable_watcher,list_automations,enable_unknown, the proposal actions) callsrecord(world, stdout, stderr, rc)and leaves assertions to a laterThen; this one is the only one that asserts inline and the only one that skipsrecord()(soworld.cli_stderr/cli_rcare never populated for this action). It isn't wrong — failing fast here with the raw stdout/stderr in the panic message is arguably clearer than lettingassert_listed_checks_enableablediscover an empty listing and panic about "no background checks found" — but it is a visible deviation from the file's own established convention, worth a one-line comment explaining the choice (or switching to the convention) rather than silent inconsistency for the next reader.- Candidate-guessing in
listed_checks/exposed_identifierscould in principle mask a defect via an id/slug collision — tradeoff. The fallback order (exposed id → label as-is → slug) is deliberately permissive so a correct fix is picked up in any of several shapes; verified against all 6 realBUILTIN_WATCHERSids and their display names that no guessed candidate for one check collides with a different check's real id today, so this isn't live, but a future watcher whose slugified display name happens to equal another watcher's real id would silently "pass" the wrong check's identity. No action needed now; worth a comment or a cross-check test if the watcher table grows. screen_logical_linescannot distinguish a logical line that starts at the visible screen's top row from one whose start scrolled off above it — non-blocking-improvement,tests/e2e-cucumber/tests/e2e/tui_driver.rs(new method). Traced the wrap-stitching logic itself and it correctly mirrors terminal auto-wrap semantics (only concatenates whenrow_wrappedis true, which only fires when a row was genuinely filled and overflowed); the only gap is the inherent screen-snapshot limitation of row 0 always being treated as a fresh line start. Not triggered in this PR's actual use (theactivate_hint:line prints ~8 lines into the session, well clear of row 0), so not a finding against this change, just worth a one-line doc caveat since the method is positioned as "read a whole value off the screen."
Positive signals
automations-10's identifier derivation is specifically designed to defend against the laziest false-pass: ids are deliberately not hard-coded "since that would pass against a listing that exposes nothing," and the step's own guard (assert!(!checks.is_empty(), ...)) exists so a parser that silently matched zero checks can't "prove" the contract by checking nothing.engine-shell-02's second assertion (runningpythonand checking$VIRTUAL_ENV/bin/activate) is deliberately there because "the same doubled path also breaks PATH and VIRTUAL_ENV, so a fix to the hint text alone does not flip the row" — traced and confirmed: both checks exec real shell behavior (actual stdout from the planted interpreter stub, actualtest -fthrough a$((6*7))-gated marker immune to the typed command line's own echo), so a hint-only cosmetic fix genuinely cannot satisfy this row.- Both pinned scenarios' technical claims were independently cross-checked against the two Jira tickets they cite (retrieved by identifier, not by following a link) and matched almost verbatim, including exact slug-mismatch examples in EAI-8047/ROCMAI-379.
Deployment notes
None. Test-only change (tests/e2e-cucumber/**); no product crate is touched, no migration, no API or behavior change for an operator to prepare for.
What this covered
Read the full diff between base 8c23a91abed959df3233651346a379feea3fa0e2 and the pinned head 0d286fddd414a72b78567d5667938ffbcab6ab58 (verified via git diff --stat matching GitHub's reported 6 files / +414/-15), all 6 changed files in full (expectations.toml, automations.feature, engine_shell.feature, automations_steps.rs, engines_steps.rs, tui_driver.rs), plus the product source each test actually exercises (apps/rocm/src/automations.rs, crates/rocm-core/src/automation.rs, apps/rocm/src/engines_cmd.rs, crates/rocm-core/src/runtime.rs, engines/vllm/src/install.rs, apps/rocm/src/runtime_install_finalize.rs) to verify the tests' factual claims against real control flow rather than trusting the PR's narrative. Confirmed the head is itself a clean merge of main into the feature branch (second parent = base sha exactly) with no leftover conflict markers and no duplicate scenario ids or step definitions.
Fan-out: two independent blind-review workers, one per coupling group (automations-10 scenario + its product listing/enable code; engine-shell-02 scenario + its product env-path/hint-composition code), each seeded with full file content and relevant product excerpts but never the PR description. First send to each worker contained an unfilled content placeholder by my own error; both workers correctly refused to fabricate findings from it rather than guessing, so both were re-sent with the actual file content and completed a full trace-level pass on the second round. Their one candidate finding each were verified above: one was refuted as describing exactly the intended, pre-disclosed xfail behavior (retired as answered-by-intent); one survived as a genuine but non-blocking convention note.
Design review (Step 7) considered durability of engine-shell-02's fixture against the real fix location, the repo's established xfail-pinning convention (consistent with cited PR #216), id/slug collision risk, the merge-commit-as-PR-head question (no CONTRIBUTING.md rule against it found; not a concern), and screen_logical_lines's scope — none rose to a fundamental approach problem. Checked git log history on all 6 files, including confirming the earlier-reviewed commit 9f4bc826117552220ff43ce0db348ec32832970b is this branch's own ancestor. Read AGENTS.md's testing-convention sections; found no violation (no product behavior changed, so the docs-sync and message-assertion rules don't apply; the mutation-test requirement is explicitly satisfied per the PR's own test plan for engine-shell-02, and implicitly satisfied for automations-10 by the scenario currently reproducing the real defect).
Read CI results rather than re-running anything (no Rust/Cargo toolchain available in this sandbox): every required check is green at this exact head (clippy, build-and-test, Test (affected crates), windows-build-and-test, the blocking E2E mock job and consolidated report, commit-signoff, license headers, prek, stale-xfail-hint, markdown links, third-party notices, coverage). Two self-hosted hardware E2E lanes (MI300X, W7900) were still QUEUED and two (Strix Halo Ubuntu release/nightly) were CANCELLED at observation time — no FAILURE conclusion anywhere; the PR's own scope states neither scenario needs GPU dispatch, so this doesn't bear on the blocking gate.
Retrieved both named tickets by identifier (EAI-8047, EAI-8049), per Step 9, after findings were settled — see the opening block.
Did not run: the "prior changes" source-reading pass (whether past review comments on these files apply here) — doing this meaningfully requires reading this PR's own review-thread content, which the blind-review rule excludes; not substituted with guesswork. Code-comments-in-files were read as part of each full-file read, not as a separate pass.
Two disclosures for auditability: (1) the consolidating context (not the fan-out workers) read the PR description and expectations.toml's own comments early, during scope/fact-gathering, ahead of the skill's prescribed Step-9 ordering — the fan-out workers that actually produced candidate findings were kept blind throughout, so this affected framing/verification speed, not the findings' independence. (2) The Step-3 pre-publish race check (get_reviews, to confirm no existing review from "nowycondro" at this exact head) returned full review bodies rather than just login/commit metadata; a small amount of other reviewers' approval text was incidentally visible in that call's output. No specific claim from it was adopted into any finding above — all findings here were reached via independent code-tracing and the blind fan-out before that call was made. Confirmed via that same call: "nowycondro" has reviews only at 20fcc4a4 and 9f4bc826 (earlier commits), none at the pinned head, so publishing this is not a duplicate. Not reconciled against any other discussion on this PR.
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/features/automations.feature
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/features/automations.feature
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review (Claude Code) · 0e4d264
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (base db7e54089ac668ab86d100bc2456df6230ee762f → head 0e4d264f73aa6c503cde4f6d92632da744f0304f, 6 files, +414/-15).
This implements test-only scenarios pinning two already-open, backlog-status defects: Jira EAI-8047→ROCMAI-379 ("automations enable/disable ids are not discoverable") and EAI-8049→ROCMAI-377 ("engines shell activate_hint has a doubled bin segment"). Both tickets were retrieved and read; their content matches what the new scenarios assert and what the production code (apps/rocm/src/automations.rs, apps/rocm/src/engines_cmd.rs, crates/rocm-core/src/runtime.rs, engines/vllm/src/install.rs) actually does — the PR does not claim to close either ticket, only to pin regression coverage for the correct eventual behaviour.
Blocking
None
Non-blocking
- Stale file path in a new comment —
tests/e2e-cucumber/expectations.toml:202
The newly added comment saysrender_automations_text's unit tests live inapps/rocm/src/main.rs. At the reviewed head that function and its unit tests (render_automations_text_marks_gpu_metrics_read_only, etc.) are inapps/rocm/src/automations.rs(confirmed bygit grepagainst the PR head);main.rsno longer holds them. Confidence 85 · mechanical · Fix: update the path in the comment toapps/rocm/src/automations.rs.
Decisions for the author
None
Positive signals
tests/e2e-cucumber/tests/e2e/automations_steps.rs'slisted_checks/exposed_identifiersparser is deliberately permissive (tries several id shapes) specifically so a real fix doesn't have to match one hard-coded format to flip the row — the comments say so and the parsing logic (checked againstrender_automations_text's actual two/four-space indentation scheme) backs that up.engines_steps.rs'sassert_engine_interpreterpairs the activation-hint text with the actual runtime state (VIRTUAL_ENV, and really runningpython) rather than only pinning wording, matching this repo's own "a message is asserted together with the state it describes" convention (AGENTS.md§3 / e2eREADME.md).- The
</dev/nullon the interpreter-exec command is a real, well-explained guard against a hung PTY teardown ifPATHresolves to a real system Python instead of the planted stub.
Deployment notes
None
What this covered
DEGRADED REVIEW: both attempts to fan out to independent sub-reviewers failed (Concurrent subagent limit reached), so this ran single-context rather than as the independent-worker fan-out the method calls for; recommend re-running with fan-out available, since it typically surfaces more than a single-context pass does.
Read completely at the PR head: tests/e2e-cucumber/expectations.toml, tests/e2e-cucumber/features/automations.feature, tests/e2e-cucumber/features/engine_shell.feature, tests/e2e-cucumber/tests/e2e/automations_steps.rs (1084 lines), tests/e2e-cucumber/tests/e2e/engines_steps.rs (249 lines), and the changed screen_logical_lines addition plus its surrounding impl TuiSession block in tests/e2e-cucumber/tests/e2e/tui_driver.rs. Cross-checked the new test/parsing logic directly against the production code it exercises: apps/rocm/src/automations.rs (render_automations_text, automations(), watcher_plain_name), crates/rocm-core/src/automation.rs (builtin_watcher/BUILTIN_WATCHERS), apps/rocm/src/engines_cmd.rs (engine_shell, resolve_engine_env), crates/rocm-core/src/runtime.rs (runtime_python_activation_hint), and engines/vllm/src/install.rs (the command.parent() env_path computation) — all confirmed to match what the new scenarios assume. Verified the vt100 crate's documented row_wrapped semantics (docs.rs) against the new soft-wrap-rejoining loop in screen_logical_lines and found the sequencing correct. Read AGENTS.md and tests/e2e-cucumber/README.md for applicable conventions (scenario/@id naming, enforced by tests/feature_naming.rs — confirmed automations-12 and engine-shell-02 are sequential and correctly keyed). Read CI results for the reviewed commit via gh pr checks/job logs: the blocking mock E2E tests / E2E consolidated report jobs pass; two non-blocking self-hosted GPU/Linux jobs (E2E tests (Strix Halo, Ubuntu, release/nightly, current)) fail, but the failure is an unrelated pre-existing regression in a different, untouched scenario (serve-stop-leaves-no-background-helper) — the job logs show the two new scenarios in this PR reconciling as expected xfail there too, not as the cause of the job failure.
Did not run: history/git blame on the modified regions, and a prior-changes pass against earlier review rounds on this PR — per the method's own rule, this review did not read this PR's discussion, comments, or earlier review rounds at all, and nothing from them fed into the findings above. No Rust toolchain was available to actually execute cargo test/mutate the test binary locally, so the two required test-quality verdicts below are argued from reading the code against the production paths it exercises, not from an actual run.
Test-quality verdicts (both new scenarios, per the project's own testing convention):
automations-12/assert_listed_checks_enableable: traced that todayrender_automations_textnever emits any of the id-shaped tokens the parser looks for (checkedwatcher_plain_name's literal strings — none contain:/[/(), andbuiltin_watchermatches only the exact internal id, so every derived candidate (display label, slug) is rejected and the scenario fails now, as itsxfailrow expects; if the listing exposed a real id in any of the four recognised shapes, the derived candidate would succeed and the row would go stale (XPASS) — so this is not a tautological test.engine-shell-02/assert_activation_hint_resolves+assert_engine_interpreter: traced thatresolve_engine_env's auto-install branch uses the install response'senv_pathlive, thatengines/vllm/src/install.rssets it tocommand.parent()(the bin dir), and thatruntime_python_activation_hintappendsbin/activateto whatever it's given — so a bin-shapedenv_pathdoubles the segment today, matching the planted fixture. A hint-wording-only fix (per the PR's own stated mutation test) would still leaveVIRTUAL_ENV/PATHwrong, soassert_engine_interpreter's separate interpreter-execution check keeps the row failing until the real environment-root fix lands — confirmed by reading the two assertions' independence, not by running the mutation myself.
Not reconciled against any prior review round or discussion on this PR.
🤖 Generated with Claude Code
…-for-the-second Signed-off-by: fredespi <fredrik.espinoza@gmail.com> # Conflicts: # tests/e2e-cucumber/expectations.toml # tests/e2e-cucumber/features/automations.feature # tests/e2e-cucumber/tests/e2e/automations_steps.rs
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 530f779f633
Outcome: no blocking findings.
Test-only change pinning two known CLI defects (EAI-8047, EAI-8049) as xfail scenarios in tests/e2e-cucumber/, following the existing expectations.toml + Gherkin + step-function pattern used elsewhere in this suite.
What I verified against the actual source (not just the diff):
automations-13's listing parser (automations_steps.rs:listed_checks/exposed_identifiers/slug_of) was traced against the real textrender_automations_text(apps/rocm/src/automations.rs) produces today — it correctly finds zero usable identifiers, matching the pinned bug, and a!checks.is_empty()guard prevents a vacuous pass.rocm automations listhas no--jsonoutput today, so text-parsing is the only option available.engine-shell-02's doubled-path claim checks out:engines/vllm/src/install.rsreportsenv_path = command.parent(), andruntime_python_activation_hint(crates/rocm-core/src/runtime.rs) appendsbin/activateagain. Both new assertions (hint-file-exists, interpreter+VIRTUAL_ENV) are independently load-bearing — a hint-text-only fix would leave the second one red, as the PR's own test plan claims.- The new
TuiSession::screen_logical_lineshelper (vt100-based soft-wrap rejoining) matches thevt1000.18 crate's actualrow_wrapped/contents_betweensemantics; no corruption at wrap boundaries. - Pulled the self-hosted GPU CI logs for this exact head commit: both new scenarios reproduce and fail for precisely the documented reason, correctly reconciled as expected xfails (not unexpected failures). The blocking mock
e2ejob is green. - Two independent blind fan-out passes (automations group, engine-shell group) separately traced the same code paths and returned no findings ≥70 confidence.
Non-blocking / FYI:
- The non-blocking self-hosted GPU jobs show one unrelated failing scenario (
serve-stop-leaves-no-background-helper), not touched by this diff.
What this covered: full diff (6 files, 962ac0c0…530f779f) read completely, plus the production code it exercises and the repo's AGENTS.md/e2e README.md conventions. Did not reconcile against the PR's existing review thread.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
juhovainio
left a comment
There was a problem hiding this comment.
Review — no blocking findings in the diff itself; mixed CI/merge state
Full review of the whole change (merge-base 962ac0c0 … head 530f779f). Test-only: pins two open defects (EAI-8047, EAI-8049) as xfail rows in expectations.toml with one BDD scenario each, in the same shape as prior art #216. No product code is touched. EAI-8047/EAI-8049 are not GitHub-trackable identifiers from here, so the ticket relationship is as stated in the description and not independently retrievable.
Blocking
-
Branch cannot currently be merged — PR-level, not a line in the diff.
GitHub reportsmergeable: CONFLICTING/mergeStateStatus: DIRTYagainst currentmain. The PR's merge-base (962ac0c0, "ci: start build-and-test without waiting for clippy (#875)") is far behindmain's current tip (which has since removed the W7900 lane frome2e-selfhosted.ymlin #1066, among many other merges) — this branch has been open since August and needs a rebase before it can land.
Confidence 100 · architectural · Fix: rebase onto currentmain, resolve conflicts, re-push. -
Three self-hosted hardware E2E lanes are red —
E2E tests (MI300X, current),E2E tests (Strix Halo, Ubuntu, release, current),E2E tests (Strix Halo, Ubuntu, nightly, current)(allFAILURE), plusE2E tests (W7900, current)CANCELLEDand the self-hosted consolidated reportFAILURE.
Read the actual failure line from each job's log (not inferred from the job name): all three failures are the identical lineFAIL: 'serve-stop-leaves-no-background-helper' was expected to pass on this host but FAILED — a regression.That scenario is untouched by this diff (the PR only addsautomations-13andengine-shell-02, bothwhen = {}/mock-scoped per the expectations.toml rows I read), and the same scenario fails identically across three independent hardware lanes, which points at a pre-existing or environment-level regression rather than something this PR introduced. I could not get a clean (non-cancelled) recent self-hosted run on currentmainto confirm it fails there too, so I can't fully close this out — but nothing in the failure output implicates this PR's two new scenarios. The mock-lane check the PR's own test plan calls out as blocking (E2E consolidated report, CI workflow) is green, as claimed.
Confidence 85 (verified from the actual log output, not assumed) · architectural · Fix: re-run self-hosted after rebasing; ifserve-stop-leaves-no-background-helperis already red onmain, this is not this PR's problem to fix, but it should be confirmed clean (or separately triaged) before merge.
Non-blocking
None
Decisions for the author
None
Positive signals
EAI-8049's premise was verified against the actual product code, not just the PR's own comments:engines/vllm/src/install.rsdoes reportenv_pathasruntime.command.parent()(the bin dir), andruntime_python_activation_hint/runtime_python_env_bin_dirincrates/rocm-core/src/runtime.rsdo treat whatever they're given as the environment root and unconditionally appendbin/activateorbin— so the doubled-path bug, and its claimed effect onVIRTUAL_ENVandPATH(both built from the same untransformedresolved.env_pathinapps/rocm/src/engines_cmd.rs), are real and accurately diagnosed.EAI-8047's premise checks out the same way:watcher_plain_nameinapps/rocm/src/automations.rsreturns only the display name, no field inrender_automations_text's output carries the raw watcher id, andbuiltin_watcher()does an exact-string lookup with no slug or label fallback — so the listing genuinely publishes nothingenablewill accept, exactly as pinned.- Good test-design judgment:
plant_engine_env_as_installedplants a manifest (rather than mocking a live install response) — tracingresolve_engine_env, both themanifestandauto_installbranches feed the same untransformedenv_pathinto the hint/PATH/VIRTUAL_ENV composition, so the fixture genuinely exercises the bug path rather than taking a shortcut around it. assert_listed_checks_enableabledeliberately derives candidate ids from the listing (detail-line keys, bracketed/parenthesised header tokens, raw label, slug) rather than hard-coding the real ids, which is the only way this scenario can go stale automatically once the product actually publishes something usable.- The rewritten
assert_engine_interpreteris a strict improvement even for the pre-existing scenario that reuses it: it now runspythonto observe real output instead of just comparingcommand -v python's printed path, and adds aVIRTUAL_ENV/bin/activatecheck that didn't exist before — this closes a real coverage gap, not just for the new scenario. - The
</dev/nullreasoning inassert_engine_interpreter's comment (avoiding a hung REPL as PTY foreground job ifpythonresolves wrong) is sound and shows real thought about E2E hygiene.
Deployment notes
None — test-only change, no product behavior, migration, or configuration affected.
What this covered
Read completely: all 6 changed files (tests/e2e-cucumber/expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs) at the PR head commit, read via git show pr-292:<path> directly from the git object store (the working-tree checkout was found to have been mutated by unrelated concurrent activity in a shared sandbox, so file reads were taken from committed blobs rather than the working tree to avoid acting on contaminated state). Also read, to verify the tests' claims against reality rather than taking them on faith: engines/vllm/src/install.rs, crates/rocm-core/src/runtime.rs (runtime_python_activation_hint, runtime_python_env_bin_dir, runtime_python_env_root_for_executable), apps/rocm/src/engines_cmd.rs (resolve_engine_env, engine_shell), apps/rocm/src/automations.rs (render_automations_text, watcher_plain_name, the Enable/Disable command handlers), and crates/rocm-core/src/config.rs (builtin_watcher, BUILTIN_WATCHERS). Checked CI via the status-check rollup and read the actual failure-log lines for every failing job rather than trusting job names. Checked current mergeability via the GitHub API.
Did not run: no test suite was executed; all correctness conclusions come from reading the diff against the actual production source it exercises, not from execution. Did not run: history/prior-changes passes beyond the targeted git log/git show lookups described above — no further value expected given the production-code tracing already done. A second independent fan-out pass over this same scope was launched for corroboration but had not completed by the time this review was finalized; it is not reflected here. Not reconciled against the PR's existing review threads or comments — those were deliberately not read, per this review's own method.
|
Addendum to the review submitted above: the required diff-id marker and footer were omitted from that review body by mistake. Appending them here since the review API does not expose an edit path from this tooling. 🤖 Automated by an AI agent |
juhovainio
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change.
Pins EAI-8047 and EAI-8049 as xfail regression scenarios in tests/e2e-cucumber/; does not claim to close either ticket, only to pin the correct eventual behaviour. Both are bare external ticket IDs with no tracker access available to this review, so their content is taken from the PR description and corroborated independently against the production code the new scenarios exercise (see below), not retrieved directly.
Blocking
- This branch is not currently mergeable into
main—tests/e2e-cucumber/expectations.toml:168(inline comment has the detail)
GitHub reportsmergeable: CONFLICTING/mergeStateStatus: DIRTYfor this exact head. Independently reproduced viagit merge-tree: the only conflict is a textual clash from two independent appends landing near the same spot in this file, becausemainhas kept moving since this branch's last sync (merge-base with the currentmaintip is962ac0c0, ~168 commits behind). Every other changed file merges cleanly.
Confidence 100 · architectural · Fix: merge/rebase onto currentmain, keep both appended blocks, push. Mechanical, not a code defect.
Non-blocking
no_background_checksduplicates an existing step —tests/e2e-cucumber/tests/e2e/automations_steps.rs:544(inline comment has the detail)
Same body asfresh_configinconfig_steps.rs(both just callsuppress_daemon_spawn(world)). Not wrong, but worth a one-line comment or reuse so the duplication reads as deliberate.
Confidence 75 · mechanical · Fix: note why, or reuse the existing step phrasing.
Decisions for the author
engine-shell-02's fixture plants a manifest shape no real install writes to disk — tradeoff
Traced and confirmed accurate:engines/vllm/src/install.rs'sinstall_responsedoes reportenv_pathascommand.parent()(the bin dir, via the auto-install branch ofresolve_engine_env), and no code anywhere writes a<env_id>.jsonmanifest — the fixture stands in for that response, as already disclosed in the PR's own comments. Consequence, also already disclosed: a fix inengines/vllmalone (reporting the environment root) closes the underlying defect but does not flip this row, since the planted manifest is unchanged — only a fix to the hint composition does. Flagging only so the maintenance burden is visibly confirmed as intended, not to re-litigate it.
Positive signals
assert_listed_checks_enableable's identifier derivation deliberately avoids hard-coding ids (tries several shapes a fix might expose, falling back to the listed label and its slug) and guards against a vacuous pass withassert!(!checks.is_empty(), …). Confirmed it fails today for the stated reason:builtin_watchermatches only the exact internal id, and none of the six watchers' display names or naive slugs equal their real id.assert_engine_interpreter's two assertions (actually runningpythonand waiting for the planted stub's own output, plus a computed$((6*7))marker gated on$VIRTUAL_ENV/bin/activate) are independently load-bearing, so a hint-text-only fix genuinely cannot flipengine-shell-02— matches the PR's own stated mutation test.screen_logical_lines' soft-wrap rejoining correctly mirrorsvt100'srow_wrappedsemantics (only concatenates when the previous row was actually filled and overflowed), reusing the same crate API as the file's existingscreen_text.
Deployment notes
None — test-only change confined to tests/e2e-cucumber/**; no production crate, API, migration, or configuration is affected.
What this covered
DEGRADED REVIEW: both attempts to fan out to independent sub-reviewers hit a concurrency limit from unrelated activity sharing this environment, so this ran single-context rather than the independent-worker split the method calls for. Recommend re-running with fan-out available. To compensate, every factual claim the new tests make was independently traced against the real production source rather than taken on faith from the tests' own comments.
Read completely at head 530f779f, diffed against origin/main merge-base 962ac0c0 (6 files, +414/-15): expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs. Cross-checked against the production code the new scenarios exercise: engines/vllm/src/install.rs, crates/rocm-core/src/runtime.rs, apps/rocm/src/engines_cmd.rs, apps/rocm/src/automations.rs, crates/rocm-core/src/automation.rs.
CI: every check on the blocking CI workflow passes at this head (build/test, clippy, the mock E2E tests job and its consolidated report, commit sign-off, license/doc checks). Failures are confined to the E2E self-hosted workflow's GPU hardware lanes, which this project's own docs state are non-blocking; one self-hosted lane was cancelled. None of this is held against the change. Mergeability was additionally checked live against the current main tip directly (not just the PR's recorded base) — see Blocking.
This PR has a long prior review history at earlier commits; that history was read only after independent findings here were reached, to reconcile rather than to seed conclusions. Several earlier blocking findings from that history (an interpreter step that could hang the whole run; an identifier-matching gap for a command id:-shaped fix) were re-verified here as fixed in the current diff, by reading the code directly rather than trusting that history.
Did not run: a test suite (no Rust toolchain available here) — the two required test-strength verdicts were argued by tracing each assertion against the real control flow it targets, not by executing the mutation tests described in the PR's own test plan. Prior-review-thread content was not used to generate findings, only for reconciliation afterward.
🤖 Automated by an AI agent
| serve_timeout_secs = 90 | ||
| flaky = true | ||
|
|
||
| # -- README-walkthrough defects (EAI-8047, EAI-8049) -- |
There was a problem hiding this comment.
Blocking: this branch is not currently mergeable into main. GitHub reports mergeable: CONFLICTING / mergeStateStatus: DIRTY for this exact head. Independently reproduced with git merge-tree $(git merge-base origin/main pr/292) origin/main pr/292: the only conflict is a textual clash right here, where two independent appends landed near the same spot in this file — main has added further xfail rows in the time since this branch's last sync with it (the branch's merge-base with the current main tip is 962ac0c0, from 2026-11-04; main is ~168 commits ahead of that point as of this review). Every other changed file merges cleanly.
Not a code defect — the fix is mechanical: merge/rebase onto the current main tip, keep both appended blocks, and push.
| } | ||
|
|
||
| #[given("a machine with no background checks turned on")] | ||
| async fn no_background_checks(world: &mut E2eWorld) { |
There was a problem hiding this comment.
Non-blocking: this step duplicates an existing one. no_background_checks calls only suppress_daemon_spawn(world) — exactly the same body as fresh_config in config_steps.rs (#[given("a fresh CLI configuration")]). Not wrong (a scenario-appropriate Given phrasing is reasonable even when two step names share an implementation), but it's a literal duplicate rather than a thin wrapper reusing the existing step text — worth a one-line comment noting the deliberate duplication, or reusing the existing phrasing, so the next reader doesn't wonder whether the two are meant to diverge later.
Summary
Test-only. Pins the correct behaviour for two open defects, in the same shape as #216: one behavioural scenario per defect asserting the CORRECT behaviour, registered as an
xfailinexpectations.tomlciting its EAI id. The mock lane stays green, and each row goes stale (XPASS) the day its bug is fixed. No product behaviour changed.This batch originally pinned four defects. Two were fixed on
mainwhile it was open and are dropped here in favour ofmain's own regression scenarios: EAI-8046 (piped chat prompt, fixed by #339) and EAI-8048 (dash --replayvalidation, fixed by #331).What it pins
automations-13) - checks listed byrocm automations listcannot be enabled from what the listing shows: it prints display names ("Server recovery") whileenableaccepts only the internal id (server-recover). The scenario derives each check's identifier from the listing, as a user would, and assertsautomations enable <id>succeeds. The ids are deliberately not hard-coded, since that would pass against a listing that exposes none. It recognises an id in four shapes: a detail line keyedid:/identifier:/slug:or any key ending inid(command id: server-recover), or a bracketed or parenthesised token on the header (Server recovery [server-recover],Server recovery (server-recover)). A fix that prints it some other way leaves the row xfailed until the parser learns that shape.engine-shell-02) - the activation hint printed byrocm engines shellnames a path that does not exist: the vLLM install response reportsenv_pathas the directory holding the command, and the hint appendsbin/activateagain. The scenario plants a manifest with that bin-shapedenv_pathas a stand-in for the install response (no install writes such a manifest to disk), runs the shell under a PTY, and asserts the hint names a file that exists and that runningpythonin the shell runs the engine's interpreter. The second assertion is there because the same doubled path also breaksPATHandVIRTUAL_ENV, so a fix to the hint text alone does not flip the row. A fix inengines/vllmthat reports the environment root closes the ticket but leaves this fixture unchanged, so the scenario and its row must then be rewritten or deleted together.Notes
automations_stepslisting parser, andscreen_logical_lineson the PTY driver so a long path that soft-wraps at 80 columns is read back whole.when = {}(platform-independent), each with a comment giving its scope and removal condition.Test plan
e2emock job green: pass-or-xfail, no XPASS.feature_naming4/4.