Skip to content

test(e2e): pin two README-walkthrough defects (EAI-8047, EAI-8049) - #292

Open
fredespi wants to merge 24 commits into
mainfrom
e2e-pin-the-contracts-for-the-second
Open

fredespi wants to merge 24 commits into
mainfrom
e2e-pin-the-contracts-for-the-second

Conversation

@fredespi

@fredespi fredespi commented Aug 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Test-only. Pins the correct behaviour for two open defects, in the same shape as #216: one behavioural scenario per defect asserting the CORRECT behaviour, registered as an xfail in expectations.toml citing its EAI id. The mock lane stays green, and each row goes stale (XPASS) the day its bug is fixed. No product behaviour changed.

This batch originally pinned four defects. Two were fixed on main while it was open and are dropped here in favour of main's own regression scenarios: EAI-8046 (piped chat prompt, fixed by #339) and EAI-8048 (dash --replay validation, fixed by #331).

What it pins

  • EAI-8047 (automations-13) - checks listed by rocm automations list cannot be enabled from what the listing shows: it prints display names ("Server recovery") while enable accepts only the internal id (server-recover). The scenario derives each check's identifier from the listing, as a user would, and asserts automations enable <id> succeeds. The ids are deliberately not hard-coded, since that would pass against a listing that exposes none. It recognises an id in four shapes: a detail line keyed id: / identifier: / slug: or any key ending in id (command id: server-recover), or a bracketed or parenthesised token on the header (Server recovery [server-recover], Server recovery (server-recover)). A fix that prints it some other way leaves the row xfailed until the parser learns that shape.
  • EAI-8049 (engine-shell-02) - the activation hint printed by rocm engines shell names a path that does not exist: the vLLM install response reports env_path as the directory holding the command, and the hint appends bin/activate again. The scenario plants a manifest with that bin-shaped env_path as a stand-in for the install response (no install writes such a manifest to disk), runs the shell under a PTY, and asserts the hint names a file that exists and that running python in the shell runs the engine's interpreter. The second assertion is there because the same doubled path also breaks PATH and VIRTUAL_ENV, so a fix to the hint text alone does not flip the row. A fix in engines/vllm that reports the environment root closes the ticket but leaves this fixture unchanged, so the scenario and its row must then be rewritten or deleted together.

Notes

  • Harness additions are confined to the e2e-cucumber test crate: an automations_steps listing parser, and screen_logical_lines on the PTY driver so a long path that soft-wraps at 80 columns is read back whole.
  • Both rows are when = {} (platform-independent), each with a comment giving its scope and removal condition.

Test plan

  • Blocking e2e mock job green: pass-or-xfail, no XPASS.
  • Native-Linux container: both rows xfail for their stated reason, 0 XPASS, 0 stale; feature_naming 4/4.
  • Applied a hint-only fix to the product as a mutation: the hint step passes and the interpreter step fails, so the row stays xfailed.
  • No GPU dispatch required - both run on the mock lane.

@fredespi
fredespi requested a review from a team as a code owner August 20, 2026 12:22
@fredespi
fredespi requested a review from rominf August 20, 2026 12:22
Add one behavioural scenario per defect, each asserting the CORRECT
behaviour and registered as an xfail in expectations.toml citing its EAI
id, so the mock lane stays green now and the row goes stale (XPASS) the
day the bug is fixed. No product behaviour changed.

- EAI-8046: `rocm chat` ignores a piped prompt (reads no stdin, sends no
  request, exits 0 with a status summary). New run_rocm_with_stdin helper
  since Command::output gives the child a null stdin; the load-bearing
  assertion is that the mock received the prompt, not stdout.
- EAI-8047: automations list publishes no usable identifier, so the
  checks it lists cannot be enabled from what it shows. The scenario
  derives each id from the listing rather than hard-coding the real ids.
- EAI-8048: `dash --replay <missing>` opens the interactive view instead
  of refusing; the step imposes its own deadline so the known hang xfails
  promptly rather than burning the CI budget.
- EAI-8049: the engine shell's activation hint names a path that does not
  exist, because the recorded env_path already includes the interpreter
  directory. Phrased as path-exists so the row goes stale on any fix.

All four re-confirmed in a plain Linux container against this branch's
binary before writing the scenarios.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/expectations.toml
#	tests/e2e-cucumber/tests/e2e.rs
Address pre-PR review: each of the three step assertions pinned the bug's
present symptom rather than the behaviour that must hold, so a valid fix
could stay xfailed and the row would never go stale.

- EAI-8047: parse the identifier the listing explicitly exposes (inline
  [id]/(id) on the header, or an `id:`/`identifier:` detail line), keeping
  the display-name slug only as the current-broken-output fallback. A fix
  that publishes the id now flips the row to XPASS.
- EAI-8048: also assert the interactive view never opened, not just a
  prompt non-zero exit. Adds a latched entered_alternate_screen() to the
  PTY driver so an "open the TUI, then exit non-zero" impl no longer
  satisfies the contract.
- EAI-8046: assert the user-role message content rather than messages[0],
  so a fix that prepends a system message still XPASSes.

Mock lane unchanged: still 6 xfail / 0 XPASS; each scenario still fails on
its assertion step alone.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Round-2 review: the latch checked the parser's alternate_screen() state
only after processing a whole PTY read, so a chunk carrying both the enter
(?1049h) and leave (?1049l) sequences left it false; and wait_for_any_exit
returned before the reader drained, so a still-buffered enter could be
missed. Either produced a false negative that would let a fixed dashboard
open the TUI and still pass the "must not open the interactive view" half.

- Detect the ?1049h enter sequence in the RAW byte stream, with a
  one-short-of-marker carry-over so a boundary-split sequence is still
  found. Independent of what follows it in the same read.
- Drain the reader (bounded by DRAIN_TIMEOUT) after the child exits, before
  the caller reads the latch — same post-exit drain wait_for_screen uses.

Mock lane unchanged: 6 xfail / 0 XPASS.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi force-pushed the e2e-pin-the-contracts-for-the-second branch from 450d4c8 to dd733fe Compare August 20, 2026 12:58
The CI Clippy (e2e harness + steps) job runs clippy -p e2e-cucumber --test e2e -D warnings, which the local container gate did not (its clippy step excludes e2e-cucumber). Four lints surfaced:

- future-not-send: drain_reader took &self, and a shared &TuiSession held across .await is not Send (the struct holds !Sync PTY handles), poisoning every step awaiting it. Take &mut self like the sibling waits.
- collapsible-if, or-fun-call (automations_steps) and map-unwrap-or (engines_steps): mechanical rewrites, no behaviour change.

Verified with the exact CI commands in a Linux container: both clippy invocations clean, e2e mock lane unchanged (6 xfail / 0 XPASS).

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/features/automations.feature
#	tests/e2e-cucumber/tests/e2e/automations_steps.rs
Upstream (EAI-8072) independently added a run_rocm_with_stdin helper that also
takes an env slice, and the merge left both definitions in e2e.rs (E0428), with
my chat step calling the 3-arg form (E0061).

Keep upstream's more general helper and delete mine; the chat stdin step now
passes an empty env slice. Behaviour is identical - both pipe stdin and close
the handle before waiting, so a command reading to EOF still terminates.

Verified with the exact CI commands in a Linux container: both clippy
invocations clean, e2e mock lane 6 xfail / 0 XPASS.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

@michaelroy-amd michaelroy-amd left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 1452ee174a87cf05693284b32a8b16fce0cc094a. The four scenarios pin the intended user-visible contracts rather than implementation details: piped chat reaches the model, listed automation identifiers are usable, a missing replay file fails without opening the TUI, and the engine-shell activation path exists. Each expectation is scoped to its ticket and removal condition; helper changes have bounded PTY drains and preserve command environment isolation. All 17 required checks pass, including the mock E2E gate and DCO/signoff. No blocking finding. Approving.

@fredespi
fredespi added this pull request to the merge queue Aug 27, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Aug 27, 2026
@rominf

rominf commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

I read through this PR's diff (scenarios, step definitions, expectations.toml, and the tui_driver additions) and traced each of the four claimed defects against the actual source:

  • EAI-8046: confirmed. Command::Chat in apps/rocm/src/main.rs only reads stdin via the interactive dash path when interactive_terminal() is true; with a piped, non-interactive run and no --prompt, it falls straight to render_chat_text, which never touches stdin or sends a request. That matches the --prompt help text's claim ("reads from standard input... when omitted") and the EXAMPLES block, which this scenario correctly proves false today.
  • EAI-8047: the derivation logic in the new automations_steps.rs (parsing the automations list output for an id inline in brackets/parens or on a detail line, falling back to slugifying the display name) looks correct against render_automations_text/watcher_plain_name in main.rs, and won't silently pass once a fix publishes a real id.
  • EAI-8048: confirmed dash::run in apps/rocm/src/dash.rs doesn't validate the --replay path before entering the interactive event loop. The new wait_for_any_exit/entered_alternate_screen latch in tui_driver.rs is a sound way to assert "refused before opening the TUI" rather than just "exited non-zero," and the byte-level alternate-screen-enter detection (with a carry buffer for reads split across the escape sequence) is correctly bounded.
  • EAI-8049: confirmed by reading resolve_engine_env plus runtime_python_activation_hint/runtime_python_env_bin_dir in rocm-core, and the vLLM install's env_path = command.parent() in engines/vllm/src/lib.rs. The doubled bin/bin/activate path is real, and phrasing the assertion as "the hint names a file that exists" (rather than "not doubled") is the right way to keep the scenario meaningful after any valid fix.

I also checked the expectations.toml additions against the file's own grammar and existing rows, and the four new @id: tags line up 1:1 between the feature files and the xfail table.

One thing worth flagging, though it's not a code issue: GitHub currently reports this PR as having a merge conflict. That's because dash.feature's scenario 10 slot was independently filled by a different, later-merged PR (the "launcher shows live serving instance" scenario) after this branch's base commit — diffed against the actual merge-base with main, this PR's own changes apply cleanly and are a pure addition. It'll need a rebase to reconcile scenario numbering with main, but that's a mechanical rebase, not a defect in the tests or steps themselves.

Nothing else stood out. This looks safe to approve after a rebase and once CI (mock e2e lane, clippy) is green — this is a diff-only read, not a substitute for CI or a maintainer's own pass.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 1452ee1

Summary

Test-only PR pinning four README-walkthrough defects as cucumber scenarios that assert the correct behaviour, plus four when = {} expected-failure rows and the TUI-driver support they need (latched alternate-screen detection, logical-line reads, a non-zero-exit wait). Approve. Verified: none of the four scenarios can pass against a binary lacking the behaviour — chat waits for an actual completion request on the mock (wait_for_chat_request) rather than for output, automations derives each check's identifier from the listing instead of hard-coding the real ids, dash asserts both a prompt non-zero exit and that the alternate screen was never entered, and engine-shell asserts the printed hint names an existing file against a fixture planted in the shape vLLM actually records (command.parent()); all four run on ci.yml's e2e job (ubuntu-latest, mock lane), documented in-repo as a required check, none carrying @requires-gpu/@nightly/@lifecycle/@merge-queue, and @requires-os:linux matches that runner; no expected-failure row is stale — I read the product source at the base (current main) for all four bugs and each is still present, with git log HEAD..prw-base confirming no main-only commit touched the chat stdin arm, render_automations_text, the dash replay path, or runtime_python_activation_hint/vLLM's env_path; no dead rows (every expectations.toml key resolves to an existing @id: tag) and no dead helpers (all eleven new/changed symbols including run_rocm_with_stdin, slug_of, drain_reader have call sites), every new step phrase has a definition, and a non-flaky XPASS exits 1 so each row self-reports the day its bug is fixed. Not run per policy: the e2e suite itself (CI reports 25 passing, 0 failing for this head). Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • tests/e2e-cucumber/tests/e2e/dash_steps.rs:250 — the exit deadline is a hard-coded Duration::from_secs(2), while every sibling TUI wait in this suite uses 30s (SCREEN_TIMEOUT / default_timeout()); harmless today (any failure satisfies the xfail), but once the dash bug is fixed and the row is deleted, a >2s cold binary start on a loaded shared runner turns the required lane red.
  • tests/e2e-cucumber/tests/e2e/chat_steps.rs:141 — assert_eq!(user_content, "Hello from standard input") requires the trailing newline to be trimmed; a fix that forwards stdin verbatim would keep the row xfailed rather than letting it go stale, which is the failure mode the file's own comments set out to avoid.
  • tests/e2e-cucumber/tests/e2e/automations_steps.rs:78 — exposed_identifier recognises [id], (id), id:/identifier:; a fix publishing the id in another shape (a trailing column, an em-dash, slug:) leaves the row permanently xfailed. Same function: a display name containing a parenthesised single word ("Server recovery (beta)") would be misread as an identifier.
  • tests/e2e-cucumber/tests/e2e/engines_steps.rs:161 — the hint is parsed by stripping a literal source prefix only; a quoted path or a .-prefixed hint would fail the existence check for a reason unrelated to the product.
  • Branch is 38 commits behind main and last absorbed main at 16252c2; the green CI was measured against that base. The stale-row risk does not materialise (verified above), but a rebase before merge would re-measure against current main.

…-for-the-second

Absorbs 38 commits of main and reconciles the four pinned scenarios with the
scenario-numbering convention main adopted repo-wide in the meantime:

- dash.feature: main's launcher scenario took the dash-10 slot, so the replay
  rejection scenario becomes dash-11 (both kept).
- chat.feature / automations.feature / engine_shell.feature: renumbered onto the
  <feature>-NN form (chat-09, automations-04, engine-shell-02).

main independently added its own automations.feature and automations_steps.rs;
the listing-discoverability scenario folds in alongside them, with no duplicate
step phrases and no dead expectation rows.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Addresses the non-blocking review points on #292. Each one was a way a CORRECT
fix could leave the expectation row xfailed instead of going stale, which is the
failure mode these scenarios exist to avoid.

- chat: compare the user message trimmed. The step pipes a trailing newline, so
  forwarding stdin verbatim -- a perfectly good fix -- would otherwise not XPASS.
- automations: recognise the identifier in the shapes a fix might publish it --
  a detail line (id/identifier/slug/key/name), bracketed, parenthesised,
  dash-separated, or as a trailing column. A parenthesised or dash/column token
  must look like an identifier (lowercase with a separator, as every real
  watcher id is), so a display-name qualifier such as "Server recovery (beta)"
  is not mistaken for one. The slug-of-display-name fallback is unchanged, so
  today's listing still fails the contract.
- engine shell: accept ". <path>" and a quoted path as well as "source <path>",
  so the existence check fails only when the CLI really named a missing file.
- dash: replace the bare 2s replay deadline with a documented
  REPLAY_REJECTION_TIMEOUT. Still bounded (today's behaviour is a hang, and that
  bound is paid every run until it is fixed), but with enough headroom that a
  cold start on a loaded runner will not flake once the row is deleted.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi

Copy link
Copy Markdown
Collaborator Author

Thanks both — all five non-blocking points are addressed in 3490bf6f, on top of a main merge in 832e1064.

Rebase / 38 commits behind (both reviewers): merged origin/main rather than rebasing, so the branch stays fast-forwardable and no force-push is needed. The conflict was narrower than it looked but not purely mechanical: main adopted the <feature>-NN scenario-numbering convention repo-wide in the meantime, and its launcher scenario had claimed the dash-10 slot. Both scenarios are kept — ours is now dash-11 — and the other three were renumbered onto the new convention (chat-09, automations-04, engine-shell-02). main had also independently added its own automations.feature and automations_steps.rs; the listing-discoverability scenario folds in alongside those with no duplicate step phrases.

The four durability notes — each was a way a correct fix could leave the row xfailed instead of going stale, which is precisely what these scenarios exist to prevent, so all four are fixed:

  • chat_steps.rs — compare the user message trimmed. The step pipes a trailing newline, so forwarding stdin verbatim would otherwise not XPASS.
  • automations_steps.rs — the identifier is now recognised as a detail line (id/identifier/slug/key/name), bracketed, parenthesised, dash-separated, or as a trailing column. To avoid the false positive you flagged, a parenthesised/dash/column token must be identifier-shaped (lowercase with a separator, as every real watcher id is), so Server recovery (beta) is not read as an id. The slug-of-display-name fallback is unchanged, so today's listing still fails the contract.
  • engines_steps.rs — accepts . <path> and a quoted path as well as source <path>.
  • dash_steps.rs — the bare Duration::from_secs(2) is now a documented REPLAY_REJECTION_TIMEOUT. I kept it bounded rather than moving to the 30s default_timeout(): today's behaviour is a hang, so this bound is paid on every mock-lane run until the bug is fixed. It now has ample headroom over a cold start, which was the flake risk you identified. Happy to align it with the siblings instead if you'd prefer.

Verification (Linux container, native-Linux host matching CI): clippy -D warnings clean, e2e-cucumber lib 113/113, and the mock lane reconciles 6 xfail, 0 XPASS, 0 stale — so all four rows still fail for their stated reason and none went stale.

Two failures in that run are pre-existing environment gaps, not regressions, and I verified each by running the identical gate against a pristine origin/main checkout in the same container: the 10 rocm-engine-lemonade child_runtime_dir tests (identical 87 passed / 10 failed on stock main; they trip on the container work dir permitting cross-UID rename), and engine-shell-marks-the-prompt (also fails on stock main there — the container shell prompt ends in # rather than $). Both are green on CI.

@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 3490bf6

Summary

Test-only PR adding four cucumber scenarios (plus supporting steps and a PTY-driver capability) that pin correct behaviour for four open CLI defects as documented xfail rows. Verdict: Approve — all five durability points are addressed, the merge is clean, and all four rows still fail for their stated reason. Verified: the five fixes (trimmed chat comparison, multi-shape identifier recognition, extra hint prefixes, the named replay bound, the merge) all present in 3490bf6; merge integrity checked four ways — every expectations row resolves to an existing @id, no duplicate step phrases or regex/literal ambiguity across the 358 step definitions, every Gherkin step in every feature file resolves, scenario ids and indexes unique and sequential post-renumber (cargo test --test feature_naming: 4/4 pass); each scenario re-derived against the product source and confirmed discriminating; no prompt-injection content found in the diff. Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Discrimination check (each row fails only against a binary lacking the behaviour)

  • chat-09 — the Given steps start the mock and plant the service record but issue no chat POST, and rocm chat --provider local on this path emits only the readiness GET. So wait_for_chat_request times out today; it can only be satisfied by a binary that actually forwards stdin. The .trim() and the "last user-role message" lookup widen what counts as a correct fix without weakening it. Discriminating.
  • automations-04 — apps/rocm/src/main.rs:16002 prints {plain name} ({on|off}) with 4-space detail lines and publishes no id; watcher_plain_name maps e.g. server-recover → "Server recovery", so the slug fallback (server-recovery) mismatches for all six watchers and enable returns rc=1. exposed_identifier returns None against today's output, and the "guard the guard" non-empty assertion stops a parse failure from vacuously passing. Discriminating.
  • dash-11 — today the command hangs, so wait_for_any_exit errors out; passing needs both a prompt non-zero exit and the latched alternate-screen flag staying false. The raw-byte \x1b[?1049h scan with a carry buffer correctly survives an enter+leave in one read. Discriminating.
  • engine-shell-02 — runtime_python_activation_hint appends bin/activate to the recorded env_path, and engines/vllm/src/lib.rs:480 records command.parent(), so the hint names <env>/bin/bin/activate while the fixture plants <env>/bin/activate. Discriminating.

On the 2s → 30s question: keep the dedicated bound, do not align

Keep REPLAY_REJECTION_TIMEOUT (tests/e2e-cucumber/tests/e2e/dash_steps.rs:22) as its own value. The 30s siblings and this one are not the same kind of wait, so the same number would mean different things:

  • Sibling waits are success waits — the budget is only spent when something has already gone wrong, so generosity is free in the normal case.
  • This is a failure-detection wait on a path that hangs by construction. The budget is spent in full, on every mock-lane run, for as long as the bug is open. Aligning to 30s triples a standing CI cost and buys no additional safety, because there is nothing slow to wait for — the contract is "refuse before opening the TUI".
  • After the fix it reverts to a success wait, and 10s is well over an order of magnitude more than a cold binary start plus PTY spawn and immediate refusal needs on a loaded shared runner — so the row can be deleted without leaving a flake behind. The original bare 2s was the genuinely risky value; raising it to 10s was the right correction, and naming it with the rationale inline is the right way to keep it from being "fixed" later by someone who only sees the inconsistency.

Non-blocking

  • tests/e2e-cucumber/tests/e2e/engines_steps.rs:91 — the fixture hard-codes the vLLM-shaped manifest, so a fix applied at the producer layer (making the install record the env root) leaves the planted manifest unchanged and the row xfailed forever. Defensible (already-installed environments keep the old manifest, so a producer-only fix genuinely doesn't repair them), but that migration requirement isn't what the expectation row's reason states — worth a sentence there.
  • tests/e2e-cucumber/tests/e2e/automations_steps.rs:96 — the detail-line key list includes name:/key: and matches on block order, so a fix printing name: Server recovery above an id: line would capture the display name. Consider trying id/identifier/slug first, or applying is_identifier_shaped to the name:/key: hits too.
  • tests/e2e-cucumber/tests/e2e/chat_steps.rs:151 — bare Duration::from_secs(2) where the sibling dash assertions use default_timeout(). The CLI has already exited by the time this Then runs, so it is a pure 2s tax on every mock-lane run today; either name it like the dash one or use the shared default.
  • tests/e2e-cucumber/tests/e2e/automations_steps.rs:61 — is_identifier_shaped requires a -/_, so a published id without a separator would go unrecognised in the parenthesised/dash/column shapes. Low risk (every real watcher id has one) and documented, but the bracketed and detail-line shapes are the only unconditional escapes.
  • tests/e2e-cucumber/tests/e2e/chat_steps.rs:151 — uses wait_for_chat_request rather than wait_for_chat_request_where. The mock module's own docs warn that a local-endpoint readiness probe is itself a chat request; rocm chat doesn't issue one on this path today, but if it ever does, this row fails for the wrong reason and never goes stale. The dash managed-chat step already guards exactly this.

…(PR review)

Second round of pr-review-watcher notes, all about a row staying xfailed
or failing for the wrong reason after a valid fix:

- chat-09 now waits for the request that carries the piped prompt rather
  than for whichever chat request lands first, so a future readiness probe
  on this path cannot satisfy the wait. Its bound is named and the reason
  for keeping it short is recorded at the constant.
- automations-04 sweeps the unambiguous identifier keys (id/identifier/
  slug) across the whole block before the ambiguous ones (key/name), which
  must additionally look like an identifier, so a listing that prints a
  display name above its id no longer yields the name.
- The engine-shell expectation reason now states that the fixture pins the
  manifest shape an install already recorded, so the row clears on the
  hint composition rather than on what a fresh install writes.

Verified in the Linux container: 6 xfail, 0 XPASS, 0 stale.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@siloteemu

Copy link
Copy Markdown
Contributor

🔴 Automated review · pr-review-watcher · 2c4b600

Summary

Test-only PR adding four e2e scenarios (chat-09, automations-04, dash-11, engine-shell-02) that pin four README-walkthrough defects, each with an always-on expected-failure row; the two new commits address the prior round's open non-blocking items and nothing else. Approve. Verified: merge/renumber integrity is clean (all 16 expectation rows name existing scenarios, the only repeated table key is the documented OR-condition pair, 122 @id tags all unique, per-file indexes sequential — automations-01..04, chat-01..09, dash-01..11, engine-shell-01..02 — 359 step definitions with no ambiguous pair and all 610 Gherkin steps resolving to exactly one definition; cargo test -p e2e-cucumber --test feature_naming passes 4/4); staleness is clean — I traced all four bugs in the current base myself and each is still live for its stated reason (Command::Chat with no --prompt and non-interactive stdin calls render_chat_text and never reads stdin; render_automations_text prints only watcher_plain_name with no id field, and builtin_watcher is exact-match, so the slug fallback server-recovery ≠ server-recover; dash::run passes replay through with no readability check before the TUI opens; runtime_python_activation_hint appends bin/activate to a vLLM-recorded env_path that is already command.parent()); on the revert question every scenario and every assertion is discriminating in both directions — each fails today for the stated reason and would go stale on a correct fix, with no trivially-true assertion, no swallowed failure, no Then whose work is done in the When, and the one vacuous-pass risk explicitly guarded (assert!(!checks.is_empty()) in automations-04); and all three open items are genuinely fixed — PIPED_PROMPT_TIMEOUT and REPLAY_REJECTION_TIMEOUT are now named constants with recorded rationale, exposed_identifier sweeps the unambiguous id/identifier/slug keys across the whole block before the ambiguous key/name (which must additionally pass is_identifier_shaped), and wait_for_chat_request_where matches on the last user message containing the piped prompt so a "Say ok." readiness probe cannot satisfy it. Blocking: 0 · Non-blocking: 3.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • tests/e2e-cucumber/tests/e2e/dash_steps.rs:29 — the 10s REPLAY_REJECTION_TIMEOUT is paid in full on every mock-lane run while the replay-rejection bug stays open; the tradeoff is documented at the constant, just worth tracking if lane runtime becomes a concern.
  • tests/e2e-cucumber/tests/e2e/automations_steps.rs:84-90 — is_identifier_shaped requires a -/_, so a fix that published a separator-less id in an ambiguous position (Server recovery (gpumetrics)) would leave the row xfailed; the narrowing is deliberate and documented, and the unambiguous id:/[bracketed] shapes are unaffected.
  • tests/e2e-cucumber/expectations.toml — the new rows carry internal tracker references, matching the file's own documented grammar and every pre-existing row on the base; flagging only because this is a public repository and the convention is repo-wide rather than introduced here.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 2c4b6002: no blocking issues, third consecutive clean pass.

Verified this round: merge and renumber integrity after absorbing the base's repo-wide scenario renumbering — every expectation row names a scenario that exists, all scenario tags unique, per-file indexes sequential, no ambiguous or duplicate step definitions, and every step resolves. Staleness checked directly: all four pinned defects were traced in the current base and each is still live for its stated reason, so none of the rows has quietly become an unexpected pass.

Each scenario discriminates in both directions — it fails today for the reason stated, and would go stale on a correct fix rather than silently continuing to pass. Three minor notes in the review comment; none needs action before merge.

…-for-the-second

Resolves a semantic conflict in tui_driver.rs: main (#354) and this branch
independently added the same capability under different names — a poll-until-
exit that returns the raw code instead of asserting zero. Converged on main's
wait_for_exit_code, since it is already published and has a caller there
(diagnose-16), and folded in this branch's bounded drain_reader() wait, which
the dash-11 alternate-screen latch needs to see the child's final bytes.
dash-11 now calls wait_for_exit_code.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi
fredespi added this pull request to the merge queue Sep 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 11, 2026
@fredespi
fredespi added this pull request to the merge queue Sep 11, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 11, 2026
@fredespi
fredespi added this pull request to the merge queue Sep 11, 2026
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
…ine's interpreter

The doubled env_path also sets a wrong VIRTUAL_ENV and a nonexistent bin/bin on
PATH, so a fix to the hint text alone would flip this row while the shell still
runs the wrong python. Reuse the existing interpreter step so the row clears
only once the user's actual problem is fixed.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi

Copy link
Copy Markdown
Collaborator Author

Took your recommendation: pushed 485d6974. engine-shell-02 now also asserts "the engine environment's interpreter is the one that runs", so the row flips only once the shell actually runs the engine's python, not when the hint text alone is fixed. The feature comment says why. Also merged current main (98b6b03c).

Native-Linux container on the merged tree: 4 xfail, 0 XPASS, 0 stale, 0 unexpected; feature_naming 4/4. The interpreter step is only reached after the hint step, which still fails today, so the scenario xfails for the stated reason.

@fredespi
fredespi requested a review from rominf October 26, 2026 19:22
@fredespi

Copy link
Copy Markdown
Collaborator Author

#292 (comment)

Pushed 485d697: engine-shell-02 now also asserts the engine interpreter runs, as you recommended; main merged, Linux gate 4 xfail / 0 XPASS / 0 stale.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 485d697

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Round outcome: this round files a change request, for one objection only: the first item under Blocking. The engine-shell-02 interpreter step that 485d6974 added to stop a hint-only fix from flipping the row cannot fail. A mutation shows a hint-only fix still flips it. The head commit's message, which lands on main in the squash, says the opposite. The other two Blocking items are published as found, but neither gates. The internal-environment comment was already on the previous head, and the name is already public in this repo's workflows. The description mismatch is description-only. Items are marked new (first raised this round) or standing (raised in an earlier round).

The failing "E2E tests (MI300X, current)" check does not bear on this. Main's latest MI300X runs fail on the same scenarios (chat-06, serve-01, serve-02, serve-07 with a Lemonade 404, runtime-02, therock-next-09). This head adds only its own two expected xfails, automations-06 and engine-shell-02. W7900 has not run yet.

Review — needs work

Full review of the whole change.
Implements the regression pins for EAI-8047 and EAI-8049, both still open, and fixes neither defect. This matches the description's "test-only" framing.

Blocking

  • new — engine-shell-02's interpreter step can never fail, so a hint-only fix flips the row while the shell still runs the wrong python — tests/e2e-cucumber/features/engine_shell.feature:39-42, tests/e2e-cucumber/tests/e2e/engines_steps.rs:144-161
    The feature comment, and the head commit, say this step is there so that "a fix to the hint text alone" cannot flip the row. It cannot do that job. assert_engine_interpreter sends command -v python and then waits for <env>/bin/python anywhere on screen. engine_shell prints that same string in its banner ( python: {python_executable}, apps/rocm/src/engines_cmd.rs) before bash starts. The wait therefore returns at once, whatever command -v prints.
    Confirmed by mutation. I changed only the hint (hint_root = env_path.parent() when env_path ends in bin) and left PATH=<env>/bin/bin:… and VIRTUAL_ENV wrong. With cargo xtask e2e -- -i features/engine_shell.feature, both scenarios passed and the run reported 1 XPASS (… 1 stale) for engine-shell-activation-hint-is-usable.
    The step was already this weak in engine-shell-01. What this change adds is a scenario whose stated contract depends on the step. That fix is also the one the EAI-8049 ticket proposes ("build the hint from the environment root"), so it is the likely path.
    Confidence 100 · logic · Fix: wait for output that only the command can produce. For example, run python and wait for planted-engine-python (the planted interpreter's output, which the banner never prints), or echo a sentinel such as py=$(command -v python) and wait for py=<expected>.

  • new (does not gate) — A comment names an internal cluster environment — tests/e2e-cucumber/expectations.toml:185
    The comment says "walking the CLI on an MI300X app-dev pod". AGENTS.md §2 forbids cluster names in code comments on upstream surfaces. The name already appears in a few workflow and docs files on the base branch. That reduces the extra exposure but does not exempt a new occurrence.
    Confidence 70 · mechanical · Fix: "on an MI300X host".

  • standing (description-only, does not gate) — The PR description misstates the change — PR description, "What it pins"

    • It credits automations-05 with deriving identifiers from the listing. The new scenario is automations-06. automations-05 is the existing binary-units scenario.
    • It says the parser "recognises an id in any shape a fix might plausibly print it in". exposed_identifier (automations_steps.rs:88) recognises only id:/identifier:/slug: detail lines and a trailing [id] on the header. Its own doc comment says "Any other shape a fix might pick leaves the row xfailed".

    Later readers trust the description as the map of the change.
    Confidence 85 · mechanical · Fix: narrow the description. Name automations-06 and state the two recognised shapes.

Non-blocking

  • standing — automations-06 stays xfailed under plausible fixes, so a fixed bug would go unnoticed — tests/e2e-cucumber/tests/e2e/automations_steps.rs:88-145
    Two plausible fixes leave the row xfailed instead of making it go stale:

    • enable accepts the display name exactly as listed. The step only ever sends the slug (server-recovery) or an exposed id, never the listed name.
    • The id is shown in parentheses beside the label, e.g. Server recovery (server-recover) (off). The ticket suggests showing the id "alongside its label". The parser would then slug the whole string.

    This fails safe: the run stays green. The cost is that the row never reports the fix.
    Confidence 75 · logic · Fix: when no identifier is exposed, try the verbatim display name before the slug, and consider recognising a parenthesised id.

  • new — "Lemonade reports the root" is inaccurate — tests/e2e-cucumber/expectations.toml:208, tests/e2e-cucumber/tests/e2e/engines_steps.rs:41
    Lemonade's install returns env_path: manifest.runtime_dir (engines/lemonade/src/lib.rs). That is the directory holding lemond/lemonade, not a Python environment root with bin/activate. The vLLM-versus-Lemonade contrast in the comment makes the defect look narrower than it is.
    Confidence 75 · mechanical · Fix: drop the Lemonade contrast, or describe what Lemonade actually reports.

  • standing — The soft-wrap branch of screen_logical_lines is not exercised on a short-TMPDIR host — tests/e2e-cucumber/tests/e2e/tui_driver.rs:374-395
    With a /tmp/rocm-e2e-XXXXXX root, the hint line is about 72 columns and does not wrap at 80. A break in the row_wrapped rejoin alone would stay green here. Only a lane with a long temp path reaches that branch.
    Confidence 70 · logic · Fix: a small unit test that feeds the vt100 parser an over-width line.

Decisions for the author

  • standing — engine-shell-02 plants a manifest shape no install writes — tradeoff
    • For: it reaches the shell-launch path without a multi-GiB install, and the comments explain the stand-in clearly.
    • Against: a vLLM-side fix (reporting the env root) closes EAI-8049 without flipping the row, so the row then needs a manual rewrite or delete. The comments already document this. Confirm that is the intended maintenance burden.

Positive signals

  • assert_listed_checks_enableable derives identifiers from the listing instead of hard-coding them, and guards against an empty parse. I confirmed that it fails today for the right reason (all six checks: unknown watcher). An id: detail line added to render_automations_text flips it to a stale XPASS.
  • The new hint step is solid. It reads logical lines and checks that the path exists, so it fails today on …/bin/bin/activate and passes once the hint names the real file.

Deployment notes

None

What this covered

  • Read: all six changed files in full, plus the product paths they exercise: engine_shell/resolve_engine_env, runtime_python_activation_hint, the vLLM and Lemonade install env_path, render_automations_text, builtin_watcher, and harness resolution and reconciliation. Range 63ecab2d…485d6974, diffed against prw-base.
  • Run, in this checkout's own target/:
    • cargo xtask e2e -- -i features/engine_shell.feature at the PR head: engine-shell-02 xfails as expected and engine-shell-01 passes.
    • Two product mutations, described above.
    • The automations feature was run only with the mutation applied. At the PR head it was run only with -n, which showed automations-06 failing for the stated reason.
  • Harness note: a -n name filter bypasses the harness's filter_run closure, because cucumber skips the closure when a name regex is set. Every xfail row then reports as an "unexpected failure", so -i was used for every reconciliation result above.
  • CI: the only failing check is "E2E tests (MI300X, current)". It fails the same way on main, apart from this PR's two expected xfails. W7900 is still queued.
  • Sign-off: all 9 commits carry a DCO sign-off.
  • Fan-out: two independent workers ran. One did a code pass over the whole change. The other did the agent-instruction, history and code-comment passes.
  • Not run:
    • The design-question fan-out (Step 7) did not run as separate workers because of the session's two-worker cap. I asked the single design question inline.
    • The prior-changes pass did not run: prior review discussion is out of bounds for this run.
  • Tickets: both named tickets were retrieved by ID; both are open.
  • Reconciliation: not reconciled against any PR discussion here.

The step waited for the interpreter's path on screen, but engine_shell prints
that path in its banner before bash starts, so the wait returned at once and a
hint-only fix still flipped engine-shell-02. Run python and wait for the planted
interpreter's own output, which the banner never prints. Verified by applying a
hint-only fix to the product: the hint step passes and the interpreter step now
fails, so the row stays xfailed.

Also drop an inaccurate Lemonade contrast and an internal host name from the
expectation comments.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi

Copy link
Copy Markdown
Collaborator Author

Verified your blocking point against the code before changing anything, and it holds. engine_shell prints python: <path> in its banner before bash starts, so waiting for the interpreter path returned at once whatever command -v python printed.

Fixed in 20fcc4a4. The step now runs python and waits for the planted interpreter's own output, which the banner never prints. I then re-ran your mutation: a product change that fixes only the hint (root-derived hint, PATH left at <env>/bin/bin). The hint step passes and the interpreter step now fails ("timed out waiting for planted-engine-python"), so the row stays xfailed. Reverted the mutation afterwards. Unmutated, in a native-Linux container: 4 xfail, 0 XPASS, 0 stale, 0 unexpected.

Also done:

  • Internal host name removed from the expectations.toml header comment ("MI300X host").
  • The inaccurate Lemonade contrast is gone from the row and the fixture doc.
  • PR description rewritten: it names automations-06 and states the two shapes the parser recognises.

Left as is:

  • Display-name-first and parenthesised-id shapes: I narrowed the parser last round on your own feedback that guessing at shapes no listing prints is speculative, so I'd rather keep it narrow and let the row tell us when a fix picks another shape.
  • Unit test for the soft-wrap rejoin: not in this PR; I'll note it for the follow-up that ports the alt-screen latch.

@fredespi
fredespi requested a review from rominf October 27, 2026 10:41
@fredespi

Copy link
Copy Markdown
Collaborator Author

Review - no blocking findings

Reviewed 63ecab2d...20fcc4a4. This test-only change pins automation identifier discoverability and engine-shell activation/interpreter selection as expected failures. The interpreter assertion now runs python and waits for output unique to the planted executable, so the shell banner cannot satisfy it. The parser limitations and the fixture retirement requirement for a producer-side fix are documented.

Looks safe to approve after final validation. This is a code-review comment, not an approval or independent runtime validation; I did not run tests. The mock E2E check is green, while hardware lanes are still pending.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 20fcc4a

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Round outcome: this round files a change request, for one objection only: the first item under Blocking (new). 20fcc4a4 makes the engine-shell interpreter step able to fail, but when it fails, the e2e run hangs instead of reporting the failure. This also hits engine-shell-01, which uses the same step. The second Blocking item is published as found but does not gate: it only bites on a partial fix (hint and PATH fixed, VIRTUAL_ENV left wrong), and the usual fix, normalising the env root, corrects all three. It is cheap to close in the same push. Items are marked new (first raised this round) or standing (raised in an earlier round).

The standing objection from the previous round is lifted from the gate: the interpreter step "can never fail". This round's blind pass did not re-find it. It also re-ran the hint-only mutation independently: the hint step passed, the interpreter step timed out on planted-engine-python, and the row stayed xfail with 0 XPASS. When the objection was then put to the reviewer, it confirmed the objection no longer holds at 20fcc4a4. That confirmation is an anchored check, because the reviewer was told what to look for. The internal-environment comment, the Lemonade wording and the wrong scenario number in the description are all gone too.

The author's reply was read. Of the items it left as is, the display-name and parenthesised-id shapes were re-found blind below as non-blocking. The soft-wrap unit test is deferred to a follow-up, as the reply says.

Review — needs work

Full review of the whole change.
Pins EAI-8047 and EAI-8049 as xfail scenarios. It does not fix either bug and does not claim to close either ticket. Both tickets were retrieved and are in Backlog.

Blocking

  • new — When the interpreter step fails, it hangs the whole e2e run instead of failing — tests/e2e-cucumber/tests/e2e/engines_steps.rs:160, tests/e2e-cucumber/tests/e2e/tui_driver.rs:1380

    • The step now types a bare python\n into the shell. If python on PATH is not the planted stub, a host with a system python starts an interactive REPL, and that REPL becomes the terminal's foreground job.
    • When the step times out, TuiSession::drop kills only the direct child (rocm). The kernel's SIGHUP then reaches the REPL's process group and not bash, so bash survives, orphaned, and keeps the PTY slave open. The reader thread never sees EOF, and handle.join() blocks forever.
    • Before this change, command -v python never started a foreground process, so a failure stayed a clean failure.
    • Reproduced: I applied a hint-only product fix as a mutation. The interpreter step failed as designed (timed out after 30s waiting for "planted-engine-python"). The e2e binary then sat in futex_do_wait for 13 minutes beside an orphaned /bin/bash --rcfile … -i, and finished only after I kill -KILLed that bash.
    • The same hang will hit engine-shell-01 the day a regression stops PATH being prepended. In CI it would turn a red scenario into a job timeout and lose every scenario after it. 20fcc4a4's message says the step "now fails". That is true only up to the panic: the run itself never completes, and the message lands on main in the squash.
    • Confidence 100 · logic · Fix: run the interpreter non-interactively, so a wrong python exits on its own. For example python </dev/null\n or python -c pass\n: the planted stub ignores its arguments and still prints its marker.
  • new (does not gate) — engine-shell-02's comment says the row flips only once the user's problem is fixed, but it flips while VIRTUAL_ENV is still wrong — tests/e2e-cucumber/features/engine_shell.feature:38-42. Commit 485d6974's message ("clears only once the user's actual problem is fixed", naming VIRTUAL_ENV) and the PR description ("the same doubled path also breaks PATH and VIRTUAL_ENV") make the same claim.

    • engine_shell sets .env("VIRTUAL_ENV", &resolved.env_path) (apps/rocm/src/engines_cmd.rs:576), and no step checks it.
    • Reproduced: I mutated engine_shell to normalise the root for the hint and for venv_bin, and left VIRTUAL_ENV as <env>/bin. The run reported 1 XPASS (0 flaky, 1 stale), so the row would be deleted with VIRTUAL_ENV still wrong.
    • The code governs here: VIRTUAL_ENV is what the shell actually exports, so the comment's claim is the side that is wrong.
    • Confidence 100 · logic · Fix: either assert VIRTUAL_ENV inside the shell (for example, echo "$VIRTUAL_ENV", then check the printed directory holds bin/activate), or narrow the comment and the description to "the hint and PATH".

Non-blocking

  • standing — A fix that makes enable accept the check names the listing prints does not flip automations-06 — tests/e2e-cucumber/tests/e2e/automations_steps.rs:143, :374

    • For each check, the step tries exactly one candidate: an identifier the listing exposes, or else a slug of the display name.
    • Reproduced: I mutated automations enable to accept the display name exactly as listed ("Server recovery"). The scenario still failed: it tried server-recovery, gpu-status-checks and so on. The row stayed xfail with 0 stale.
    • That fix satisfies the scenario's own wording ("can be turned on from what the listing shows"), and it is the one route that keeps the product's existing unit tests green (see Decisions).
    • The description discloses that id shapes the parser does not know leave the row xfailed. This is a different kind of fix, though, not a new shape.
    • Confidence 100 · logic · Fix: also try the display name verbatim, and count the check as reachable if any candidate succeeds.
  • standing — The parser does not recognise command id: <id>, or an id in parentheses beside the label — tests/e2e-cucumber/tests/e2e/automations_steps.rs:88-105

    • strip_prefix("id:") misses command id: server-recover. That is the one id format the product's own tests name (apps/rocm/src/main.rs:19292, :19316, :19454 assert !rendered.contains("command id:")), so it is the likeliest format to come back.
    • The ticket also asks for the id "alongside its label". A form like Server recovery (server-recover) (off) falls back to a slug.
    • Confidence 75 · logic · Fix: accept any … id: key, and a single whitespace-free parenthesised token before the on/off state.

Decisions for the author

  • new — automations-06 pins as "correct" a listing that the product's unit tests currently forbid — tradeoff
    • render_automations_text_* asserts the listing does not contain gpu-metrics, gpu-thermal-protect or command id: (apps/rocm/src/main.rs:19292-19294, :19316-19318, :19353, :19454). So hiding the ids was a tested, deliberate choice.
    • The ticket (EAI-8047, now keyed ROCMAI-379) asks for the opposite: show the ids, or list them in --help and in the "unknown watcher" error.
    • Either is defensible. Whoever fixes it has to delete those unit assertions or take the display-name route. The scenario or the row comment should say which is intended, so the fixer does not get contradictory signals.

Positive signals

  • The interpreter step now runs python and waits for the stub's own output, rather than waiting for the path. The banner already prints the path, so the old wait could be satisfied without the interpreter ever running. The PR's own hint-only mutation claim held when I re-ran it: the hint step passed, the interpreter step failed, and the row stayed xfail.
  • screen_logical_lines correctly rejoins soft-wrapped rows using vt100's row_wrapped, so a long path is read whole. That avoids the truncated-path false failure.

Deployment notes

None

What this covered

  • Read: all six changed files in full at HEAD, plus the product code they exercise: automations.rs, render_automations_text and its tests, engine_shell, resolve_engine_env, runtime_python_activation_hint, and the vLLM install.rs env_path. Also the harness pieces (TuiSession drop and reader, reconciliation). Range: fe1dbabec0…20fcc4a452.
  • Ran locally (each report.json mtime from this run, named scenarios executed):
    • cargo xtask e2e -- -i features/automations.feature: 6 scenarios, automations-06 xfail.
    • cargo xtask e2e -- -i features/engine_shell.feature: engine-shell-02 xfail at bin/bin/activate.
    • Three product mutations, described in the findings above.
  • CI at the head: all reported checks green. Skillscope and Sphinx were skipped. No GPU lane is listed for this test-only change.
  • Sign-off: the check script reported 10 commits checked, none missing.
  • Fan-out: one independent worker did the code pass. I did the agent-instruction pass, the history pass and the verification runs myself, so those were not independent. Step 7 design questions were answered in this context rather than by separate workers.
  • Did not run: the prior-changes pass, because earlier reviews are out of bounds for this run.
  • Not reconciled against any PR discussion here.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 20fcc4a

Review — needs work

Test-only PR pinning two README-walkthrough defects (automations-06, engine-shell-02) as xfail cucumber scenarios, plus the TUI-driver support they need. I read all six changed files at 20fcc4a4 against the main merge-base in full, and traced the product code they exercise (apps/rocm/src/engines_cmd.rs, apps/rocm/src/main.rs's render_automations_text and its unit tests). One of the two new/changed assertions in engines_steps.rs/tui_driver.rs introduces a real hang risk in test teardown; everything else is sound test design with a couple of smaller gaps.

Blocking

  • The new interpreter check can hang the whole e2e binary on failure instead of failing cleanly — tests/e2e-cucumber/tests/e2e/engines_steps.rs:160, tests/e2e-cucumber/tests/e2e/tui_driver.rs:1340-1370, tests/e2e-cucumber/tests/e2e/tui_driver.rs:1406-1441
    assert_engine_interpreter now types a bare python\n into the shell and waits for the planted stub's own output (replacing the old command -v python path check). engine_shell (apps/rocm/src/engines_cmd.rs:645) launches the user's shell via command.status(), i.e. as a child process of rocm, not of the test harness directly — the harness's tracked self.child is only the rocm process. If the shell resolves a real, non-stub python (wrong PATH composition, or any regression in the fix this scenario is meant to pin), that python becomes bash's interactive foreground job and keeps running past the 30s wait_for_screen timeout. TuiSession::drop (tui_driver.rs:1340-1357) only calls self.child.kill()/wait() on the rocm process — it does not reach bash or python as separate PIDs — and spawn_reader's read loop (tui_driver.rs:1412-1439) only checks the stop flag between reads; it is blocked inside a read() syscall that won't return until the pty slave's last holder closes it. With bash/python still alive and holding the slave open, reader.read() never returns, so handle.join() in Drop (line 1368) blocks indefinitely. The scenario's own assertion still fails correctly, but the test process itself never returns — in CI this turns a red scenario into a job timeout and takes every scenario queued after it down too. Confidence 85 · logic · Fix: run the interpreter non-interactively so a wrong python exits on its own (e.g. python </dev/null\n or python -c pass\n — the planted stub ignores arguments and still prints its marker).

Non-blocking

  • The row/scenario comments claim the row only flips once VIRTUAL_ENV is also fixed, but nothing asserts VIRTUAL_ENV — tests/e2e-cucumber/features/engine_shell.feature:38-42, tests/e2e-cucumber/expectations.toml (EAI-8049 block)
    engine_shell sets VIRTUAL_ENV to the unmodified resolved.env_path (apps/rocm/src/engines_cmd.rs:586), and the new assert_engine_interpreter step only checks that running python executes the planted stub — it never reads or asserts the VIRTUAL_ENV value. A fix that corrects the activation hint and PATH but leaves VIRTUAL_ENV doubled would still make this scenario pass (XPASS), contradicting the comment. Fix: either assert VIRTUAL_ENV inside the shell (e.g. echo "$VIRTUAL_ENV") or narrow the comment/description to "the hint and PATH".
  • The id-parsing fallback misses two plausible real-fix shapes — tests/e2e-cucumber/tests/e2e/automations_steps.rs:88-105
    exposed_identifier only recognizes a detail line that starts with id:/identifier:/slug:, so a rendering like command id: server-recover (a form the product's own tests explicitly name, see below) is missed. It also only recognizes a bracketed id on the header (Server recovery [server-recover]); a parenthesised form (Server recovery (server-recover) (off)) falls through to the display-name slug, which does not reduce to the real id. Either shape would leave the row needlessly xfailed after a reasonable fix. Not a correctness bug — the scenario is still sound today — but worth tightening so the row doesn't outlive the behaviour it pins.

Decisions for the author

  • automations-06 pins a listing contract that the product's own unit tests currently forbid — tradeoff
    render_automations_text's existing tests assert the rendered listing does not contain gpu-metrics, gpu-thermal-protect, or command id: (apps/rocm/src/main.rs:19292-19294, :19310+), i.e. hiding raw ids was a deliberate, tested choice. This scenario asserts the opposite: every listed check must be enableable from what the listing shows. Either a display-name-based fix to automations enable or an id-exposing fix to the listing resolves it, but whoever picks up EAI-8047 has to either delete those unit assertions or take the display-name route (which, per the non-blocking item above, the current parser wouldn't even recognize fully). Worth a one-line note in the row/scenario saying which direction is intended, so the fixer isn't working against contradictory signals.

Positive signals

  • listed_checks/exposed_identifier in automations_steps.rs genuinely derives ids from the rendered listing rather than hard-coding them — verified directly against render_automations_text's indentation (" {name} ({state})" header, " "-indented details), so the scenario can't pass against a listing that exposes nothing.
  • screen_logical_lines correctly rejoins soft-wrapped rows via vt100's row_wrapped, avoiding a false failure from a long activation path splitting across two terminal rows.

Deployment notes

CI at this head is green apart from one failing check, E2E tests (Strix Halo, Windows, current), and a few hardware lanes still pending (W7900, MI300X, Strix Halo nightly/release); I could not open the failing job's log from this environment to tell whether it's related to this change (the only new OS-scoped scenario here is tagged @requires-os:linux, and automations-06 carries no GPU/OS tag). gh pr view reports mergeable: MERGEABLE but mergeStateStatus: BLOCKED, consistent with the outstanding change-request review rather than a merge conflict. This is test-only — no migration or runtime compatibility concerns.

What this covered

Read all six changed files in full (tests/e2e-cucumber/expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs) against the merge-base 63ecab2d..20fcc4a4, plus the product code they exercise (apps/rocm/src/engines_cmd.rs's engine_shell/resolve_engine_env, apps/rocm/src/main.rs's render_automations_text and its unit tests). Did not run cargo xtask e2e or any test/lint locally — no Rust toolchain or network access for cargo in this sandbox, so the hang finding above is a traced-through-the-code argument, not a reproduced run. Read the PR description and the existing review threads for context, including the prior automated review at this same head; where its findings overlapped mine I re-derived them independently from the source before including them here, and I did not treat that review's conclusions as given.

The interpreter step typed a bare python into the shell. When PATH is wrong that
can start a real REPL as the terminal's foreground job; it outlives the kill of
rocm, keeps the PTY open and hangs teardown instead of failing the scenario. Run
python with stdin at EOF so a wrong interpreter exits on its own.

The same step now also checks that VIRTUAL_ENV names an environment holding
bin/activate, so engine-shell-02 flips only once the hint, PATH and VIRTUAL_ENV
are all right. Verified with two product mutations: a hint-only fix and a
hint+PATH fix each leave the row xfailed, and the run completes.

automations-06 now tries every plausible identifier per check (exposed ids,
including a 'command id:' key or a parenthesised token, then the label as
listed, then its slug) and counts the check reachable if any works. The row
comment says which fix directions are accepted.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
@fredespi

Copy link
Copy Markdown
Collaborator Author

Thanks both. The hang is real and now fixed in 9f4bc826; nowycondro reached it by tracing and rominf reproduced it. Same cause, one change.

Blocking: the interpreter step could hang the run. A wrong PATH can resolve python to a real interpreter, which starts a REPL as the terminal's foreground job; it outlives the kill of rocm, holds the PTY open, and blocks teardown. The step now runs python </dev/null, so a wrong interpreter exits on its own, and the planted stub ignores stdin. The constant's comment says why, so it isn't "simplified" later. I re-ran rominf's hint-only mutation through the gate: the interpreter step fails and the run completes (about 7 minutes in the container, no orphaned bash).

VIRTUAL_ENV. You were right that nothing asserted it. The step now also runs test -f "$VIRTUAL_ENV/bin/activate" && echo venv-ok-$((6*7)) and waits for the computed marker, so the typed command line (which is echoed) cannot satisfy it. I mutated the product to fix the hint and PATH but leave VIRTUAL_ENV wrong: the row stays xfailed, 0 XPASS. Both mutations were reverted; unmutated gate is 4 xfail, 0 XPASS, 0 stale, 0 unexpected.

automations-06 candidates. The step now tries every plausible identifier per check and counts it reachable if any works: exposed ids (an id: / identifier: / slug: key, any key ending in id such as command id:, a bracketed token, a single parenthesised token), then the label exactly as listed, then its slug. So a fix that makes enable accept the displayed name, or prints command id: or (server-recover), now flips the row. I checked the parser standalone over 11 shapes, including that name: / setting: lines and multi-word parentheses are not read as ids.

Decision (the unit tests forbid the ids). The row comment now says either direction closes it and the scenario accepts both, and that render_automations_text's tests currently assert the listing does not contain command id: or the raw ids, so a fix that shows them has to update those assertions.

Still deferred: the soft-wrap unit test, with the alt-screen follow-up.

The Strix Halo Windows failure is unrelated: repo.amd.com returned HTTP 403 for the TheRock tarball during the Lemonade install.

@fredespi
fredespi requested review from nowycondro and rominf October 27, 2026 13:08

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 9f4bc82

Review

Full review of the whole change, against the current head (9f4bc826), a fresh pass independent of any earlier round's conclusions. Test-only PR pinning two README-walkthrough defects (automations-06 for EAI-8047, engine-shell-02 for EAI-8049) as xfail cucumber scenarios, plus the TUI-driver/parsing support they need. I read all six changed files in full at this head against the origin/main merge-base (63ecab2d), traced the product code both scenarios exercise (apps/rocm/src/main.rs's render_automations_text and apps/rocm/src/engines_cmd.rs's engine shell/resolve_engine_env, plus crates/rocm-core/src/runtime.rs), and verified vt100 0.16's row_wrapped/contents_between semantics against the crate itself. Two independent fan-out passes (one per coupled file group: the automations-06 slice, and the engine-shell-02 slice) additionally traced the same code blind and corroborate: no confirmed code-level defect survived scrutiny in either slice. This implements the two remaining xfail pins from a four-defect batch; the PR description says the other two (EAI-8046, EAI-8048) were dropped because main fixed them independently while this branch was open — consistent with the current diffstat, which touches neither chat nor dash files. No ticket tracker is reachable from here (EAI-xxxx is not a GitHub issue), so that claim is taken on the diff's own evidence, not verified against a tracker.

Blocking

  • The PR description's account of the id-matching logic is inaccurate — it says "two shapes" and claims other shapes stay unrecognized, but the code recognizes more and would not stay xfailed on those shapes — tests/e2e-cucumber/tests/e2e/automations_steps.rs:59-105 (exposed_identifiers), PR description "What it pins" bullet for EAI-8047
    The description: "It recognises an id in two shapes: an id:/identifier:/slug: detail line, or a bracketed token on the header... A fix that prints it some other way leaves the row xfailed until the parser learns that shape." The actual exposed_identifiers also matches (a) any detail-line key ending in " id" — so command id: <x> matches too, which is exactly the shape the product's own unit tests spell out literally (apps/rocm/src/main.rs: assert!(!rendered.contains("command id:")), three occurrences) — and (b) a parenthesised token on the header (Server recovery (server-recover)), not just a bracketed one; both are explicitly named in the function's own doc comment just above it. So the description's closing sentence is false for exactly the two shapes it omits: a command id:-style fix, or a parenthesised-id fix, would not leave the row xfailed, contrary to what the description tells a reader to expect. No functional risk — the code is strictly more permissive than described, so nothing is silently missed by the test itself — but it misstates what was built, which is unreviewed work by the description's own logic. This is also the exact category of description/code mismatch earlier rounds on this PR treated as blocking.
    Confidence 85 · logic · Fix: update the description to list all four matching strategies (two detail-line forms, two header forms), or drop the itemization and point at the code comment instead. One-line/paragraph fix, not a rework.

Non-blocking

  • A header-embedded id placed after the on/off marker would silently drop that check from the test rather than making it fail — tests/e2e-cucumber/tests/e2e/automations_steps.rs:110-147 (listed_checks)
    listed_checks only recognizes a header as a check at all if the trimmed line's literal suffix is "(on)" or "(off)" (strip_suffix). Bracket/paren extraction then runs on the header text with that suffix already stripped, so an id token placed before the state marker (Server recovery [server-recover] (off)) is found correctly — but a hypothetical fix that instead appends it after (Server recovery (off) [server-recover]) would make the line's literal suffix ] instead of (off), fail the strip_suffix check, and cause the whole check to vanish from checks silently — not "tried and failed," just absent — rather than being caught by assert!(!checks.is_empty(), ...) (which only guards a totally empty listing, not a partially-dropped one). All of the candidate shapes the code's own comments enumerate place the id before the marker, so this is speculative rather than demonstrated, which is why it is non-blocking.
    Confidence 70 · logic · Fix: strip "(on)"/"(off)" via contains (matching anywhere in the line) rather than requiring it as a literal suffix, so an id appended after the marker is still recognized as the same header.

Decisions for the author

  • The parsing helpers (slug_of, exposed_identifiers, listed_checks) have no isolated Rust unit tests — non-blocking-improvement
    This logic has gone through several rounds of tightening across this PR's history and is non-trivial (multi-shape candidate extraction, header/detail boundary detection). It is exercised end-to-end by the automations-06 scenario against the real CLI's current output, which is real coverage, but a focused #[test] table over synthetic listing text would catch a regression in the parser itself faster and more precisely than an e2e failure would. Current approach is not wrong, just a cheaper alternative exists.

Positive signals

  • The EAI-8049 fixture (plant_engine_env_recorded_as, planting a bin-shaped env_path) is traced precisely against the real bug mechanism in engines/vllm's install response and resolve_engine_env's unnormalized use of it — this is a fixture that reproduces the actual reported defect, not a stand-in that merely looks similar.
  • python </dev/null for the interpreter check is a well-targeted fix for the PTY-hang risk a prior round found: it makes a wrong interpreter exit on EOF instead of becoming bash's foreground job, while leaving the planted stub (which ignores stdin) unaffected.
  • screen_logical_lines's use of row_wrapped is correct in both directions checked (continuation set from the current row and consulted on the next one), and concatenates wrapped rows with no inserted separator — correct for a raw-terminal hard wrap, as opposed to the codebase's existing word-wrap reconstruction helper for ratatui Wrap, which deliberately does insert one. Good instinct not to reuse logic built for a different wrapping model.
  • The VIRTUAL_ENV check added in this head's own commit closes a real, previously-identified gap (a hint-only fix could otherwise flip the row while VIRTUAL_ENV stayed wrong) with a minimal, non-echoable assertion (venv-ok-$((6*7)), computed by the shell, not just echoed input).

Deployment notes

None. Test-only change (tests/e2e-cucumber/**); no product crate touched, no new dependency, no migration.

What this covered

Every line of all six changed files (tests/e2e-cucumber/expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs) at 63ecab2d…9f4bc826, plus the product code both new scenarios exercise (apps/rocm/src/main.rs's render_automations_text and its unit tests, apps/rocm/src/engines_cmd.rs's engine-shell command, crates/rocm-core/src/runtime.rs's activation-hint helpers, and engines/vllm's install response shape) and AGENTS.md for applicable conventions (internal-content leak scan: clean; DCO sign-off present; CI fully green including the required e2e/mock lanes). Two independent background passes additionally reviewed the automations-06 and engine-shell-02 file groups blind and corroborated no code-level defect in either.

Did not run: history (git log/git blame on the modified regions) and prior-changes passes were not run as separate dedicated passes — this PR's own long review history was read for context and for Step 9 framing, which means this round is not a strictly blind pass in the sense the method otherwise requires; every finding above was nonetheless independently re-derived and verified directly against the current code rather than carried over from that history. No ticket-tracker tool is available here to retrieve EAI-8047/EAI-8049 directly; their status is taken from the PR description's own account. Not reconciled against unresolved discussion on this PR beyond what's stated above.

@fredespi

Copy link
Copy Markdown
Collaborator Author

Fixed the description: it now lists all four id shapes the parser accepts (id:/identifier:/slug: or any key ending in ' id' detail lines; bracketed or parenthesised header token). No code change, head stays 9f4bc82. Not taking the non-blocking items (id-after-marker, parser unit tests) in this PR; the parser is exercised end to end by automations-06. Re-review of 9f4bc82 requested.

…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/features/automations.feature
#	tests/e2e-cucumber/tests/e2e/automations_steps.rs
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/features/automations.feature
#	tests/e2e-cucumber/tests/e2e/automations_steps.rs

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 0d286fd

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (test-only; no product code touched).
Implements regression-pinning test coverage for two open, Backlog-status Jira tickets — EAI-8047 (resolved via lookup to ROCMAI-379, "automations enable/disable ids are not discoverable from the CLI") and EAI-8049 (resolved via lookup to ROCMAI-377, "rocm engines shell prints an activate_hint with a doubled bin segment") — retrieved independently by identifier; both tickets' repro details match this PR's scenarios almost verbatim (e.g. ROCMAI-379's exact "Server recovery → server-recover, not server-recovery" slug mismatch). The PR does not claim to close either ticket, only to pin them as xfail, which matches what the diff does.

Blocking

None

Non-blocking

None

Decisions for the author

  • plant_engine_env_as_installed's bin-shaped env_path means engine-shell-02 cannot go stale from an engines/vllm-only fix — answered-by-intent. Traced: the fixture always writes python/activate at <root>/engine-env/bin/*, but for scenario 02 the planted manifest records env_path as that same bin directory (mimicking engines/vllm/src/install.rs's env_path = command.parent()). resolve_engine_env's manifest branch passes it through unchanged, and runtime_python_activation_hint appends bin/activate again, so the hint resolves to a bin/bin/activate that was never created — reproducing EAI-8049 exactly, which is why the row is xfail today. One fan-out worker initially flagged this as a defect ("the fixture can't pass"), but failing today is the intended state for a pinned defect; the real question is whether it can ever flip. The PR's own expectations.toml comment already discloses the narrowed scope: "A fix in engines/vllm that reports the environment root... does NOT flip this row... Only a fix to the hint composition itself flips it." ROCMAI-377's own "Expected" field ("build the hint from the environment root rather than env_path") points at exactly that hint-composition fix, so the scenario is well-targeted at the most likely real fix location, not a dead end. Confirmed independently that no code in apps/rocm, crates/, or engines/ ever writes a <env_id>.json manifest to disk (only load_engine_env_manifest reads one) — so the PR's claim that "no install writes such a manifest to disk" is accurate, not a cover for an untested path. Nothing to fix; flagging only so the narrowed-flip-condition tradeoff is visible and confirmed as still the intended design.
  • user_lists_checks asserts rc == 0 inline instead of deferring to a Then step via record() — non-blocking-improvement, tests/e2e-cucumber/tests/e2e/automations_steps.rs:526-534. Every other When step in the file (enable_observe, enable_propose, disable_watcher, list_automations, enable_unknown, the proposal actions) calls record(world, stdout, stderr, rc) and leaves assertions to a later Then; this one is the only one that asserts inline and the only one that skips record() (so world.cli_stderr/cli_rc are never populated for this action). It isn't wrong — failing fast here with the raw stdout/stderr in the panic message is arguably clearer than letting assert_listed_checks_enableable discover an empty listing and panic about "no background checks found" — but it is a visible deviation from the file's own established convention, worth a one-line comment explaining the choice (or switching to the convention) rather than silent inconsistency for the next reader.
  • Candidate-guessing in listed_checks/exposed_identifiers could in principle mask a defect via an id/slug collision — tradeoff. The fallback order (exposed id → label as-is → slug) is deliberately permissive so a correct fix is picked up in any of several shapes; verified against all 6 real BUILTIN_WATCHERS ids and their display names that no guessed candidate for one check collides with a different check's real id today, so this isn't live, but a future watcher whose slugified display name happens to equal another watcher's real id would silently "pass" the wrong check's identity. No action needed now; worth a comment or a cross-check test if the watcher table grows.
  • screen_logical_lines cannot distinguish a logical line that starts at the visible screen's top row from one whose start scrolled off above it — non-blocking-improvement, tests/e2e-cucumber/tests/e2e/tui_driver.rs (new method). Traced the wrap-stitching logic itself and it correctly mirrors terminal auto-wrap semantics (only concatenates when row_wrapped is true, which only fires when a row was genuinely filled and overflowed); the only gap is the inherent screen-snapshot limitation of row 0 always being treated as a fresh line start. Not triggered in this PR's actual use (the activate_hint: line prints ~8 lines into the session, well clear of row 0), so not a finding against this change, just worth a one-line doc caveat since the method is positioned as "read a whole value off the screen."

Positive signals

  • automations-10's identifier derivation is specifically designed to defend against the laziest false-pass: ids are deliberately not hard-coded "since that would pass against a listing that exposes nothing," and the step's own guard (assert!(!checks.is_empty(), ...)) exists so a parser that silently matched zero checks can't "prove" the contract by checking nothing.
  • engine-shell-02's second assertion (running python and checking $VIRTUAL_ENV/bin/activate) is deliberately there because "the same doubled path also breaks PATH and VIRTUAL_ENV, so a fix to the hint text alone does not flip the row" — traced and confirmed: both checks exec real shell behavior (actual stdout from the planted interpreter stub, actual test -f through a $((6*7))-gated marker immune to the typed command line's own echo), so a hint-only cosmetic fix genuinely cannot satisfy this row.
  • Both pinned scenarios' technical claims were independently cross-checked against the two Jira tickets they cite (retrieved by identifier, not by following a link) and matched almost verbatim, including exact slug-mismatch examples in EAI-8047/ROCMAI-379.

Deployment notes

None. Test-only change (tests/e2e-cucumber/**); no product crate is touched, no migration, no API or behavior change for an operator to prepare for.

What this covered

Read the full diff between base 8c23a91abed959df3233651346a379feea3fa0e2 and the pinned head 0d286fddd414a72b78567d5667938ffbcab6ab58 (verified via git diff --stat matching GitHub's reported 6 files / +414/-15), all 6 changed files in full (expectations.toml, automations.feature, engine_shell.feature, automations_steps.rs, engines_steps.rs, tui_driver.rs), plus the product source each test actually exercises (apps/rocm/src/automations.rs, crates/rocm-core/src/automation.rs, apps/rocm/src/engines_cmd.rs, crates/rocm-core/src/runtime.rs, engines/vllm/src/install.rs, apps/rocm/src/runtime_install_finalize.rs) to verify the tests' factual claims against real control flow rather than trusting the PR's narrative. Confirmed the head is itself a clean merge of main into the feature branch (second parent = base sha exactly) with no leftover conflict markers and no duplicate scenario ids or step definitions.

Fan-out: two independent blind-review workers, one per coupling group (automations-10 scenario + its product listing/enable code; engine-shell-02 scenario + its product env-path/hint-composition code), each seeded with full file content and relevant product excerpts but never the PR description. First send to each worker contained an unfilled content placeholder by my own error; both workers correctly refused to fabricate findings from it rather than guessing, so both were re-sent with the actual file content and completed a full trace-level pass on the second round. Their one candidate finding each were verified above: one was refuted as describing exactly the intended, pre-disclosed xfail behavior (retired as answered-by-intent); one survived as a genuine but non-blocking convention note.

Design review (Step 7) considered durability of engine-shell-02's fixture against the real fix location, the repo's established xfail-pinning convention (consistent with cited PR #216), id/slug collision risk, the merge-commit-as-PR-head question (no CONTRIBUTING.md rule against it found; not a concern), and screen_logical_lines's scope — none rose to a fundamental approach problem. Checked git log history on all 6 files, including confirming the earlier-reviewed commit 9f4bc826117552220ff43ce0db348ec32832970b is this branch's own ancestor. Read AGENTS.md's testing-convention sections; found no violation (no product behavior changed, so the docs-sync and message-assertion rules don't apply; the mutation-test requirement is explicitly satisfied per the PR's own test plan for engine-shell-02, and implicitly satisfied for automations-10 by the scenario currently reproducing the real defect).

Read CI results rather than re-running anything (no Rust/Cargo toolchain available in this sandbox): every required check is green at this exact head (clippy, build-and-test, Test (affected crates), windows-build-and-test, the blocking E2E mock job and consolidated report, commit-signoff, license headers, prek, stale-xfail-hint, markdown links, third-party notices, coverage). Two self-hosted hardware E2E lanes (MI300X, W7900) were still QUEUED and two (Strix Halo Ubuntu release/nightly) were CANCELLED at observation time — no FAILURE conclusion anywhere; the PR's own scope states neither scenario needs GPU dispatch, so this doesn't bear on the blocking gate.

Retrieved both named tickets by identifier (EAI-8047, EAI-8049), per Step 9, after findings were settled — see the opening block.

Did not run: the "prior changes" source-reading pass (whether past review comments on these files apply here) — doing this meaningfully requires reading this PR's own review-thread content, which the blind-review rule excludes; not substituted with guesswork. Code-comments-in-files were read as part of each full-file read, not as a separate pass.

Two disclosures for auditability: (1) the consolidating context (not the fan-out workers) read the PR description and expectations.toml's own comments early, during scope/fact-gathering, ahead of the skill's prescribed Step-9 ordering — the fan-out workers that actually produced candidate findings were kept blind throughout, so this affected framing/verification speed, not the findings' independence. (2) The Step-3 pre-publish race check (get_reviews, to confirm no existing review from "nowycondro" at this exact head) returned full review bodies rather than just login/commit metadata; a small amount of other reviewers' approval text was incidentally visible in that call's output. No specific claim from it was adopted into any finding above — all findings here were reached via independent code-tracing and the blind fan-out before that call was made. Confirmed via that same call: "nowycondro" has reviews only at 20fcc4a4 and 9f4bc826 (earlier commits), none at the pinned head, so publishing this is not a duplicate. Not reconciled against any other discussion on this PR.

…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/features/automations.feature
…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/features/automations.feature

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 0e4d264

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (base db7e54089ac668ab86d100bc2456df6230ee762f → head 0e4d264f73aa6c503cde4f6d92632da744f0304f, 6 files, +414/-15).
This implements test-only scenarios pinning two already-open, backlog-status defects: Jira EAI-8047→ROCMAI-379 ("automations enable/disable ids are not discoverable") and EAI-8049→ROCMAI-377 ("engines shell activate_hint has a doubled bin segment"). Both tickets were retrieved and read; their content matches what the new scenarios assert and what the production code (apps/rocm/src/automations.rs, apps/rocm/src/engines_cmd.rs, crates/rocm-core/src/runtime.rs, engines/vllm/src/install.rs) actually does — the PR does not claim to close either ticket, only to pin regression coverage for the correct eventual behaviour.

Blocking

None

Non-blocking

  • Stale file path in a new comment — tests/e2e-cucumber/expectations.toml:202
    The newly added comment says render_automations_text's unit tests live in apps/rocm/src/main.rs. At the reviewed head that function and its unit tests (render_automations_text_marks_gpu_metrics_read_only, etc.) are in apps/rocm/src/automations.rs (confirmed by git grep against the PR head); main.rs no longer holds them. Confidence 85 · mechanical · Fix: update the path in the comment to apps/rocm/src/automations.rs.

Decisions for the author

None

Positive signals

  • tests/e2e-cucumber/tests/e2e/automations_steps.rs's listed_checks/exposed_identifiers parser is deliberately permissive (tries several id shapes) specifically so a real fix doesn't have to match one hard-coded format to flip the row — the comments say so and the parsing logic (checked against render_automations_text's actual two/four-space indentation scheme) backs that up.
  • engines_steps.rs's assert_engine_interpreter pairs the activation-hint text with the actual runtime state (VIRTUAL_ENV, and really running python) rather than only pinning wording, matching this repo's own "a message is asserted together with the state it describes" convention (AGENTS.md §3 / e2e README.md).
  • The </dev/null on the interpreter-exec command is a real, well-explained guard against a hung PTY teardown if PATH resolves to a real system Python instead of the planted stub.

Deployment notes

None

What this covered

DEGRADED REVIEW: both attempts to fan out to independent sub-reviewers failed (Concurrent subagent limit reached), so this ran single-context rather than as the independent-worker fan-out the method calls for; recommend re-running with fan-out available, since it typically surfaces more than a single-context pass does.

Read completely at the PR head: tests/e2e-cucumber/expectations.toml, tests/e2e-cucumber/features/automations.feature, tests/e2e-cucumber/features/engine_shell.feature, tests/e2e-cucumber/tests/e2e/automations_steps.rs (1084 lines), tests/e2e-cucumber/tests/e2e/engines_steps.rs (249 lines), and the changed screen_logical_lines addition plus its surrounding impl TuiSession block in tests/e2e-cucumber/tests/e2e/tui_driver.rs. Cross-checked the new test/parsing logic directly against the production code it exercises: apps/rocm/src/automations.rs (render_automations_text, automations(), watcher_plain_name), crates/rocm-core/src/automation.rs (builtin_watcher/BUILTIN_WATCHERS), apps/rocm/src/engines_cmd.rs (engine_shell, resolve_engine_env), crates/rocm-core/src/runtime.rs (runtime_python_activation_hint), and engines/vllm/src/install.rs (the command.parent() env_path computation) — all confirmed to match what the new scenarios assume. Verified the vt100 crate's documented row_wrapped semantics (docs.rs) against the new soft-wrap-rejoining loop in screen_logical_lines and found the sequencing correct. Read AGENTS.md and tests/e2e-cucumber/README.md for applicable conventions (scenario/@id naming, enforced by tests/feature_naming.rs — confirmed automations-12 and engine-shell-02 are sequential and correctly keyed). Read CI results for the reviewed commit via gh pr checks/job logs: the blocking mock E2E tests / E2E consolidated report jobs pass; two non-blocking self-hosted GPU/Linux jobs (E2E tests (Strix Halo, Ubuntu, release/nightly, current)) fail, but the failure is an unrelated pre-existing regression in a different, untouched scenario (serve-stop-leaves-no-background-helper) — the job logs show the two new scenarios in this PR reconciling as expected xfail there too, not as the cause of the job failure.

Did not run: history/git blame on the modified regions, and a prior-changes pass against earlier review rounds on this PR — per the method's own rule, this review did not read this PR's discussion, comments, or earlier review rounds at all, and nothing from them fed into the findings above. No Rust toolchain was available to actually execute cargo test/mutate the test binary locally, so the two required test-quality verdicts below are argued from reading the code against the production paths it exercises, not from an actual run.

Test-quality verdicts (both new scenarios, per the project's own testing convention):

  • automations-12/assert_listed_checks_enableable: traced that today render_automations_text never emits any of the id-shaped tokens the parser looks for (checked watcher_plain_name's literal strings — none contain :/[/(), and builtin_watcher matches only the exact internal id, so every derived candidate (display label, slug) is rejected and the scenario fails now, as its xfail row expects; if the listing exposed a real id in any of the four recognised shapes, the derived candidate would succeed and the row would go stale (XPASS) — so this is not a tautological test.
  • engine-shell-02/assert_activation_hint_resolves + assert_engine_interpreter: traced that resolve_engine_env's auto-install branch uses the install response's env_path live, that engines/vllm/src/install.rs sets it to command.parent() (the bin dir), and that runtime_python_activation_hint appends bin/activate to whatever it's given — so a bin-shaped env_path doubles the segment today, matching the planted fixture. A hint-wording-only fix (per the PR's own stated mutation test) would still leave VIRTUAL_ENV/PATH wrong, so assert_engine_interpreter's separate interpreter-execution check keeps the row failing until the real environment-root fix lands — confirmed by reading the two assertions' independence, not by running the mutation myself.

Not reconciled against any prior review round or discussion on this PR.

🤖 Generated with Claude Code

…-for-the-second

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>

# Conflicts:
#	tests/e2e-cucumber/expectations.toml
#	tests/e2e-cucumber/features/automations.feature
#	tests/e2e-cucumber/tests/e2e/automations_steps.rs

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 530f779f633

Outcome: no blocking findings.

Test-only change pinning two known CLI defects (EAI-8047, EAI-8049) as xfail scenarios in tests/e2e-cucumber/, following the existing expectations.toml + Gherkin + step-function pattern used elsewhere in this suite.

What I verified against the actual source (not just the diff):

  • automations-13's listing parser (automations_steps.rs: listed_checks/exposed_identifiers/slug_of) was traced against the real text render_automations_text (apps/rocm/src/automations.rs) produces today — it correctly finds zero usable identifiers, matching the pinned bug, and a !checks.is_empty() guard prevents a vacuous pass. rocm automations list has no --json output today, so text-parsing is the only option available.
  • engine-shell-02's doubled-path claim checks out: engines/vllm/src/install.rs reports env_path = command.parent(), and runtime_python_activation_hint (crates/rocm-core/src/runtime.rs) appends bin/activate again. Both new assertions (hint-file-exists, interpreter+VIRTUAL_ENV) are independently load-bearing — a hint-text-only fix would leave the second one red, as the PR's own test plan claims.
  • The new TuiSession::screen_logical_lines helper (vt100-based soft-wrap rejoining) matches the vt100 0.18 crate's actual row_wrapped/contents_between semantics; no corruption at wrap boundaries.
  • Pulled the self-hosted GPU CI logs for this exact head commit: both new scenarios reproduce and fail for precisely the documented reason, correctly reconciled as expected xfails (not unexpected failures). The blocking mock e2e job is green.
  • Two independent blind fan-out passes (automations group, engine-shell group) separately traced the same code paths and returned no findings ≥70 confidence.

Non-blocking / FYI:

  • The non-blocking self-hosted GPU jobs show one unrelated failing scenario (serve-stop-leaves-no-background-helper), not touched by this diff.

What this covered: full diff (6 files, 962ac0c0…530f779f) read completely, plus the production code it exercises and the repo's AGENTS.md/e2e README.md conventions. Did not reconcile against the PR's existing review thread.

Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings in the diff itself; mixed CI/merge state

Full review of the whole change (merge-base 962ac0c0 … head 530f779f). Test-only: pins two open defects (EAI-8047, EAI-8049) as xfail rows in expectations.toml with one BDD scenario each, in the same shape as prior art #216. No product code is touched. EAI-8047/EAI-8049 are not GitHub-trackable identifiers from here, so the ticket relationship is as stated in the description and not independently retrievable.

Blocking

  • Branch cannot currently be merged — PR-level, not a line in the diff.
    GitHub reports mergeable: CONFLICTING / mergeStateStatus: DIRTY against current main. The PR's merge-base (962ac0c0, "ci: start build-and-test without waiting for clippy (#875)") is far behind main's current tip (which has since removed the W7900 lane from e2e-selfhosted.yml in #1066, among many other merges) — this branch has been open since August and needs a rebase before it can land.
    Confidence 100 · architectural · Fix: rebase onto current main, resolve conflicts, re-push.

  • Three self-hosted hardware E2E lanes are red — E2E tests (MI300X, current), E2E tests (Strix Halo, Ubuntu, release, current), E2E tests (Strix Halo, Ubuntu, nightly, current) (all FAILURE), plus E2E tests (W7900, current) CANCELLED and the self-hosted consolidated report FAILURE.
    Read the actual failure line from each job's log (not inferred from the job name): all three failures are the identical line FAIL: 'serve-stop-leaves-no-background-helper' was expected to pass on this host but FAILED — a regression. That scenario is untouched by this diff (the PR only adds automations-13 and engine-shell-02, both when = {}/mock-scoped per the expectations.toml rows I read), and the same scenario fails identically across three independent hardware lanes, which points at a pre-existing or environment-level regression rather than something this PR introduced. I could not get a clean (non-cancelled) recent self-hosted run on current main to confirm it fails there too, so I can't fully close this out — but nothing in the failure output implicates this PR's two new scenarios. The mock-lane check the PR's own test plan calls out as blocking (E2E consolidated report, CI workflow) is green, as claimed.
    Confidence 85 (verified from the actual log output, not assumed) · architectural · Fix: re-run self-hosted after rebasing; if serve-stop-leaves-no-background-helper is already red on main, this is not this PR's problem to fix, but it should be confirmed clean (or separately triaged) before merge.

Non-blocking

None

Decisions for the author

None

Positive signals

  • EAI-8049's premise was verified against the actual product code, not just the PR's own comments: engines/vllm/src/install.rs does report env_path as runtime.command.parent() (the bin dir), and runtime_python_activation_hint/runtime_python_env_bin_dir in crates/rocm-core/src/runtime.rs do treat whatever they're given as the environment root and unconditionally append bin/activate or bin — so the doubled-path bug, and its claimed effect on VIRTUAL_ENV and PATH (both built from the same untransformed resolved.env_path in apps/rocm/src/engines_cmd.rs), are real and accurately diagnosed.
  • EAI-8047's premise checks out the same way: watcher_plain_name in apps/rocm/src/automations.rs returns only the display name, no field in render_automations_text's output carries the raw watcher id, and builtin_watcher() does an exact-string lookup with no slug or label fallback — so the listing genuinely publishes nothing enable will accept, exactly as pinned.
  • Good test-design judgment: plant_engine_env_as_installed plants a manifest (rather than mocking a live install response) — tracing resolve_engine_env, both the manifest and auto_install branches feed the same untransformed env_path into the hint/PATH/VIRTUAL_ENV composition, so the fixture genuinely exercises the bug path rather than taking a shortcut around it.
  • assert_listed_checks_enableable deliberately derives candidate ids from the listing (detail-line keys, bracketed/parenthesised header tokens, raw label, slug) rather than hard-coding the real ids, which is the only way this scenario can go stale automatically once the product actually publishes something usable.
  • The rewritten assert_engine_interpreter is a strict improvement even for the pre-existing scenario that reuses it: it now runs python to observe real output instead of just comparing command -v python's printed path, and adds a VIRTUAL_ENV/bin/activate check that didn't exist before — this closes a real coverage gap, not just for the new scenario.
  • The </dev/null reasoning in assert_engine_interpreter's comment (avoiding a hung REPL as PTY foreground job if python resolves wrong) is sound and shows real thought about E2E hygiene.

Deployment notes

None — test-only change, no product behavior, migration, or configuration affected.

What this covered

Read completely: all 6 changed files (tests/e2e-cucumber/expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs) at the PR head commit, read via git show pr-292:<path> directly from the git object store (the working-tree checkout was found to have been mutated by unrelated concurrent activity in a shared sandbox, so file reads were taken from committed blobs rather than the working tree to avoid acting on contaminated state). Also read, to verify the tests' claims against reality rather than taking them on faith: engines/vllm/src/install.rs, crates/rocm-core/src/runtime.rs (runtime_python_activation_hint, runtime_python_env_bin_dir, runtime_python_env_root_for_executable), apps/rocm/src/engines_cmd.rs (resolve_engine_env, engine_shell), apps/rocm/src/automations.rs (render_automations_text, watcher_plain_name, the Enable/Disable command handlers), and crates/rocm-core/src/config.rs (builtin_watcher, BUILTIN_WATCHERS). Checked CI via the status-check rollup and read the actual failure-log lines for every failing job rather than trusting job names. Checked current mergeability via the GitHub API.

Did not run: no test suite was executed; all correctness conclusions come from reading the diff against the actual production source it exercises, not from execution. Did not run: history/prior-changes passes beyond the targeted git log/git show lookups described above — no further value expected given the production-code tracing already done. A second independent fan-out pass over this same scope was launched for corroboration but had not completed by the time this review was finalized; it is not reflected here. Not reconciled against the PR's existing review threads or comments — those were deliberately not read, per this review's own method.

@juhovainio

Copy link
Copy Markdown
Collaborator

Addendum to the review submitted above: the required diff-id marker and footer were omitted from that review body by mistake. Appending them here since the review API does not expose an edit path from this tooling.


🤖 Automated by an AI agent

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change.
Pins EAI-8047 and EAI-8049 as xfail regression scenarios in tests/e2e-cucumber/; does not claim to close either ticket, only to pin the correct eventual behaviour. Both are bare external ticket IDs with no tracker access available to this review, so their content is taken from the PR description and corroborated independently against the production code the new scenarios exercise (see below), not retrieved directly.

Blocking

  • This branch is not currently mergeable into main — tests/e2e-cucumber/expectations.toml:168 (inline comment has the detail)
    GitHub reports mergeable: CONFLICTING / mergeStateStatus: DIRTY for this exact head. Independently reproduced via git merge-tree: the only conflict is a textual clash from two independent appends landing near the same spot in this file, because main has kept moving since this branch's last sync (merge-base with the current main tip is 962ac0c0, ~168 commits behind). Every other changed file merges cleanly.
    Confidence 100 · architectural · Fix: merge/rebase onto current main, keep both appended blocks, push. Mechanical, not a code defect.

Non-blocking

  • no_background_checks duplicates an existing step — tests/e2e-cucumber/tests/e2e/automations_steps.rs:544 (inline comment has the detail)
    Same body as fresh_config in config_steps.rs (both just call suppress_daemon_spawn(world)). Not wrong, but worth a one-line comment or reuse so the duplication reads as deliberate.
    Confidence 75 · mechanical · Fix: note why, or reuse the existing step phrasing.

Decisions for the author

  • engine-shell-02's fixture plants a manifest shape no real install writes to disk — tradeoff
    Traced and confirmed accurate: engines/vllm/src/install.rs's install_response does report env_path as command.parent() (the bin dir, via the auto-install branch of resolve_engine_env), and no code anywhere writes a <env_id>.json manifest — the fixture stands in for that response, as already disclosed in the PR's own comments. Consequence, also already disclosed: a fix in engines/vllm alone (reporting the environment root) closes the underlying defect but does not flip this row, since the planted manifest is unchanged — only a fix to the hint composition does. Flagging only so the maintenance burden is visibly confirmed as intended, not to re-litigate it.

Positive signals

  • assert_listed_checks_enableable's identifier derivation deliberately avoids hard-coding ids (tries several shapes a fix might expose, falling back to the listed label and its slug) and guards against a vacuous pass with assert!(!checks.is_empty(), …). Confirmed it fails today for the stated reason: builtin_watcher matches only the exact internal id, and none of the six watchers' display names or naive slugs equal their real id.
  • assert_engine_interpreter's two assertions (actually running python and waiting for the planted stub's own output, plus a computed $((6*7)) marker gated on $VIRTUAL_ENV/bin/activate) are independently load-bearing, so a hint-text-only fix genuinely cannot flip engine-shell-02 — matches the PR's own stated mutation test.
  • screen_logical_lines' soft-wrap rejoining correctly mirrors vt100's row_wrapped semantics (only concatenates when the previous row was actually filled and overflowed), reusing the same crate API as the file's existing screen_text.

Deployment notes

None — test-only change confined to tests/e2e-cucumber/**; no production crate, API, migration, or configuration is affected.

What this covered

DEGRADED REVIEW: both attempts to fan out to independent sub-reviewers hit a concurrency limit from unrelated activity sharing this environment, so this ran single-context rather than the independent-worker split the method calls for. Recommend re-running with fan-out available. To compensate, every factual claim the new tests make was independently traced against the real production source rather than taken on faith from the tests' own comments.

Read completely at head 530f779f, diffed against origin/main merge-base 962ac0c0 (6 files, +414/-15): expectations.toml, features/automations.feature, features/engine_shell.feature, tests/e2e/automations_steps.rs, tests/e2e/engines_steps.rs, tests/e2e/tui_driver.rs. Cross-checked against the production code the new scenarios exercise: engines/vllm/src/install.rs, crates/rocm-core/src/runtime.rs, apps/rocm/src/engines_cmd.rs, apps/rocm/src/automations.rs, crates/rocm-core/src/automation.rs.

CI: every check on the blocking CI workflow passes at this head (build/test, clippy, the mock E2E tests job and its consolidated report, commit sign-off, license/doc checks). Failures are confined to the E2E self-hosted workflow's GPU hardware lanes, which this project's own docs state are non-blocking; one self-hosted lane was cancelled. None of this is held against the change. Mergeability was additionally checked live against the current main tip directly (not just the PR's recorded base) — see Blocking.

This PR has a long prior review history at earlier commits; that history was read only after independent findings here were reached, to reconcile rather than to seed conclusions. Several earlier blocking findings from that history (an interpreter step that could hang the whole run; an identifier-matching gap for a command id:-shaped fix) were re-verified here as fixed in the current diff, by reading the code directly rather than trusting that history.

Did not run: a test suite (no Rust toolchain available here) — the two required test-strength verdicts were argued by tracing each assertion against the real control flow it targets, not by executing the mutation tests described in the PR's own test plan. Prior-review-thread content was not used to generate findings, only for reconciliation afterward.


🤖 Automated by an AI agent

serve_timeout_secs = 90
flaky = true

# -- README-walkthrough defects (EAI-8047, EAI-8049) --

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: this branch is not currently mergeable into main. GitHub reports mergeable: CONFLICTING / mergeStateStatus: DIRTY for this exact head. Independently reproduced with git merge-tree $(git merge-base origin/main pr/292) origin/main pr/292: the only conflict is a textual clash right here, where two independent appends landed near the same spot in this file — main has added further xfail rows in the time since this branch's last sync with it (the branch's merge-base with the current main tip is 962ac0c0, from 2026-11-04; main is ~168 commits ahead of that point as of this review). Every other changed file merges cleanly.

Not a code defect — the fix is mechanical: merge/rebase onto the current main tip, keep both appended blocks, and push.

}

#[given("a machine with no background checks turned on")]
async fn no_background_checks(world: &mut E2eWorld) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this step duplicates an existing one. no_background_checks calls only suppress_daemon_spawn(world) — exactly the same body as fresh_config in config_steps.rs (#[given("a fresh CLI configuration")]). Not wrong (a scenario-appropriate Given phrasing is reasonable even when two step names share an implementation), but it's a literal duplicate rather than a thin wrapper reusing the existing step text — worth a one-line comment noting the deliberate duplication, or reusing the existing phrasing, so the next reader doesn't wonder whether the two are meant to diverge later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants