Skip to content

ci(e2e): fail the pre-warm when the runtime cannot import torchvision.ops - #1103

Merged
tomastola merged 5 commits into
mainfrom
ci/prewarm-torchvision-import-check
Nov 17, 2026
Merged

tomastola merged 5 commits into
mainfrom
ci/prewarm-torchvision-import-check

Conversation

@tomastola

@tomastola tomastola commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Why

A pre-warmed runtime that persists between jobs can pass every freshness check and still be unusable: when its torch and torchvision come from different installs, import torchvision dies with operator torchvision::nms does not exist. Nothing before the suite imports it, so every vLLM serve scenario finds out for itself after its 300 s readiness timeout (bench-06 waits twice).

That is what turned the MI300X lane from about 33 minutes and green (ten consecutive green PR runs on 11-12, the last at 11:06 UTC) into 79 to 90 minutes and red on every run since: 11 of the 12 failed steps in job 131099458567 and 12 of 13 in the 11-16 run 43839208429 are this one error. The persisted release-wheel-multi-arch-10-1-0 runtime holds torch 2.14.0, torchvision 0.28.0 and a vLLM built for torch 2.13; a fresh install of the same SDK ends on torch 2.13.0 and serves in about 60 s.

What changes

Two steps in the pre-warm (xtask/src/e2e_prewarm_vision.rs), both probing the active runtime's own Python for import torchvision.ops. That is the interpreter vLLM installs its packages into (install_vllm_with_uv(&managed.python_executable, ..) in engines/vllm/src/install.rs), so there is no separate engine venv to probe. The runtime is found through the tree's active.json and its registry manifest.

  • Repair, before the freshness decision. When the import fails with torchvision's operator-mismatch error (torchvision:: and does not exist) and the default engine imports torch, the runtime is removed with rocm runtimes uninstall (the repair repair_poisoned_runtimes already uses), so decide sees no runtime for the channel and installs afresh. It declines, and leaves the runtime for the check below to fail on, when the run is pinned to an exact version, when rocm update marks the runtime ahead_of_index (a reinstall would roll it back), or when that report cannot be read. Any other failure, including a hang, is left alone here, so a flaky probe cannot delete a healthy runtime. A mismatch the engine install itself creates on a reused runtime is found by the check below, fails that run, and is removed at the start of the next.
  • Check, after the engine check and before the reuse return, so a warm runner is checked too. A runtime that still cannot import fails the pre-warm, naming the runtime and the tail of the Python error. When this same run installed the runtime, the error says the installer produced the broken pair.
  • The probe puts the manifest's rocm_sdk.library_paths (and the amd-smi shim) on the loader path, as the CLI's own probes do, and is killed after two minutes.
  • Where the default engine does not import torch (Lemonade; engines/lemonade/src/lib.rs reports torch_version: None), the finding is a warning, and nothing is removed.
  • Skipped: a runtime with no torchvision, a tree with no active runtime or no recorded interpreter, and a shared library that fails to load when the manifest records no library directories to supply (a warning). Failed: an active.json or registry manifest that cannot be read or parsed, a marker without runtime_key, an interpreter that will not start.

docs/ci-hardware-testing.md lists both steps.

There is no Gherkin scenario: this is CI plumbing in xtask that changes what the pre-warm does on a lane, not what any rocm command prints, exits with, creates or selects.

Verification

I reproduced the failure with the persisted runtime's own Python: import torchvision.ops exits 1 in 2 seconds with the error above.

The new tests cover the probe's verdicts, the state-reading branches, the library directories reaching the probe's environment, a probe that never finishes being killed, the repair against a stand-in rocm that records its calls (an uninstall for a mismatch on a torch engine, none for another error, a working or torchvision-free runtime, or Lemonade), both kinds of advice, and a real python3 running the probe text against throwaway torchvision packages. Source-order tests keep the repair before the dangling-pointer repair and the check before the reuse return, since no test calls run. cargo test -p xtask passes and clippy -D warnings is clean; breaking the library-path, engine, deadline and pipe-wait code each turns its test red.

Not verified:

  • The probe on a Strix runtime, and Windows (the fake-interpreter tests are unix-only).
  • The repair against the real CLI. The tests show the uninstall is requested with the right arguments and that an empty registry resolves to Decision::Install (no_managed_runtime_installs); no test runs the real rocm runtimes uninstall and then decide on its output.

Scope

Extending the CLI's post-install device probe (which imports only torch) to torchvision.ops would protect rocm serve users as well, and is a follow-up.

  • If this PR fixes a bug, searched tests/e2e-cucumber/expectations.toml for the fixed ticket ID and removed/narrowed any now-stale xfail rows. (Not applicable: no ticketed bug is fixed and no xfail row exists for this.)
  • If this PR adds a new subcommand or subsystem, its domain implementation lives in its own file per docs/architecture.md. (Not applicable.)
  • If this PR adds, renames or removes a production source file, docs/architecture.md has the matching entry. (xtask/src/e2e_prewarm_vision.rs is new, split out of e2e_prewarm.rs for the module-size limit; its entry is in docs/architecture.md, and cargo xtask check-architecture-doc passes.)
  • Every new or changed user-facing message was read against the code path that runs after it, and its test asserts the resulting state, not only the wording. (Partly: the wording and the command are asserted; the state the advice produces is not, as listed under Not verified.)

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@tomastola
tomastola force-pushed the ci/prewarm-torchvision-import-check branch from c3fde96 to 446ee41 Compare November 17, 2026 10:33
@tomastola
tomastola marked this pull request as ready for review November 17, 2026 13:02
@tomastola
tomastola requested a review from a team as a code owner November 17, 2026 13:02
@tomastola

Copy link
Copy Markdown
Collaborator Author

Rebased onto main (446ee41), which includes #1064. E2E tests (MI300X, current) is still queued behind the runner backlog; the earlier red on c3fde96 was torch seeing 0 devices on that runner, which #1064 fixed. I'll check its log for any torchvision::nms / torchvision.ops error once it runs.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Detect-and-fail placement is good (after ensure_default_engine, before the reuse return, with a source-order test), and the real-python3 test covers working/broken/absent packages. Nothing blocking in the mechanism, but a few things should be fixed before merge:

  1. The shared-library skip can turn the check into a silent no-op on the lanes it targets (inline, ~L1006).
  2. The probe has no timeout (inline, ~L927).
  3. The reinstall remediation is pinned by wording only; it isn't shown that the next pre-warm actually installs (inline, ~L1022).
  4. The doc comment states CI history/timings as fact (inline, ~L980).
  5. The PR body should say why there is no Gherkin scenario (CI-plumbing change), per the contribution guide.

Questions: is leaving the MI300X lane red until a manual reinstall deliberate, given repair_poisoned_runtimes already repairs other bad runtimes? Is it intended that the check is fatal on Lemonade lanes whose serves don't import torch? And does the vLLM engine venv carry its own torchvision, given the probe checks the runtime's Python rather than the venv?

Comment thread xtask/src/e2e_prewarm.rs Outdated
println!("pre-warm: runtime {key} has no torchvision; nothing to check");
}
VisionImport::Working => println!("pre-warm: runtime {key} imports torchvision.ops"),
// This probe runs the interpreter without the library paths the CLI's own

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The CLI records the runtime's rocm_sdk.library_paths and its own probes add them (runtime_library_paths_for_key). Running the interpreter bare and skipping on "cannot open shared object file" makes this check a silent no-op wherever torch needs those paths, and it also skips a genuinely broken runtime that fails with a missing .so. Please add the manifest library paths to the probe environment and keep the skip only as a last resort.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f2791ec. The probe now reads rocm_sdk.library_paths from the runtime's manifest and puts them, plus the amd-smi shim from rocm_core::runtime_loader_path_entries, on the loader path ahead of the job's own. A library that still will not load is now a broken runtime and fails the pre-warm; the skip survives only for a manifest that records no library directories to supply. Tests: the directory reaches the probe's environment, a load failure fails when directories were supplied, and it is skipped only when none were.

Comment thread xtask/src/e2e_prewarm.rs Outdated
}

/// Run [`TORCHVISION_PROBE`] with `python`.
fn probe_vision_import(python: &Path) -> std::io::Result<VisionImport> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

output() has no deadline: a hung import torchvision.ops stalls the pre-warm until the job timeout with no message, which is the long-red-lane failure this PR removes. Consider spawn + bounded wait + kill (with kill_on_drop-style cleanup).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f2791ec. The probe is spawned with its pipes drained on their own threads, polled against a two-minute deadline, then killed and reported as did not finish within …s and was killed. Test: a stand-in interpreter that sleeps for 60 s is reported in well under that (the test fails if the deadline is removed).

Comment thread xtask/src/e2e_prewarm.rs Outdated
""
};
let advice = if installed_this_run {
"This run installed the runtime, so the installer produced the broken pair and \

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This advice promises that uninstalling the runtime makes the next pre-warm reinstall it, but the test asserts wording only. Can decide() return Reuse or Update instead (e.g. another runtime of the channel still in the registry)? Either cover that state in a test or soften the wording.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, decide() can return Reuse or Update when another runtime of the channel is still in the registry, so the old wording was not guaranteed. Changed in f2791ec: a mismatch is now removed by repair_torchvision_mismatch at the start of the next pre-warm, whichever runtime is active then (the same rocm runtimes uninstall repair as repair_poisoned_runtimes), and the advice says to run the pre-warm again. A test asserts the uninstall request is made for a mismatch and not for other errors, and that an empty registry resolves to Decision::Install. Still not run against the real CLI's uninstall; the PR body says so. Other failures keep the manual rocm runtimes uninstall command in the message.

Comment thread xtask/src/e2e_prewarm.rs Outdated

/// Fail the pre-warm when the active runtime cannot import torchvision's operators.
///
/// A persisted runtime can pass every freshness check and still be broken: when

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These doc lines state CI history and SDK pinning facts ("before the ROCm 10 SDK pinned them", "33 minutes ... 79 to 90 minutes") that are not behavior of this code and will go stale. Keep the mechanism sentence and move the incident data to the commit message or PR body.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f2791ec: the incident data and SDK pinning history are gone from the doc comment, which now keeps the mechanism. The numbers stay in the PR body.

@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed f2791ec for the review. Your three questions, answered from the code:

  1. Is leaving the MI300X lane red until a manual reinstall deliberate? No. Hooking into the repair was small and safe, so I did it: repair_torchvision_mismatch runs right after repair_poisoned_runtimes and uninstalls the active runtime when the import fails with torchvision's operator-mismatch error, so decide sees no runtime and installs afresh. It does not act on a hang, a missing library or another import error, because nothing shows a reinstall cures those and a flaky probe would otherwise delete a healthy runtime. The check after the engine install still fails if the fresh install is itself broken.
  2. Fatal on Lemonade lanes? No longer. The default engine decides: Lemonade does not import torch (engines/lemonade/src/lib.rs reports torch_version: None), so there the finding is a warning and nothing is removed. Every other engine is treated as needing torch.
  3. Does the vLLM engine venv carry its own torchvision? No, not for a managed runtime: install_vllm_with_uv(&managed.python_executable, ..) in engines/vllm/src/install.rs installs vLLM and its torch stack into the runtime's own interpreter and creates no venv, so the runtime's Python is the right target. I did not check an adopted (non-managed) runtime.

The PR body now has the sentence on why there is no Gherkin scenario, and e2e_prewarm.rs was over the module-size limit after this round, so the check moved to xtask/src/e2e_prewarm_vision.rs.

@tomastola
tomastola requested a review from juhovainio November 17, 2026 13:34

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blocking defects found in the probe/repair logic itself; requesting changes for the items below, mostly test-honesty and one unbounded wait.

Should fix

  • The reader-thread join after the kill has no deadline. A grandchild holding the pipes would hang the pre-warm past PROBE_TIMEOUT, and the comment justifying it ("starts no child of its own") does not name the code that makes it true. Bound the join or detach the readers after the kill (inline).
  • The final decide(...) == Install assertion in the mismatch-repair test cannot go red from this change, since the fake rocm never edits the registry (inline).
  • The "only a real mismatch is uninstalled" table has a duplicated, mislabelled case and lacks the hang-wording case its comment promises (inline).

Worth considering

  • The repair-then-engine-reinstall path is untested, and the surviving folder after uninstall is unchecked (the PR body lists this as not verified; fine if so intended).
  • The source-text ordering test matches exact lines of run and will break on harmless edits.
  • Two probes of up to 120 s each now run on every pre-warm.
  • The mismatch signature is matched only against the last three lines of stderr.
  • The PR body checklist says "no file is added", but xtask/src/e2e_prewarm_vision.rs is new.

CI: lint, clippy, prek, lychee, commit-signatures and hawkeye were green when checked; build-and-test, Windows and coverage were still pending.

Comment thread xtask/src/e2e_prewarm_vision.rs Outdated
break Some(status);
}
if Instant::now() >= deadline {
// The probe starts no child of its own, so killing it closes both

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The join on the reader threads after this block has no deadline: if a hung import left a grandchild holding the pipes, the pre-warm blocks past PROBE_TIMEOUT. "The probe starts no child of its own" is the reason this is safe, but it does not name the code that makes it true. Bound the join, or detach the readers after the kill.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9e5a9e5. The readers now send their bytes over a channel and the probe waits at most PIPE_GRACE (2 s) for each after the probe ends; a reader still blocked on a pipe something else holds is detached and ends when the pipe closes. The "starts no child of its own" comment is gone. Test: a stand-in interpreter starts a background process that inherits the pipes and then sleeps; the probe returns within 30 s of a 300 ms deadline, and with the grace set to 100 s the test fails.

// What the uninstall leaves behind, per `repair_poisoned_runtimes`, is a
// registry with no runtime for the channel, and that resolves to a fresh
// install (`no_managed_runtime_installs`).
assert_eq!(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assert does not depend on the repair (the fake rocm never edits the registry), so it cannot go red from this PR's code. Drop it, or say it only pins decide.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The stand-in rocm now removes the registry entry and the active marker on runtimes uninstall <key>, and the test asserts both are gone (and active_runtime reads None), so it fails when the uninstall is not requested for that key. The decide(...) == Install assertion is dropped from this test; that is decide's own behavior, pinned by no_managed_runtime_installs in e2e_prewarm.rs.

Comment thread xtask/src/e2e_prewarm_vision.rs Outdated
("vllm", "exit 0", "a working runtime"),
("vllm", "echo absent", "no torchvision"),
("lemonade", MISMATCH, "an engine that does not import torch"),
("vllm", "exec sleep 0", "a runtime that imports"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates the "a working runtime" case and the label does not describe it; the comment above promises a hang-wording case that is not in the table. Add one (Broken text containing "did not finish") to pin that a hang is not uninstalled.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 9e5a9e5. The duplicate is replaced by a real hang case: a stand-in interpreter that never finishes, run with a 300 ms deadline, so the probe produces its own "did not finish" text, and the test asserts nothing is uninstalled. The labels now describe their cases.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent pass over the diff at f2791ec, on top of the existing review thread.

New, currently blocking: this head commit does not compile its own test target on Windows. xtask/src/e2e_prewarm_vision.rs:385 (use crate::e2e_prewarm::{Decision, decide};) and :660 (const MISMATCH) are unconditional, but every consumer of either is behind #[cfg(unix)]. With -D warnings that's unused_imports + dead_code, and it's failing for real right now: the "Windows build and test (debug)" job and the aggregate "windows-build-and-test" check are both red on this commit (job 134076126582), with exactly those two errors. Left inline on both spots with a suggested fix. This wasn't mentioned in the review round that just went in at 13:48, which looked at CI before this Windows result had landed.

Corroborating the open human-review thread (verified independently against the code, not just taken on faith — all three still hold at this head and I have nothing to add beyond them):

  • The reader-thread .join()s after the timeout-kill (probe_vision_import, ~line 131-132) have no bound of their own; a grandchild of the probed interpreter holding the pipes open would stall past PROBE_TIMEOUT, and nothing in the probe prevents the probed script from execing or forking.
  • a_torch_and_torchvision_mismatch_is_uninstalled_so_the_next_decision_installs's closing decide("runtimes:\n", "release", None) == Decision::Install assertion calls decide on a hardcoded literal, not on anything the fake rocm/the repair actually produced — it can't regress with the repair logic.
  • only_a_real_mismatch_on_a_torch_engine_is_uninstalled's ("vllm", "exec sleep 0", "a runtime that imports") case is functionally the same as the preceding exit 0 case (sleep 0 isn't a hang) and mislabeled, and the "a hang's wording" case the comment above the table promises isn't actually in the array.

I also checked the three open questions from the thread (MI300X-red-until-reinstall, Lemonade fatality, vLLM venv) against the code as it stands after f2791ec (repair_torchvision_mismatch, engine_imports_torch, and install_vllm_with_uv in engines/vllm/src/install.rs) and they match what's in tomastola's comment — no discrepancy found there.

Nothing further to add beyond the Windows build break and the existing thread; the probe's timeout/kill structure, the library-path precedence, and the mismatch-signature matching are otherwise sound.

🤖 Automated by an AI agent

Comment thread xtask/src/e2e_prewarm_vision.rs Outdated
#[cfg(test)]
mod tests {
use super::*;
use crate::e2e_prewarm::{Decision, decide};

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This use (and the const MISMATCH at line 660) are unconditional, but every test that consumes Decision/decide and every test that consumes MISMATCH is #[cfg(unix)]-gated. On a non-unix target both become unused, and with -D warnings that's a hard compile error, not a lint.

This is exactly what's happening right now on the PR's current head (f2791ec): the "Windows build and test (debug)" job (and the aggregate "windows-build-and-test" gate) are failing with

error: unused imports: `Decision` and `decide`
  --> xtask\src\e2e_prewarm_vision.rs:385:30
error: constant `MISMATCH` is never used
  --> xtask\src\e2e_prewarm_vision.rs:660:11
error: could not compile `xtask` (bin "xtask" test) due to 2 previous errors

(job 134076126582, run 44093935870, at this commit). This blocks CI as it stands. Gating the use statement and the const themselves with #[cfg(unix)] (or moving the unix-only tests, including the shared MISMATCH/fake_rocm/uninstalls helpers, into their own #[cfg(unix)] mod beneath mod tests) should fix it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already fixed: the Decision/decide import was removed in 9e5a9e5 (the test module no longer uses them) and MISMATCH is #[cfg(unix)] since 05518c5.

}
}

const MISMATCH: &str =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same issue as the use crate::e2e_prewarm::{Decision, decide}; at line 385: this constant has no #[cfg(unix)] of its own, but its only three call sites (lines ~667, ~707, ~736) are all inside #[cfg(unix)]-gated tests, so it's dead code on Windows under -D warnings. See the sibling comment on line 385 for the current CI failure this produces.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05518c5: MISMATCH is #[cfg(unix)].

@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed 9e5a9e5 for the second review round; the inline threads have their replies. On the items under "worth considering":

  • Source-order test: loosened. It now drops comment lines and asserts the order of the calls (repair, then dangling repair, then the decision, then the engine check, then the verify, then the reuse return), so formatting and unrelated edits do not break it.
  • Mismatch signature: matched against all of stderr; the first line carrying it is kept in the error tail even when it is above the last three lines.
  • PR body checklist: corrected; e2e_prewarm_vision.rs is new and its architecture.md entry exists.
  • Two probes of up to 120 s: the bound is the worst case, a hung import; a normal import takes about 2 s, so a healthy pre-warm pays two short probes. The first probe runs only when the default engine imports torch and engines list works, so Lemonade lanes pay one. I left the timeout as is rather than guess a tighter figure with no measurement of a slow cold import on these runners; it is one constant (PROBE_TIMEOUT) to lower if you would rather.
  • Repair-then-reinstall path: still untested end to end, and the PR body says so. The tests show the uninstall is requested with the right arguments and that the registry entry is gone afterwards in the stand-in; nothing runs the real rocm runtimes uninstall and then the install it leads to, or checks whether a runtime folder survives the uninstall. The first MI300X run on a tree with a mismatched runtime will be that test.

Also: tests that write and run a script now hold a lock, because a concurrent fork made one fail with ETXTBSY once the suite had more of them.

@tomastola
tomastola requested a review from juhovainio November 17, 2026 14:23

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I went through the whole change. One thing blocks merge, and a few others are worth a look.

Blocking

  • The Windows build and test (debug) job fails at the Windows clippy step with constant MISMATCH is never used. The const is only used by #[cfg(unix)] tests, so it is dead code on Windows. Gating it with #[cfg(unix)] fixes it (inline comment).

Worth addressing or answering

  • The torchvision repair runs before decide, so it also uninstalls a pinned or ahead_of_index runtime that docs/ci-hardware-testing.md says is never rolled back or offered a repair. If that is intended, say so in the doc bullet. If not, fail with the advice instead of uninstalling.
  • The repair runs before the engine install. A torchvision mismatch that the engine install itself creates on a warm runtime fails the first run and is only repaired on the next. Either re-check within the same run or note it.
  • "torchvision::" is a loose substring and now decides whether a runtime gets deleted. Consider anchoring it more tightly.
  • The Unconfigured branch (no library dirs recorded in the manifest) is a warning, so a genuinely broken runtime passes. That is a documented tradeoff, but it fails open.
  • the_probe_imports_torchvision_ops_in_a_real_interpreter hard-requires python3 on PATH and fails instead of skipping on an image without it.
  • The first commit's body ("names what to remove") no longer matches the branch head, where a mismatch triggers an automatic uninstall. Since the commit messages land in history, it is worth rewording before merge.
  • rocm runtimes uninstall <key> --yes on the active runtime is only exercised with a shell stand-in, as the PR notes. If the real CLI refuses or prompts for an active runtime, the repair path aborts the pre-warm.

Docs, the architecture entry and the leak scan all look fine. This is from a read of the diff, not a local build or run.

}
}

const MISMATCH: &str =

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MISMATCH is only used by #[cfg(unix)] tests, so on Windows it is dead code and the Windows clippy step (-D warnings) fails with "constant MISMATCH is never used". Gate the const with #[cfg(unix)].

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 05518c5: MISMATCH is now #[cfg(unix)] like its only users. The use crate::e2e_prewarm::{Decision, decide}; the earlier Windows failure named was already removed in 9e5a9e5; the 9e5a9e5 job log (134106002857) shows MISMATCH as the only error. I could not run cargo clippy --target x86_64-pc-windows-gnu here (a dependency build needs x86_64-w64-mingw32-gcc, which is not installed), so I read every cfg in the test module instead: the non-unix items (tree_with_python, tree_with_runtime, use super::*) are all used by non-gated tests, and everything else that only unix tests use is gated. The Windows job on the new head is the confirmation.

"::warning::pre-warm: removing the shared runtime {key} — it cannot `import \
torchvision.ops` ({tail}), so its torch and torchvision do not match. A reinstall follows."
);
rocm_command(rocm, prewarm_dir)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This uninstall runs before decide, so it also removes a pinned or ahead_of_index runtime that docs/ci-hardware-testing.md says is never rolled back or offered a repair. Is that intended? If so, say so in the doc bullet; if not, fail with the advice instead of uninstalling.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not intended, and I have not changed it in this commit. A pinned runtime is fine: after the uninstall decide finds no runtime for the pin and installs that pin again. An ahead_of_index runtime is the problem: a reinstall resolves to the index's version, which is older, so the repair would downgrade a runtime the doc says is never rolled back. The repair does not consult decide at all today. The options are to skip the repair for an ahead_of_index runtime and fail with the manual advice instead, or to say in the doc bullet that it is removed; I lean to the first. I'd rather you or the engineer pick before I change behavior in a review-fix commit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed (now 22b1d4b after a history rewrite, see the PR comment). The repair declines, and returns that it declined, when the run is pinned to an exact version, when rocm update marks the runtime ahead_of_index, or when that report cannot be read (the safe side: nothing is deleted that cannot be classified). The check after the engine install then fails with the manual advice, which no longer says the pre-warm will reinstall it. The doc bullet states the exclusions. Tests: each declined case leaves the runtime and reports it, and a mismatch on a runtime the index is ahead of is removed; the pinned case goes red if the pin check is removed.

Comment thread xtask/src/e2e_prewarm_vision.rs Outdated
else:\n import torchvision.ops\n";

/// The text torchvision's failed operator registration puts in its error.
const VISION_MISMATCH_SIGNATURE: &str = "torchvision::";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"torchvision::" is a loose substring and it now gates deleting a runtime. Consider also anchoring on does not exist, so an unrelated error that mentions torchvision:: cannot trigger an uninstall.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anchored in 05518c5: the repair, the error text and the tail all go through is_vision_mismatch, which needs both torchvision:: and does not exist. A test case covers an error that names torchvision::nms without saying it is missing, and checks nothing is uninstalled.

@tomastola
tomastola requested a review from juhovainio November 17, 2026 14:35

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I read the whole diff and ran the e2e_prewarm xtask tests locally (77 pass). I found nothing blocking; the inline comments are small things you can take or leave.

Two broader notes, neither blocking:

  • The mismatch repair runs before the engine install, so it can remove a runtime that the install step would have healed on its own. The cost is a reinstall, and the rocm engines list call also runs twice per pre-warm.
  • The advice text naming rocm runtimes uninstall <key> --yes is asserted for wording only (the PR body says so).

This reflects a read of the diff plus the xtask tests; CI was still pending when I looked, and I did not exercise the GPU lanes.

// job already exports: `runtime_loader_path_entries` is what a served
// process gets, so a probe that skips it judges a healthy runtime
// unloadable.
let mut entries = rocm_core::runtime_loader_path_entries(python, library_paths);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runtime_loader_path_entries writes the amd-smi shim into the runtime tree (there is a _read_only variant). That is probably intended so the probe matches a real launch, but the comment should say the probe mutates the tree.

let stderr = drain(child.stderr.take());
let deadline = Instant::now() + timeout;
let status = loop {
if let Some(status) = child.try_wait()? {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

child.try_wait()? returns without killing the child, so an error here leaks the probe process. Kill and wait before propagating.

/// writes it as optional, so a renamed field cannot be told from that). A marker
/// or manifest that exists but cannot be read or parsed, or a marker without
/// `runtime_key` (which the CLI always writes), is an error.
fn active_runtime(prewarm_dir: &Path) -> Result<Option<ActiveRuntime>> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The active runtime is read from active.json, while the CLI's own selection prefers config.active_runtime_key and falls back to the latest installed. Can the two diverge, and if so which one does serving actually use?

VisionImport::Working => ImportCheck::Working { key },
VisionImport::Broken(tail)
if runtime.library_paths.is_empty()
&& tail.contains("cannot open shared object file") =>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"cannot open shared object file" is the Linux loader message. On Windows an empty library_paths would be classified as Broken (fatal) instead of a warning. Either scope the doc claim to Linux or make the check platform-aware.

rocm = rocm.display()
)
} else {
// `repair_torchvision_mismatch` removes this runtime at the start

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"once it is the active one" reads oddly: the runtime is already the active one at this point. The next run removes it because repair_torchvision_mismatch runs before decide.

….ops

A persisted pre-warm runtime can pass every freshness check and still be
unusable: when its torch and torchvision come from different installs,
`import torchvision` dies with `operator torchvision::nms does not exist`.
Nothing ran that import before the suite, so each vLLM serve scenario found
out for itself after waiting out its readiness timeout, and a lane that takes
about 33 minutes when healthy ran 79 to 90 minutes and failed.

The pre-warm now probes the active runtime's own Python, which is where vLLM
installs its packages, for `import torchvision.ops`, with the runtime's
recorded library directories on the loader path and a two-minute deadline:

- Before the freshness decision, a runtime whose torch and torchvision do not
  match is uninstalled, so the decision installs afresh. A pinned run, a
  runtime marked ahead of the index, and a hang or any other import error are
  left alone.
- After the engine check, which also runs when the runtime is reused, a
  runtime that still cannot import fails the pre-warm, naming the runtime and
  what to do. Where the default engine does not import torch (Lemonade) the
  finding is a warning.

A runtime without torchvision is skipped.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
Review round on the torchvision import check:

- The probe puts the runtime's recorded `rocm_sdk.library_paths` (and the
  amd-smi shim) on the loader path, as the CLI's own probes do. A library that
  still will not load is a broken runtime; the skip remains only for a manifest
  that records no library directories.
- The probe is killed after two minutes and reported, instead of stalling the
  pre-warm until the job times out.
- A runtime whose torch and torchvision do not match is uninstalled before the
  freshness decision, the same repair as for a poisoned runtime, so that decision
  sees a registry without it and installs afresh. Other import failures still
  fail the pre-warm without deleting anything.
- The finding is a warning where the default engine does not import torch
  (Lemonade).
- The check moves to its own file to keep e2e_prewarm.rs under the module-size
  limit.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
- The wait on the probe's reader threads is bounded, so a process that
  inherited the pipes cannot hold the pre-warm past the probe's deadline. The
  readers are detached and end when the pipes close.
- The mismatch signature is kept in the error tail wherever it appears in
  stderr, not only in its last three lines.
- The repair test's stand-in `rocm` removes the runtime's registry entry, so its
  final assertion fails when the uninstall is not requested; the table of cases
  that must not be uninstalled has a real hang case in place of a duplicate.
- The source-order test checks the order of the calls rather than exact lines.
- Tests that write and run a script hold a lock, since a concurrent fork makes
  the exec fail with ETXTBSY.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
- `MISMATCH` is used only by unix tests, so it is dead code under
  `-D warnings` on Windows; gate it with them.
- The operator-mismatch test requires both `torchvision::` and `does not
  exist`, since it decides whether a runtime is deleted.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
The torchvision repair runs before the freshness decision, so it would also
have removed a runtime the pre-warm never rolls back. It now declines when the
run is pinned to an exact version, when `rocm update` marks the runtime
`ahead_of_index`, or when that report cannot be read; the check after the engine
install then fails with the manual advice. The real-interpreter test skips,
rather than fails, when `python3` is not on PATH. The documentation says that a
mismatch the engine install itself creates on a reused runtime is repaired on
the next run.

Signed-off-by: Tomas Saaristola <tsaarist@amd.com>
@tomastola
tomastola force-pushed the ci/prewarm-torchvision-import-check branch from 05518c5 to 22b1d4b Compare November 17, 2026 14:59
@tomastola

Copy link
Copy Markdown
Collaborator Author

Pushed the points from the 14:32Z review. I rewrote the branch's history (force-push with a lease, tree identical to what Juho read apart from the new changes) to reword the first commit, so the commit hashes in earlier replies are stale: the branch head is now 22b1d4b.

  1. Pinned / ahead_of_index: not uninstalled any more; see the inline reply. The doc bullet matches.
  2. Mismatch the engine install creates on a warm runtime: documented, not re-checked in the same run. A same-run repair would mean re-running the decision, the install and the engine install, which is not small. The check after the engine install fails that run and the next run's repair removes the runtime; if that reinstall again ends in a mismatch, the error says the installer produced the broken pair.
  3. Anchor: both torchvision:: and does not exist are required (05518c5).
  4. python3: the real-interpreter test skips, and says so, when none is on PATH.
  5. First commit body: reworded to describe the final behaviour (automatic uninstall, the exclusions, the Lemonade warning). Every commit is signed and signed off.

Unconfigured failing open: deliberate, and narrow. It is reached only when the manifest records no library directories and the import dies on a missing shared library; the probe then has nothing to supply, so the finding says nothing about the runtime, and failing would turn a probe limitation into a red lane. It warns rather than stays silent so it is visible. A runtime that records directories and still cannot load fails the pre-warm. If a manifest with no library directories turns out to be common on these runners, the warning is the signal to revisit it.

Also: the probe now retries a spawn refused with ETXTBSY a few times. A full-suite run failed once on the_runtimes_library_directories_reach_the_probe and I could not capture why; I think it was that race, since three further full runs passed, but I did not confirm it.

Juho's five inline comments from the approving review (shim comment, killing the probe on a try_wait error, active.json against config.active_runtime_key, the Linux-only loader message, the "once it is the active one" wording) are not done in this push; none is blocking.

@tomastola
tomastola requested a review from juhovainio November 17, 2026 15:00

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change (diff origin/main...pr-1103, head 22b1d4b). No ticket named; the PR body is itself the full account of motivation and scope (MI300X E2E lane reliability), and the diff matches what it describes.

Blocking

None

Non-blocking

  • Doc comment on a test describes the next test, not itself — xtask/src/e2e_prewarm.rs:2063-2070 (test ahead_of_index_is_read_for_the_named_runtime_only)
    The doc block ("run is never called by a test, so what keeps the checks where the comments on them say is this reading of run's own code…") is attached to this trivial pure-function test but actually describes the source-order test right after it, the_import_checks_run_where_their_comments_say, which has no doc comment of its own. Looks like a copy/paste slip from inserting this test ahead of the other. Confidence 90 · mechanical · Fix: move the comment down to the test it describes.
  • Pre-existing invariant comment is now stale given a second repair step — xtask/src/e2e_prewarm.rs:1020-1021 (inside repair_dangling_active_runtime, unchanged by this PR)
    "Deliberately a second listing rather than one shared with the repair above: that repair uninstalls, which rewrites exactly the pointers read here." This was written when exactly one repair (repair_poisoned_runtimes) ran above. This PR inserts a second one, repair_torchvision_mismatch, which can equally uninstall the active runtime (its own test a_torch_and_torchvision_mismatch_is_uninstalled_so_the_next_decision_installs confirms this). The conclusion (do a second listing) still holds, but "that repair" (singular) no longer says which of the two it means. Confidence 75 · mechanical · Fix: reword to "either repair above" or similar. (Not inline-commentable: the line isn't part of this PR's diff.)
  • New torchvision-check paragraph doesn't link its implementing file — docs/ci-hardware-testing.md:580-598
    The sibling paragraph just below it links e2e_prewarm.rs; this new paragraph describes all of e2e_prewarm_vision.rs's behavior without linking it, the only place in this diff doing so is docs/architecture.md. Confidence 72 · mechanical · Fix: add the link, matching the surrounding convention.

Decisions for the author

  • Mismatch repair runs before the engine install, not after — tradeoff
    A torchvision mismatch the engine install itself would have healed on a reused runtime can instead get caught and repaired by repair_torchvision_mismatch first, costing an extra reinstall. Already raised and accepted in an earlier round of this PR and now documented in docs/ci-hardware-testing.md ("A mismatch that the engine install itself creates on a reused runtime is found by that check, fails this run, and is removed at the start of the next one"); restating here only because this review is blind to prior rounds. Still sound: re-probing after the engine install (as verify_runtime_imports does) is required regardless, to catch a mismatch the install step creates, so the two probes-per-run aren't wasted duplication.
  • engine_imports_torch is engine != "lemonade" — non-blocking-improvement
    Verified against crates/rocm-core/src/host_gpu/engine_default.rs: the *-marked default rocm engines list reports can currently only ever be "vllm" or "lemonade", so the binary split is correct today. It's an exclusion-list disguised as a two-way match, though — a future third host-default engine that also doesn't import torch (e.g. something container-based like aim, which already reports python_executable: "docker" and bypasses the managed runtime's Python entirely, though it is never the host default today) would silently be treated as needing torch unless this function is updated too.

Positive signals

  • The four concrete, specific defects raised in earlier rounds of this same PR (Windows dead-code break from unconditionally-used MISMATCH/imports; a decide() assertion in a test that called it on a hardcoded literal instead of the fake rocm's actual effect; a duplicated/mislabeled case in the mismatch-matrix test with the promised "hang" case missing; a python3-on-PATH test that hard-failed instead of skipping) are all independently confirmed fixed by direct reading of the current head, not just by commit message.
  • The probe reuses the CLI's own cross-platform loader-path helpers (rocm_core::runtime_loader_path_entries, RUNTIME_LIBRARY_PATH_ENV) instead of hand-rolling LD_LIBRARY_PATH, which is exactly why this works unmodified on the Strix/Windows lane too (confirmed: e2e-selfhosted.yml runs the same cargo xtask e2e-prewarm binary on both platforms, no separate PowerShell reimplementation).
  • The new pinned/ahead_of_index decline logic correctly mirrors decide()'s own "never roll back an ahead-of-index runtime" semantics (verified by reading decide() directly), closing the exact gap an earlier round's question raised.

Deployment notes

This only changes xtask (CI/dev tooling, not the shipped rocm/rocmd binaries), run identically on Linux and Windows self-hosted runners via cargo xtask e2e-prewarm. At review time the self-hosted GPU "E2E tests" workflow run for this exact commit was still in progress (the actual MI300X lane this PR targets); worth confirming it completes before merge, though every other gating check had already gone green.

What this covered

Full read of xtask/src/e2e_prewarm_vision.rs (new, ~1123 lines, including its test module) and xtask/src/e2e_prewarm.rs (full file plus the diff), xtask/src/main.rs, docs/architecture.md, docs/ci-hardware-testing.md — the whole diff origin/main...pr-1103 at head 22b1d4badb030003cb5fb47db26b59e8edbbf4f8 (5 files, 1241 insertions, 11 deletions).

Two independent background workers ran blind (no PR description, no review threads): one on the probe/repair/check logic in isolation, one covering agent-instruction adherence (AGENTS.md/CONTRIBUTING.md), git history/conventions, code-comment compliance, and impact analysis (docs accuracy, a Windows-equivalent search, dead-code/unused-export check). Their candidates were verified against the source before inclusion; one candidate (a claimed logic duplication with tests/e2e-cucumber/src/torch_alignment_outcome.rs) was traced and dropped — that module's engine_import_verdict is a generic last-stderr-line extractor whose correctness doesn't depend on the specific mismatch wording the way this PR's is_vision_mismatch does, so the two aren't actually coupled.

Did not run cargo test -p xtask / cargo clippy directly — no Rust toolchain in this environment. Relied on direct source reading for the logic itself, and on this PR's own live CI for build/test/clippy confirmation: at review time, build-and-test, clippy, Windows build and test (debug) (including its "Clippy (Windows targets)" step — the exact step that caught this PR's earlier Windows dead-code break), Windows release build and lifecycle, and the docs/lint/license/signature checks had all completed green on this head commit; the self-hosted GPU E2E run was still in progress (see Deployment notes).

Reconciled against this PR's own prior review rounds only to the narrow extent of checking whether their blocking findings are actually fixed at this head (done by reading the source directly, not by trusting commit messages) — not reconciled against the rest of that discussion or its reasoning.


🤖 Automated by an AI agent

Comment thread xtask/src/e2e_prewarm.rs
/// after the engine check and before the reuse return, which a warm tree
/// takes every time. Formatting and unrelated edits do not move these.
#[test]
fn ahead_of_index_is_read_for_the_named_runtime_only() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misattached doc comment: the block above this test (run is never called by a test… reading of run's own code… comment lines are dropped, each call must appear in order…) describes the_import_checks_run_where_their_comments_say below, not this 3-line pure-function test. Looks like a copy/paste slip from when this test was inserted ahead of that one. Not a functional bug — both tests are individually correct — but it misdescribes intent for whoever reads this next. Suggest moving the doc comment down to the_import_checks_run_where_their_comments_say and giving this test its own one-liner.

- activates the runtime a reuse actually means — after a repair that is the
replacement named by `target=`, not the superseded manifest the line belongs to;
- ensures the default engine is installed even when the runtime itself is reused;
- removes the active runtime before deciding when its Python cannot `import

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor doc-convention inconsistency: this paragraph (and the "then fails the pre-warm…" one below it) describes the whole new e2e_prewarm_vision.rs module's behavior but never links to it. The very next paragraph in this same file cites its implementing file as [xtask/src/e2e_prewarm.rs](../xtask/src/e2e_prewarm.rs). Not CI-enforced (lychee only validates links that exist, it doesn't require one here), but worth matching the surrounding convention — docs/architecture.md is otherwise the only place in this diff that links the new file.

@tomastola
tomastola added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 6e23942 Nov 17, 2026
40 checks passed
@tomastola
tomastola deleted the ci/prewarm-torchvision-import-check branch November 17, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants