Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 49 additions & 11 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -539,8 +539,10 @@ jobs:
# in the merge queue the selection is forced to `--workspace`, so post-merge
# `main` and every queued PR run the full suite — a required check is never
# starved and the queue cannot stall. The full Windows suite
# (the windows-debug job) also runs the whole workspace on every change, so it
# backstops any crate a graph-based selection could miss (e.g. an integration
# (the windows-debug job) also runs the whole workspace on every Rust change
# (its `heavy` filter matches every `.rs` file), and so does the Linux
# `coverage` job for every crate but e2e-cucumber, so they
# backstop any crate a graph-based selection could miss (e.g. an integration
# test that drives another crate only at runtime).
#
# This job is deliberately left off sccache, unlike the other compile-heavy
Expand Down Expand Up @@ -623,12 +625,14 @@ jobs:

- name: Test rocm-core with e2e-test-hooks
# The simulated-host-root tests (`hardware_root.rs`, and the planted-root
# probe tests in `examine.rs`) only exist with this feature, which no
# other unit-test, nextest or clippy invocation enables. Without this
# step they are not even compiled in CI, so a probe reverted to a raw
# path would pass everything automated. Runs only when rocm-core is in
# the affected selection (or the whole workspace is), since nothing
# outside the crate can change these results.
# probe tests in `examine.rs`) only exist with this feature, which the
# plain nextest step above does not enable. The `rocm` step below
# turns it on in rocm-core too but runs only rocm's own tests, and the
# feature-on clippy step compiles these tests without running them, so
# without this step a probe reverted to a raw path would pass
# everything automated. Runs only when rocm-core is in the affected
# selection (or the whole workspace is), since nothing outside the
# crate can change these results.
#
# This is a sample, not full coverage of the routed reads. The probe
# tests assert only the KFD topology (GPU node count and gfx target),
Expand Down Expand Up @@ -658,6 +662,26 @@ jobs:
|| contains(format('{0} ', steps.sel.outputs.sel), '-p rocm-core '))
run: cargo nextest run --locked -p rocm-core --lib --features e2e-test-hooks

- name: Test rocm with e2e-test-hooks
# Runs the whole rocm suite with the feature on, not only the tests
# that exist only with it (the low-VRAM hook in `gpu_policy.rs`, and
# the `system_packages.rs` tests that plant a simulated host root),
# which the feature-on clippy step compiles but only this step runs.
# Tests gated on the feature being off (`cfg(not(..))`, as in
# `dash.rs`) are not built here; the plain nextest step runs them.
# The selection gate is the rocm-core step's: the trailing space stops
# `-p rocm-core` from satisfying a check for `-p rocm `. The step uses
# nextest so that each test is its own process: the feature-only tests
# set process-wide variables the hooks read (`ROCM_E2E_FORCE_LOW_VRAM`,
# and the simulated host root) while holding `ScopedTestEnv`'s lock,
# but the hooks read them without it, so under a threaded `cargo test`
# another test reaching a hook in that window could see them.
if: >-
needs.changes.outputs.rust == 'true'
&& (steps.sel.outputs.sel == '--workspace'
|| contains(format('{0} ', steps.sel.outputs.sel), '-p rocm '))
run: cargo nextest run --locked -p rocm --features rocm/e2e-test-hooks

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This re-runs the whole rocm suite feature-on in every test leg, which is about 1.5 min extra per leg. Fine if intended; the three feature-only tests could be selected with a filter, at the cost of a looser contract pin.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intended, so no code change. The whole rocm suite runs with the feature on because ungated code also behaves differently in that build (the hooks are compiled in and read environment variables), and that build is what the e2e lanes run. A -E filter would silently drop a test gated on the feature later. The contract pins this exact command so that || true, --no-run or a filter cannot satisfy it. The cost is about 1.5 min per test leg, inside the job's 20-minute timeout.


# The Windows lane is two parallel jobs (`windows-debug`, `windows-release`)
# joined by `windows-build-and-test` below. One serial job was the longest pole
# of the whole run (it alone decided when CI went green); the debug side
Expand Down Expand Up @@ -777,9 +801,12 @@ jobs:
# of every platform conditional and never sees the `cfg(windows)` side.
# Windows-gated code was therefore outside the warnings-as-errors gate
# entirely — not merely under-linted — so a lint could land there and no
# check in the set could fail. This mirrors the two invocations the Linux
# `clippy` job runs; the second is separate because the `e2e` target sets
# `test = false`, so `--all-targets` does not build it.
# check in the set could fail. This mirrors the Linux `clippy` job's
# plain workspace pass and its e2e-harness pass. The e2e-harness pass is
# separate because the `e2e` target sets `test = false`, so
# `--all-targets` does not build it. The `e2e-test-hooks` feature-on
# pass runs on Linux only, so the feature's code as compiled for Windows
# is not linted by any CI clippy step.
#
# Runs before the build so a lint failure costs seconds rather than the
# full build and test cycle.
Expand Down Expand Up @@ -1447,6 +1474,17 @@ jobs:
if: needs.changes.outputs.rust == 'true'
run: cargo clippy --locked --workspace --all-targets -- -D warnings

# A second pass, not a replacement: each `e2e-test-hooks` seam has a
# feature-off half (the `cfg(not(..))` fallbacks in, e.g., comfyui.rs,
# service_control.rs, amd_smi.rs and hardware_root.rs) that only the run
# above compiles, and a feature-on half, including the tests that exist
# only with the feature, that only this one does. Enabling it on `rocm`
# turns it on in rocm-core and the engine crates as well, so the
# rocm-core and Lemonade seams are linted here too.
- name: Clippy (e2e-test-hooks)
if: needs.changes.outputs.rust == 'true'
run: cargo clippy --locked --workspace --all-targets --features rocm/e2e-test-hooks -- -D warnings

# The e2e-cucumber `e2e` target sets `test = false` (so nextest/`cargo test
# --all-targets` skip the custom harness), which also excludes it from
# `clippy --all-targets` above — leaving the ~1.6k lines of harness + step
Expand Down
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -399,9 +399,12 @@ Minimum quality gate before upstream-ready status:
```bash
cargo test --workspace --all-targets
cargo clippy --workspace --all-targets -- -D warnings
cargo clippy --workspace --all-targets --features rocm/e2e-test-hooks -- -D warnings
cargo xtask smoke
```

The feature-on test steps are not part of this minimum gate: CI's `test` job runs them, and the pre-push hooks build with `e2e-test-hooks` off, so when a change touches code or tests behind `e2e-test-hooks`, also run the commands in [CI test selection](docs/testing.md#ci-test-selection) by hand.

Deleting or weakening a test is a change in its own right: every crate has a committed
coverage floor in `coverage-floors.toml`, and `cargo xtask coverage` fails when one drops
below it. If a drop is legitimate, re-bless with `cargo xtask coverage --bless` and say in
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,9 @@ prek install # fast checks on every commit
prek install -t pre-push # heavier checks on push (clippy + tests of the crates the push affects)
```

`prek` runs the same checks locally that CI enforces: `cargo fmt`, `clippy`, `cargo test`, `ruff` (Python), `shellcheck` (shell), PowerShell syntax, license headers (`hawkeye`), markdown links (`lychee`), the module map (`cargo xtask check-architecture-doc`), and the generated manifests (`MANIFEST.md`, `THIRD_PARTY_NOTICES.txt`).
`prek` runs the same checks locally that CI enforces: `cargo fmt`, `clippy`, `cargo test`, `ruff` (Python), `shellcheck` (shell), PowerShell syntax, license headers (`hawkeye`), markdown links (`lychee`), the module map (`cargo xtask check-architecture-doc`), and the generated manifests (`MANIFEST.md`, `THIRD_PARTY_NOTICES.txt`). Its `clippy` and `cargo test` build with the `e2e-test-hooks` feature off, so CI's feature-on clippy step and the feature-on test steps run only in CI, or by hand with the commands in `docs/testing.md`.

The pre-push hooks (`cargo clippy`, the e2e harness clippy and `cargo test`) check only the crates your push affects: the crates whose files changed plus every crate that depends on them, selected the way CI's `cargo xtask affected` does. They check the whole workspace when a change cannot be confined to specific crates (the lockfile, the root manifest, the toolchain file, workflow files, a doc an xtask test reads, or a path no crate owns), and when the affected crates cannot be determined at all, for example because the remote branch tip is not fetched locally or the pushed range lists no changed files. They also add the crates whose tests read files from outside their own crate: any Rust file adds `xtask`, whose contract tests scan every source file, and `README.md`, `docs/` and `skills/` add `rocm` and `xtask`. Each step prints the scope it chose and why. A step that already passed on this exact content is skipped, and says so: after a green run it records a stamp keyed on the git tree hash of `HEAD`, the exact cargo command (so a pass for a narrow selection never covers a wider one), the toolchain and the `CARGO_*` and `RUST*` environment, under the clone's `.git` directory where every worktree shares it, so a retried push, a manual run before the push or an amend that only rewords the message does not pay for the same checks twice. Stamps are only used and written when the working tree is clean (cargo checks the working tree, not the pushed commit, and an untracked file can hide a missing `git add`), expire after seven days, and do not cover your user-level cargo configuration, installed tools, files git ignores or flaky tests. A skip shows that this tree passed those checks earlier, not that they ran during this push; the `cargo test --workspace` and `cargo clippy --workspace` commands you run by hand are unaffected, and `--rerun` gives a fresh run when you want proof the checks ran for this push. `cargo xtask pre-push-check <step> --rerun`, or `ROCM_PRE_PUSH_RERUN=1` for the hook, ignores them, drops this step's earlier pass so a failed rerun cannot leave it to skip the next run, and records a fresh pass when the step succeeds; `SKIP=<hook id>` still skips a hook outright, and the commit-signature hook runs on every push. A push that changes no Rust file runs none of the cargo steps, whatever else it changes (a lockfile-only push included). A green hook is an early warning and not the gate: on pull requests CI lints the whole workspace, but its Linux test job runs only the affected crates, and the whole workspace's tests run in the Windows debug job, in the merge queue and on `main`. To see what a push would check without compiling anything, run `cargo xtask pre-push-check test --dry-run --base origin/main` (the steps are `clippy`, `clippy-e2e` and `test`).
The pre-push hooks (`cargo clippy`, the e2e harness clippy and `cargo test`) check only the crates your push affects: the crates whose files changed plus every crate that depends on them, selected the way CI's `cargo xtask affected` does. They check the whole workspace when a change cannot be confined to specific crates (the lockfile, the root manifest, the toolchain file, workflow files, a doc an xtask test reads, or a path no crate owns), and when the affected crates cannot be determined at all, for example because the remote branch tip is not fetched locally or the pushed range lists no changed files. They also add the crates whose tests read files from outside their own crate: any Rust file adds `xtask`, whose contract tests scan every source file, and `README.md`, `docs/` and `skills/` add `rocm` and `xtask`. Each step prints the scope it chose and why. A step that already passed on this exact content is skipped, and says so: after a green run it records a stamp keyed on the git tree hash of `HEAD`, the exact cargo command (so a pass for a narrow selection never covers a wider one), the toolchain and the `CARGO_*` and `RUST*` environment, under the clone's `.git` directory where every worktree shares it, so a retried push, a manual run before the push or an amend that only rewords the message does not pay for the same checks twice. Stamps are only used and written when the working tree is clean (cargo checks the working tree, not the pushed commit, and an untracked file can hide a missing `git add`), expire after seven days, and do not cover your user-level cargo configuration, installed tools, files git ignores or flaky tests. A skip shows that this tree passed those checks earlier, not that they ran during this push; the `cargo test --workspace` and `cargo clippy --workspace` commands you run by hand are unaffected, and `--rerun` gives a fresh run when you want proof the checks ran for this push. `cargo xtask pre-push-check <step> --rerun`, or `ROCM_PRE_PUSH_RERUN=1` for the hook, ignores them, drops this step's earlier pass so a failed rerun cannot leave it to skip the next run, and records a fresh pass when the step succeeds; `SKIP=<hook id>` still skips a hook outright, and the commit-signature hook runs on every push. A push that changes no Rust file runs none of the cargo steps, whatever else it changes (a lockfile-only push included). A green hook is an early warning and not the gate: on pull requests CI lints the whole workspace, but its Linux test job runs only the affected crates, and the whole workspace's tests run in the Windows debug job on every Rust change, in the merge queue and on `main`, and the Linux coverage job runs every crate's tests but `e2e-cucumber`'s on every Rust change. To see what a push would check without compiling anything, run `cargo xtask pre-push-check test --dry-run --base origin/main` (the steps are `clippy`, `clippy-e2e` and `test`).

The manifest hooks only run when you change the dependency graph. They *rewrite* the generated file instead of only reporting that it is stale. When that happens, the commit stops so you can re-stage the refreshed file.

Expand Down
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 3 additions & 3 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -747,7 +747,7 @@ Repository tasks, run through the workspace alias `cargo xtask <command>`: relea
| [`eval_fixtures.rs`](../xtask/src/eval_fixtures.rs) | `cargo xtask eval-fixtures`: regenerates or checks the recorded `rocm` output the rocm-doctor skill evals replay, rendered from the compiled catalog. | flat |
| [`lychee.rs`](../xtask/src/lychee.rs) | `cargo xtask lychee`: runs lychee with [`lychee.toml`](../lychee.toml) over the working tree, the local counterpart of the `docs-links` CI job. | flat |
| [`manifest.rs`](../xtask/src/manifest.rs) | `cargo xtask manifest`: regenerates or checks the Cargo dependency table in [`MANIFEST.md`](../MANIFEST.md) from `cargo metadata`. | flat |
| [`module_size.rs`](../xtask/src/module_size.rs) | `cargo xtask check-module-size`: fails when a source file exceeds `MAX_PRODUCTION_LINES` production lines, or when a `.rs` file under a crate's `src/` is declared by no `mod`. | flat |
| [`module_size.rs`](../xtask/src/module_size.rs) | `cargo xtask check-module-size`: fails when a source file exceeds `MAX_PRODUCTION_LINES` production lines, or when a `.rs` file under a crate's `src/` is declared by no `mod`. Also provides a test-only walk of every package's module tree that [`workflow_contract.rs`](../xtask/src/workflow_contract.rs) reuses. | flat |
| [`package.rs`](../xtask/src/package.rs) | `cargo xtask package`: builds the release bundle directory from built binaries, its archive, the SHA-256 sidecar and, when a key is configured, a detached signature. | flat |
| [`paths.rs`](../xtask/src/paths.rs) | Workspace paths shared by the tasks: the workspace root, the active `target/` directory, release binary locations and platform executable names. | flat |
| [`powershell.rs`](../xtask/src/powershell.rs) | `cargo xtask powershell-lint`: runs PSScriptAnalyzer over `.ps1` files through `pwsh` or Windows PowerShell, which also reports syntax errors. | flat |
Expand All @@ -764,11 +764,11 @@ Repository tasks, run through the workspace alias `cargo xtask <command>`: relea
| [`tpn.rs`](../xtask/src/tpn.rs) | `cargo xtask tpn`: regenerates or checks [`THIRD_PARTY_NOTICES.txt`](../THIRD_PARTY_NOTICES.txt) through cargo-about with the committed config and template. | flat |
| [`verify_commits.rs`](../xtask/src/verify_commits.rs) | `cargo xtask verify-commits`: checks that every commit in a range is signed and carries a DCO `Signed-off-by` trailer, or only that signing is configured. | flat |
| [`verify_pinned_keys.rs`](../xtask/src/verify_pinned_keys.rs) | `cargo xtask verify-pinned-keys`: checks that the public keys pinned in the installers and the CLI match the canonical keys under `docs/keys/`. | flat |
| [`workflow_contract.rs`](../xtask/src/workflow_contract.rs) | Source-scan tests over the GitHub workflow files and CI scripts: lane placement and concurrency groups, runner labels, timeouts, permissions and other CI invariants. | flat |
| [`workflow_contract.rs`](../xtask/src/workflow_contract.rs) | Source-scan tests over the GitHub workflow files and CI scripts: lane placement and concurrency groups, runner labels, timeouts, permissions and other CI invariants, including that CI lints and runs every crate's `e2e-test-hooks`-only tests, found by walking the module tree [`module_size.rs`](../xtask/src/module_size.rs) walks. | flat |

Boundaries:

- The eight `*_contract.rs` files define no command: each is one `#[cfg(test)]` module run by `cargo test -p xtask`, and [`rust_source.rs`](../xtask/src/rust_source.rs) is declared test-only in [`main.rs`](../xtask/src/main.rs). The pull-request test selection narrows to the crates a change touches, so a change confined to another crate may not select them there; pushes to `main` and the merge queue run the whole workspace.
- The eight `*_contract.rs` files define no command: each is one `#[cfg(test)]` module run by `cargo test -p xtask`, and [`rust_source.rs`](../xtask/src/rust_source.rs) is declared test-only in [`main.rs`](../xtask/src/main.rs). The pull-request test selection narrows to the crates a change touches, so a change confined to another crate may not select them there; the `windows-debug` job still runs the whole workspace's tests on every Rust change, as do pushes to `main` and the merge queue, and the Linux `coverage` job runs every crate's tests but `e2e-cucumber`'s.
- [`affected.rs`](../xtask/src/affected.rs) is the one crate selection used by both CI's affected-crates job and [`prepush.rs`](../xtask/src/prepush.rs), which only widens it with `CROSS_CRATE_READS`; a change to how files are attributed to crates changes both.
- [`paths.rs`](../xtask/src/paths.rs) defines the workspace root, the active target directory and binary file names, honouring `CARGO_TARGET_DIR` and a cross-compilation target; a new task uses it rather than recomputing them.

Expand Down
Loading
Loading