Skip to content

test(core): prove the env-restore guard actually restores (ROCMAI-536) - #511

Open
rominf wants to merge 4 commits into
mainfrom
test/env-guard-restore-coverage
Open

rominf wants to merge 4 commits into
mainfrom
test/env-guard-restore-coverage

Conversation

@rominf

@rominf rominf commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

RestoredEnvVar (in crates/rocm-core/src/test_env.rs, added in #337) exists to put an environment variable back when a test unwinds, so a panicking test can't leak a value into the next test that takes the same lock. Nothing tested that it does. Review of #337 showed three mutants that left the whole rocm-core suite green:

  • a Drop that does nothing;
  • a capture that conflates an unset variable with an empty one;
  • a capture taken after the mutation.

Two tests for the restore

  • dropping_the_guard_restores_the_previous_state covers both prior states, because they catch different mistakes.
    • A key that was unset must come back unset: not empty and not still planted. That catches the conflation and the late capture.
    • A key that was set must get its exact value back. That catches the no-op Drop.
    • It also asserts the value was actually planted.
  • unwinding_past_the_guard_restores_the_previous_state panics while holding the guard, inside catch_unwind, and asserts the restore.
    • The panic is raised with resume_unwind, so a passing run prints nothing.
    • Its payload carries the value seen just before the panic, so catch_unwind cannot accept an unrelated panic.
    • Only an unwind catches a Drop that skips the restore while std::thread::panicking().

Cleaning up the probe on failure

UnsetKeyOnExit is a test-only drop guard. It removes the probe key on every exit, including a failed assertion; without it, a failing test would leak the probe. It is built with UnsetKeyOnExit::arm(&lock), which takes the held lock guard by reference, so arming one without a lock does not compile.

Its Drop mutates the environment where the env-mutation guard cannot see it. So UnsetKeyOnExit::arm( joins RestoredEnvVar::set( in the guard's MUTATIONS list, under the existing rule for calls. holding_a_mutating_guard_still_needs_the_lock pins that an unguarded arm is flagged and a guarded one is not.

The same rule now covers EnvVarGuard::set(, a pre-existing guard in crates/rocm-core/src/examine/framework.rs's tests that sets a variable on construction and restores it in Drop. Its one call site already takes the lock, but a second unguarded one would have passed the scan. every_mutating_call_is_flagged_inside_an_unguarded_test now covers it as a held guard. This was added after r0x0r's approval, in reply to a later review comment. Also added after that approval: every_listed_mutation_is_still_called_in_the_tree fails when a MUTATIONS entry no longer matches any call in the tree (on code with literals and comments stripped), so renaming a wrapper or guard cannot silently drop it out of the scan.

docs/testing.md

  • "Taken" now says "acquired": a *_TEST_LOCK counts only with a .lock( in the same statement, so let _x = &SOME_TEST_LOCK; names a lock without holding it.
  • The delegation blind-spot bullet now says that a guard whose Drop mutates must have the call that builds it listed in MUTATIONS, and should be built through a named call rather than as a bare unit struct.

Test plan

  • The test(core): stop env-mutating tests racing on a shared process global #337 mutants: each of the three (no-op Drop, empty-for-unset capture, late capture) fails a new test. So do a set that plants nothing and a Drop that skips the restore while panicking; the last one fails only the unwind test.
  • The lock: removing it from either test fails a_test_mutating_the_environment_serializes_itself, naming every mutation in it, UnsetKeyOnExit::arm included.
  • The MUTATIONS entry: changing it fails holding_a_mutating_guard_still_needs_the_lock.
  • EnvVarGuard::set(: with the lock removed from asking_to_skip_the_frameworks_is_recorded_as_skipped, a_test_mutating_the_environment_serializes_itself fails at framework.rs:524: EnvVarGuard::set; with it in place, it passes. Re-ran cargo test -p xtask -p rocm-core after merging current main (all pass).
  • Gate:
    • cargo fmt --all --check
    • cargo test -p xtask (248, 46 of them env_mutation)
    • cargo test -p rocm-core --lib (605 passed, 1 ignored)
    • cargo test -p e2e-cucumber --lib (142) and --test feature_naming (4)
    • cargo clippy --workspace --all-targets -- -D warnings on Linux, and for x86_64-pc-windows-gnu against this branch merged with current main

This is a test-, tooling- and docs-only change, so no user-observable behaviour changes and no scenario is needed (AGENTS.md §3).

  • Not a bug fix in shipped code; no xfail rows apply.
  • No new subcommand or subsystem.
  • No user-facing message added or changed.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point of this PR is that the test is a real proof, so that is what I checked. It holds up. I reverted the restore three separate ways in a local checkout and ran the new test against each: a Drop that does nothing, a capture that turns an unset variable into an empty one, and a capture taken after the mutation rather than before. All three fail, and the assertion messages point straight at what broke (Some("planted") vs None, Some("") vs None). The unmodified version passes. I also ran the whole-tree env-mutation scan with this test in place and it passes, so the new lock really is seen and the mutations inside the test are covered rather than silently exempt.

The sentence added to the testing page is accurate against the scanner, not just plausible. It credits a *_TEST_LOCK only when a .lock( appears in the same statement, walking forward a bounded number of lines to the terminating semicolon, so "statement" is the right word: a line-only rule would reject the wrapped form that every lock site in this crate actually uses, and a self-terminating let _x = &SOME_TEST_LOCK; correctly gets no credit. The example in the docs is the same one the scanner's own test pins.

Two things I would still raise, both left inline. Neither is a correctness defect and neither changes my read that the approach here is right - they are the kind of thing worth a sentence from the author rather than a rewrite.

A couple of concerns I went looking for did not survive. The brand-new mutex rather than reusing an existing one is correct here, not a deviation: the documented rule is one lock per key, and this probe key exists nowhere else in the tree, so there is no lock to join. And the probe key cannot be picked up indirectly either - the only code in the crate that enumerates the whole environment is Windows-only, has no test caller, and asserts nothing about its contents.

Some(std::ffi::OsStr::new("planted")),
"the guard must actually set the value it was given"
);
drop(restore);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both halves drive the restore with an explicit drop(...). The reason this type exists, per the module docs right above, is that the restore has to run while a test is unwinding - and that path is never exercised here.

In practice drop(x) and an unwind run the identical destructor, so I cannot point at a bug today that this would catch and the others would not. What it would catch is the profile moving to panic = "abort": destructors stop running on panic, the guard quietly degrades back to a straight-line restore, and this test keeps passing because it never panics. A catch_unwind around a closure that panics while holding a guard would close that, and it is only a few lines. Reasonable to leave as is if you would rather not spend them - but then the test name and the module docs are promising slightly more than the test delivers.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in d94b2f55. A new test, unwinding_past_the_guard_restores_the_previous_state, panics inside catch_unwind while holding a guard, then asserts the prior value is back. The panic goes through resume_unwind, which skips the panic hook, so a passing run prints nothing. I checked your panic = "abort" case with -Zpanic-abort-tests: the existing drop test still passes there and the new one fails. The new test also fails if the guard is mem::forget-ed inside the closure. It takes the same TEST_ENV_TEST_LOCK, and removing that lock makes the env-mutation guard flag both mutations in the test.

Comment thread crates/rocm-core/src/test_env.rs Outdated
);

// SAFETY: as above.
unsafe { std::env::remove_var(KEY) };

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cleanup only runs when every assertion above passed. If either assertion in the second half fails, the function unwinds from there, restore puts the key back to before, and the probe stays set for the remainder of the cargo test process.

Nothing else in the tree reads this key, so it cannot flip another test's result - the cost is only that a test about not leaking environment state leaks a little when it fails, which is the state someone would be debugging in. Worth noting that swapping the two halves (set case first, unset case second) would leave the key unset at the end of the happy path and make this line unnecessary, though it would not help on a mid-test failure; making the failure path clean too needs a drop-guard around the body.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d94b2f55 with a drop guard (UnsetKeyOnExit), declared right after the lock in both tests. It removes the probe key on every exit, including a failed assertion, while the lock is still held, so the trailing remove_var is gone. I checked it by breaking the last assertion and running a follow-up test that expects the key to be unset: that test passes with the guard and fails without it.

@rominf
rominf dismissed volen-silo’s stale review October 5, 2026 11:27

Dismissed as stale: this review is of 36d57e2. Both findings are addressed in d94b2f5 — an unwind-path test, and a drop guard that cleans up the probe key on failure — with a reply in each thread. Please re-review the current head.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at d94b2f55. Both points from my earlier review are properly closed, and I checked them by breaking things rather than by reading.

The unwind path — closed. I wanted to be sure the new test discriminates the unwind path rather than riding on the explicit-drop test, so I made RestoredEnvVar::drop return early when std::thread::panicking(). That is a faithful stand-in for the panic = "abort" degradation I was worried about: it breaks restore-on-unwind and leaves restore-on-drop intact. dropping_the_guard_restores_the_previous_state stays green and unwinding_past_the_guard_restores_the_previous_state fails, which is exactly the signal that was missing before. A no-op Drop fails both. I also confirmed the closure is not a blind spot for the contract guard: removing the new test's lock makes the whole-tree scan report both the set_var and the RestoredEnvVar::set inside it.

Cleanup on a failing run — closed. I broke the last assertion of the drop test so it unwinds with the key still set, and put a probe test immediately after it (the runner orders by name, so an e_... probe lands between dropping_... and unwinding_...). With UnsetKeyOnExit the probe sees the key unset; with that one line removed it sees Some("before"). The control discriminates, so the drop guard is doing real work. The new unwind test cleans up on its own failure path too, which I checked separately.

Two things I would still like to see, left inline below. Neither is a bug today. The first is a test that certifies less than its name claims; the second is a hole in the very enforcement this PR exists to strengthen. I am raising them as changes requested because both are cheap to close, not because either is serious.

One thing I looked at and would not ask you to change: the four *_TEST_LOCK statics in this test binary are separate mutexes, so env-mutating tests in test_env.rs, fix.rs, examine.rs and lib.rs can still interleave under the parallel runner. That is the per-key locking design the module docs describe, it predates this PR, and the probe key is unique to this module, so nothing here makes it worse.

cargo fmt --check, cargo clippy -p rocm-core --all-targets -- -D warnings, cargo test -p rocm-core --lib (455, both parallel and --test-threads=1) and cargo test -p xtask env_mutation (45) are all clean at this head.

Comment thread crates/rocm-core/src/test_env.rs Outdated
std::panic::resume_unwind(Box::new("a test failing mid-way"));
});

assert!(unwound.is_err(), "the closure must have unwound");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing in this test checks that anything was ever planted. KEY is already "before" when the closure starts, so if RestoredEnvVar::set stopped setting the variable at all, the closure would change nothing, unwind, and this test would still pass. I made set a no-op and confirmed it: only dropping_the_guard_restores_the_previous_state fails.

catch_unwind widens the same hole from the other side. If set itself panicked on entry, is_err() is satisfied by the wrong panic, the value is still "before", and this test passes against a completely broken set. I checked that one too.

One change closes both halves — carry the observed value out in the panic payload instead of a fixed string:

    std::panic::resume_unwind(Box::new(std::env::var_os(KEY)));
});

assert_eq!(
    unwound.unwrap_err().downcast_ref::<Option<std::ffi::OsString>>(),
    Some(&Some(std::ffi::OsString::from("planted"))),
    "the closure must have unwound, with the guard's value in place"
);

The payload then proves both that the closure got as far as planting and that the panic is the one it raised. I ran this: it passes against your code and fails against the no-op set. Worth noting an assertion inside the closure would not work here — catch_unwind swallows it and is_err() still holds.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 59a1193d with your payload approach. The closure now unwinds with var_os(KEY), and the test asserts the downcast payload is Some("planted") before it checks the restore. Both of your mutants now fail it: a no-op set gives Some(Some("before")) against Some(Some("planted")), and a set that panics on entry gives None, because that panic's payload doesn't downcast. The doc comment records why an assertion inside the closure wouldn't work.

fn drop(&mut self) {
// SAFETY: dropped inside the caller's `TEST_ENV_TEST_LOCK` scope,
// and nothing else reads this key.
unsafe { std::env::remove_var(KEY) };

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This moves a remove_var out of a test body, where the contract guard watches it, into a Drop impl, where it does not. The whole-tree scan stops at the end of the test function, and UnsetKeyOnExit is not in the guard's MUTATIONS list, so any test that uses this helper mutates the environment invisibly. I wrote one — a #[test] holding UnsetKeyOnExit and no lock at all — and cargo test -p xtask env_mutation stayed green.

That is the case the docs on RestoredEnvVar thirty lines above already call out: "wrapping a mutation in a method hides it by default — a restoring helper added later is outside the guard until it is listed too." This is such a helper. It is also the mirror image of the ScopedEnvVar mistake the guard's own NAMED_SERIALIZERS comment records, where a save-and-restore type handed out an exemption it could not back.

Correct today, since both call sites lock first. But what keeps it correct is now the doc comment "Declare it after the lock", not the guard — and this PR is about making that contract enforceable rather than assumed. Adding "UnsetKeyOnExit" to MUTATIONS in xtask/src/env_mutation_contract.rs restores enforcement; note that list has a sibling test asserting every entry is detected, so it wants a fixture case alongside.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is the helper the RestoredEnvVar docs warn about. In 59a1193d, UnsetKeyOnExit is in MUTATIONS, with no ( because a unit struct is only named, never called. The every-entry fixture has a matching case, built at runtime so the xtask file doesn't match its own scan. Your experiment now fails: a #[test] holding UnsetKeyOnExit with no lock fails cargo test -p xtask env_mutation at that line. With the entry removed, the whole-tree scan goes green again and the every-entry fixture fails, so the listing can't be dropped silently. Both real call sites lock first, and the whole-tree scan still passes.

@rominf
rominf dismissed volen-silo’s stale review October 5, 2026 12:27

Dismissed as stale: this review is of d94b2f5. Both remaining points are addressed in 59a1193 — the unwind test now proves the plant through the panic payload, and UnsetKeyOnExit is listed in the env-mutation guard — with a reply in each thread. Please re-review the current head.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Third round. Both points from last time are closed, and I checked each by breaking it rather than reading it.

The unwind test now proves the plant. The payload approach went in, and both mutants die against it: a RestoredEnvVar::set that plants nothing fails on Some(Some("before")) against Some(Some("planted")), and a set that panics on entry fails with None, because that panic's payload does not downcast. A Drop that restores nothing fails the second assertion. I pinned execution order too, since the earlier false pass on this PR came from a neighbouring test cleaning up in between: the unwind test fails on its own under --exact --test-threads=1, and fails in five consecutive runs of the pair under the normal parallel runner, in every order.

The new shape still cleans up when it fails partway. With the no-op set mutant making the first assertion fail, a probe test running immediately after it (order read off the single-threaded output, not assumed) finds the key unset. Take UnsetKeyOnExit out of that test and the same probe finds "before" left behind, so the probe is live and the cleanup covers the new first-assertion path.

The cleanup helper is back under the guard. With "UnsetKeyOnExit" listed, my unlocked holder is now reported as crates/rocm-core/src/test_env.rs:184: UnsetKeyOnExit. Drop the entry and the whole-tree scan goes green again on that same probe while every_mutating_call_is_flagged_inside_an_unguarded_test fails on its length assertion, so the listing cannot be removed quietly. cargo test -p xtask and cargo test -p rocm-core --lib are green at this head, as are fmt and clippy.

Two small things the new list entry brought with it, left inline. Neither can hide a mutation and neither is wrong today; both are about the guard's own credibility and about what the next person to read the docs will do.

Comment thread xtask/src/env_mutation_contract.rs Outdated
"set_var(",
"remove_var(",
"RestoredEnvVar::set(",
"UnsetKeyOnExit",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The paragraph twenty lines up says the trailing ( "pins the match to a call rather than a prefix", and ends "Both directions have fixtures." This entry has no (, and nothing takes its place, so it matches as a plain substring. I ran the scan against three shapes that hold nothing and mutate nothing, and all three come back as offences:

let _h = UnsetKeyOnExitHelper;        -> UnsetKeyOnExit, col 9
let _h = MyUnsetKeyOnExit;            -> UnsetKeyOnExit, col 11
let _h: Option<UnsetKeyOnExit> = None;-> UnsetKeyOnExit, col 15

That is the over-reporting a_call_merely_starting_like_a_mutation_is_not_flagged exists to prevent, and whose doc calls it "the same credibility problem as a miss". It cannot mask anything real: the line is still flagged either way, just under the wrong name. So this is noise, not a hole, but it does make the sentence above untrue for the fourth entry.

The machinery is already in this file. lock_name_column's ends_word check does exactly this for _TEST_LOCK, and an_identifier_merely_carrying_the_lock_suffix_is_not_a_lock pins it. Matching this entry on an identifier boundary, plus a fixture for the non-match direction, would make "Both directions have fixtures" true again.

While you are in here: the guarded direction is not pinned for this entry either. RestoredEnvVar::set( has a_mutation_delegated_to_a_restoring_wrapper_still_needs_the_lock asserting both halves, that an unguarded call is an offence and that a locked one is not. For UnsetKeyOnExit only the unguarded half exists. The exempt half currently rests on the whole-tree scan happening to walk the two locked holders in test_env.rs, so it would vanish if those ever moved or changed shape.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 492cd82. A bare-name entry, one with no trailing (, now matches only as a whole identifier where a value of it is built or reached: the next non-blank character must be ;, (, { or ::. This is mutation_column, which shares is_identifier_char with lock_name_column. UnsetKeyOnExitHelper;, MyUnsetKeyOnExit; and Option<UnsetKeyOnExit> are no longer flagged; I treated the last as a false positive, since a type mention holds nothing. The MUTATIONS doc now limits the trailing-( claim to the entries that have one and describes the bare-name rule separately.

Both directions are now pinned:

  • a_bare_type_entry_matches_only_a_whole_constructed_name covers the three shapes, each breaking a different half of the rule.
  • holding_a_mutating_guard_still_needs_the_lock asserts that an unguarded holder is flagged and a locked one is not, mirroring the RestoredEnvVar::set test. The exempt half no longer rests on the whole-tree scan.

Mutation-checked:

  • Matching bare names as plain text again, or dropping only the check before the name, fails the shapes fixture.
  • Replacing the entry fails the guarded/unguarded fixture and every_mutating_call_is_flagged_inside_an_unguarded_test (45 passed, 2 failed).

Comment thread docs/testing.md
a lock does not cover its neighbours.
a lock does not cover its neighbours. "Taken" means acquired: a `*_TEST_LOCK`
counts only with a `.lock(` in the same statement, so `let _x = &SOME_TEST_LOCK;`
names a lock without holding it and does not satisfy the guard.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not about this hunk. The line I mean is docs/testing.md:110, in the blind-spots list a little further down, which the section this hunk edits leads into. GitHub will only let me anchor inside the diff.

That bullet is the only place a contributor is told how to keep the guard honest, and it describes exactly one shape: "a wrapper hides the mutation until the wrapper itself is named in the scanner's MUTATIONS list, as RestoredEnvVar::set( is. Add a new restoring helper to that list, or it is outside the guard."

UnsetKeyOnExit is not that shape. Nothing delegates to it and no call text carries the mutation. It is a type whose Drop mutates at scope exit, listed by bare name. Someone adding the next cleanup guard and following this bullet would list MyGuard::new( and get nothing back, which is the omission this commit just fixed in code. A sentence saying that a guard whose Drop mutates needs its type name listed too would close it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 492cd82. The blind-spots bullet now says that a guard type whose Drop mutates the environment needs its type name listed too, as UnsetKeyOnExit is: nothing calls a ::new( that could be matched instead. It also explains how such a name is matched (a whole identifier followed by ;, (, { or ::), so let _g = UnsetKeyOnExit; is flagged and Option<UnsetKeyOnExit> is not.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked both points from last round against the code, and both are closed.

The three shapes I listed are no longer reported, and each half of the new rule carries its own weight: matching bare names as plain text again, dropping the check before the name, and dropping the one after it each fail a_bare_type_entry_matches_only_a_whole_constructed_name. holding_a_mutating_guard_still_needs_the_lock now pins both directions, and removing the entry from MUTATIONS fails it alongside the every-entry fixture (45 passed, 2 failed), so the exempt half no longer depends on the whole-tree scan happening to walk the two locked holders. The docs/testing.md bullet now names the Drop-guard shape and how such a name is matched, which was the gap.

I also re-ran the three mutants from the earlier rounds to check nothing loosened: a no-op set, a set that panics on entry, and a no-op Drop each still fail both tests, and mem::forget inside the closure still fails only the unwind one. Whole-tree scan, 253 xtask tests, 455 rocm-core tests, cargo fmt --check and clippy are all clean.

One thing survives, and it is in the direction that matters more than the over-reporting it replaced: the new boundary rule also stops matching some genuine constructions. Details inline.

Comment thread xtask/src/env_mutation_contract.rs Outdated
.is_none_or(|c| !is_identifier_char(c));
let after = line[at + needle.len()..].trim_start();
starts_word
&& (after.starts_with([';', '(', '{']) || after.starts_with("::"))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rule requires the next non-blank character to be ;, (, { or ::, so a construction whose next character is ), , or ] is not matched at all. I planted each of these as an unguarded #[test] in test_env.rs and ran the whole-tree scan:

let _g = UnsetKeyOnExit;                          -> flagged
drop(UnsetKeyOnExit);                             -> green
let _g = Some(UnsetKeyOnExit);                    -> green
let _g = vec![UnsetKeyOnExit];                    -> green
let (_a, _b) = (UnsetKeyOnExit, UnsetKeyOnExit);  -> green
std::mem::forget(UnsetKeyOnExit);                 -> green

drop(UnsetKeyOnExit); builds the guard and removes the key on the spot, with no lock anywhere in the test, and the scan says nothing. All five were flagged before this commit, so this is coverage the boundary rule gave up. A miss is the failure the scan comment calls "the failure mode this guard exists to prevent", and it is worse than the noise I asked about, which could only ever mislabel a line that was flagged anyway.

The doc two lines up says a bare name counts "in a position where a value of it is built or reached". Some(UnsetKeyOnExit) and drop(UnsetKeyOnExit) are exactly that, so the stated rule and the implemented one have drifted apart. The Option<..> case I raised is a type mention rather than noise worth a hole — storing a conditionally-held guard as let mut g: Option<UnsetKeyOnExit> = None; and later g = Some(UnsetKeyOnExit); is now invisible on both lines.

Inverting it reads better: flag any whole-identifier occurrence, then carve out only the type positions — preceded by <, preceded by a type-ascription : that is not ::, or followed by >. I tried that here and it flags all six shapes above, still rejects UnsetKeyOnExitHelper, MyUnsetKeyOnExit, Option<UnsetKeyOnExit>, Vec<UnsetKeyOnExit> and fn f(_x: UnsetKeyOnExit), and leaves all 47 tests in this file green including the new shapes fixture. It errs toward reporting rather than missing, which is the side this guard should land on. Whatever shape it ends up, a_bare_type_entry_matches_only_a_whole_constructed_name would want a drop(..) or Some(..) case in the flagged direction, and the docs/testing.md sentence describing the match needs to follow.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: misses are worse than over-reports for this guard. Fixed in cccf818 by inverting the rule as you suggested.

The rule: a bare-name entry now counts at any whole-identifier occurrence unless it is in a type position, meaning right after <, right after a type-ascription : that is not part of ::, or right before >. Whitespace is skipped, so Option< T > and _x : T are type positions too.

The fixture is renamed a_bare_type_entry_matches_a_whole_name_outside_type_positions.

  • Flagged: all six of your shapes, the two-line let mut g: Option<..> / g = Some(..) reassignment, and a path-qualified test_env::UnsetKeyOnExit.
  • Not flagged: the longer and prefixed names, Option<..>, Vec<..> and _x: T parameters.
  • Also added: Result<UnsetKeyOnExit, ()> and BTreeMap<u8, UnsetKeyOnExit>. With only Vec<T>-style fixtures, neither the < nor the > carve-out could fail on its own.

Mutation checks:

  • Going back to the old rule fails all six of your shapes.
  • Removing each carve-out, or the :: exclusion, fails a fixture.
  • Removing all three carve-outs fails the type-position cases. I re-checked that one myself: 46 passed, 1 failed.

The one known over-report is a -> UnsetKeyOnExit return type inside a test body, which the docs accept. The MUTATIONS doc and the docs/testing.md bullet now describe this rule.

Separately: 492cd82a is now 1372d121. Its content is unchanged; it was reworded to drop a co-author trailer that does not belong in this repo's history.

@rominf
rominf force-pushed the test/env-guard-restore-coverage branch from 492cd82 to cccf818 Compare October 6, 2026 12:21
@rominf
rominf dismissed stale reviews from volen-silo and volen-silo October 7, 2026 06:58

Stale (left on 59a1193); both items addressed. The bare-name entry now matches on an identifier boundary with fixtures in both directions, and docs/testing.md covers Drop guards: 1372d12, refined in cccf818. Replies are in the threads.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The constructions from the previous round are now detected, and the longer-name and Option<UnsetKeyOnExit> exclusions still hold. The unwind test also still rejects a no-op setter, an early setter panic, a no-op restore, and a forgotten guard.

One ordinary construction still bypasses the lock check: a struct field initialized with UnsetKeyOnExit. The colon is treated as a type annotation even though this occurrence constructs a value. I reproduced the missed report and confirmed that the actual cleanup guard removes the variable when the containing struct drops. Details inline.

Local checks passed: 253 xtask tests, 455 rocm-core library tests, formatting, and clippy for both affected crates with warnings denied. CI is not fully green: WSL2 reports an unexpected ComfyUI download-progress timeout, Windows E2E was cancelled, and MI300X remains queued. Those results do not establish a regression from this test-only change. I did not run the full workspace suite, smoke test, or hardware E2E tests locally.

Comment thread xtask/src/env_mutation_contract.rs Outdated
let after = line[at + len..].trim_start();
let whole = !line[..at].ends_with(is_identifier_char)
&& !line[at + len..].starts_with(is_identifier_char);
let ascribed = before.ends_with(':') && !before.ends_with("::");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The : exclusion also suppresses a struct-literal field value, not just a type annotation:

struct Holder { guard: UnsetKeyOnExit }
let _holder = Holder { guard: UnsetKeyOnExit };

In an unlocked test, env_mutations_in_unserialized_tests(&unguarded_test("let _holder = Holder { guard: UnsetKeyOnExit };")) returns no offenses. The normal multiline form with guard: UnsetKeyOnExit, also returns none. I compiled and ran a probe in the real guard's test module: constructing this field and dropping the holder removes the environment variable.

The previous rule also missed this form, so this is a remaining gap in the construction fix rather than a new race in an existing test. Please distinguish field-value colons from type-ascription colons, and add single-line and multiline fixtures while retaining the non-mutating type cases. Otherwise a test can still construct this mutating guard without its lock and pass the gate, despite the updated promise to detect constructions.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, confirmed. Before this change, Holder { guard: UnsetKeyOnExit } produced no offence on one line, split across lines, or after an earlier field (Holder { n: 1, guard: … }). I reproduced it independently: with the previous rule and only the new fixture added, the scan returns [] for your exact line.

Fixed in ef39e90. No rule that looks at a single line can separate these: guard: UnsetKeyOnExit, on its own line reads the same in a split literal, a split struct and a split parameter list. So the scan now keeps a small bracket stack across lines and marks brackets opened after struct/enum as declarations.

  • Field initializer, flagged: the : sits in a { that is not a declaration, and the field name follows {, , or the start of the line.
  • Type position, not flagged: anything inside a declaration. This also stops struct H(UnsetKeyOnExit); and tuple variants from being over-reported, as they were before.

union is deliberately not a declaration keyword. It is not reserved, so a.union(&b) would otherwise hide a guard built inside that call.

New fixtures:

  • Flagged: all three literal forms.
  • Not flagged: braced, split, where-layout, tuple and enum declarations; let/const ascriptions; a split parameter list.

Each half of the rule was mutation-checked against them. The remaining over-reports are documented in the MUTATIONS doc and docs/testing.md: struct patterns, a later parameter of a typed closure, and a return type in a test body.

On the unwind thread from earlier: unwinding_past_the_guard_restores_the_previous_state uses catch_unwind/resume_unwind around RestoredEnvVar. UnsetKeyOnExit there is the test's own cleanup, dropped on normal scope exit.

@rominf
rominf dismissed volen-silo’s stale review October 8, 2026 09:47

Stale (left on cccf818). Addressed in ef39e90: a struct-literal field value (Holder { guard: UnsetKeyOnExit }, single-line, split, and after another field) is now flagged, while struct/enum declarations stay type positions, with fixtures in both directions. The reply is in the thread.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · ef39e90

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — needs work

Full review of the whole change.
No ticket named. The description asks for the two RestoredEnvVar tests and the docs/testing.md clause, and both are here. It does not mention the xtask scanner extension, which is most of the diff.

Blocking

  • new — The new bare-type matching in the scanner has branches no test pins, and breaking them makes the scanner miss real constructions — xtask/src/env_mutation_contract.rs:417, xtask/src/env_mutation_contract.rs:256, xtask/src/env_mutation_contract.rs:287, xtask/src/env_mutation_contract.rs:413
    I made each mutation below on its own and ran cargo test -p xtask env_mutation. All 48 committed tests still passed every time:

    • Removing the ; reset of declaring (:417). A test body with struct Marker; followed by drop(UnsetKeyOnExit); is then no longer flagged. The leftover declaring marks the ( as a declaration. The only fixture with a ; after struct is "struct Holder(UnsetKeyOnExit);", and nothing comes after it.
    • Changing .find(..) to "first occurrence only" (:256). let _g: UnsetKeyOnExit = UnsetKeyOnExit; is then no longer flagged, because the first occurrence is a type ascription. No fixture has a type mention and a construction on the same line.
    • Dropping .trim_start() on after (:287). BTreeMap<u8, UnsetKeyOnExit > is then flagged wrongly. The doc says the blank-padded spellings are pinned, but Option< UnsetKeyOnExit > is rejected by the < check alone.

    I added a probe test with these three shapes. It passes on HEAD and fails under each mutant. A separate case: removing the } reset at :413 changed nothing I could find, and I could not find valid Rust where } arrives while declaring is still set. That branch looks unreachable.
    The scanner is a permanent guard, and the missed constructions above are unguarded env mutations. That is exactly the failure its doc comment says it is built to avoid.
    Confidence 100 · logic · Fix: add fixtures for a local struct X; or enum item followed by a construction, for an ascription plus a construction on one line, and for a blank before >. Then pin or remove the } reset.

  • new — The PR description leaves out most of the change — PR body; xtask/src/env_mutation_contract.rs (+381 lines)
    The description covers the RestoredEnvVar test and the docs/testing.md "taken means acquired" clause. It does not mention any of these:

    • the new UnsetKeyOnExit drop guard;
    • the second (unwind) test;
    • the new MUTATIONS entry;
    • BracketContext, names_a_value and initializes_a_field;
    • three new xtask tests;
    • the long new docs/testing.md paragraph on bare-type matching.

    The description also says "adds one test" and gives test counts of 45 for xtask and 454 for rocm-core. At this head the counts are 48 and 455.
    Confidence 100 · mechanical · Fix: update the description to cover the scanner extension and its reasoning, and refresh the test plan.

Non-blocking

  • new — The new docs paragraph sits in a list it contradicts — docs/testing.md:104-128
    The list opens with "Three known blind spots, all of which the scan reports nothing for". The new text inside it then says that struct patterns, typed-closure parameters and -> UnsetKeyOnExit "are flagged although they build nothing". Those are over-reports, the opposite of blind spots.
    The same rules are also written out four times: in this page, the MUTATIONS comment, names_a_value and initializes_a_field.
    Confidence 85 · mechanical · Fix: move the bare-name rules out of the blind-spots list, keep one example, and point to the code comment for the details.

  • new — A test comment describes a rule that only ever existed on this branch — xtask/src/env_mutation_contract.rs:1089-1092
    The comment says "each one was missed by the earlier rule that required the next character to be ;, (, { or ::". That rule was added in this PR's own commit 1372d12 and replaced before HEAD. The base branch never had it, so after a squash merge no reader can find it.
    Confidence 85 · mechanical · Fix: describe the shapes on their own terms and drop the reference to the intermediate rule.

  • new — The stated reason for the unwind test does not hold — crates/rocm-core/src/test_env.rs:139-142
    The comment says that only an unwind proves the restore survives "a change such as panic = "abort"… this test aborts instead of passing". Cargo ignores the profile panic setting for test targets, so a panic = "abort" profile change would not make this test abort.
    The test itself is worth keeping. A Drop that skips the restore while std::thread::panicking() passes the explicit-drop test and fails this one (I ran that mutant).
    Confidence 85 · mechanical · Fix: give that mutant as the reason instead.

Decisions for the author

  • Making a unit struct visible to a text scanner costs about 200 lines of bracket tracking — non-blocking-improvement
    BracketContext, names_a_value, initializes_a_field, three tests and a docs paragraph all exist so that let _x = UnsetKeyOnExit;, in one private test module, is visible to a text scanner. The docs give the reason: the test "never calls a ::new(".
    A call-shaped constructor would remove the need. For example, UnsetKeyOnExit::arm() could be listed as "UnsetKeyOnExit::arm(" under the existing rule for entries ending in (, with no scanner change. A constructor that takes the lock guard by reference would go further and make an unlocked use impossible to compile.
    The chosen route is more general for future guard types. It does add a state machine to a permanent whole-tree guard: if it misreads one line, it stays desynchronised for the rest of the file, and it has documented over-reports. Confirm that trade is deliberate.

Positive signals

  • Both new RestoredEnvVar tests are effective. Five mutants (no-op Drop, late capture, empty-instead-of-unset, a set that plants nothing, and a Drop that skips on panicking()) each fail at least one of them.
  • Passing the observed value out as the resume_unwind payload is a good way to stop catch_unwind from accepting an unrelated panic.

Deployment notes

None

What this covered

I read all three changed files completely (xtask/src/env_mutation_contract.rs, crates/rocm-core/src/test_env.rs, docs/testing.md), along with the scanner's literal stripping and statement handling. Range: base…ef39e902.

Two independent workers ran: a code pass over the coupled group, and a pass covering agent instructions, history and code comments. Mutation runs: cargo test -p xtask env_mutation and cargo test -p rocm-core --lib test_env.

CI at ef39e90: every reported check passed. Two checks did not report a result for this head: Sphinx docs build (-W) and Skill checks (skillscope) were skipped.

Did not run: the prior-changes pass, because reading earlier PR review comments is outside this run's remit. The Step 7 design question was settled by the driver, not by an independent worker.

Not reconciled against any PR discussion here.

@rominf

rominf commented Oct 12, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for the review. Addressed in d7935ff8, which takes the route from your design question.

Design decision. The guard is now built with UnsetKeyOnExit::arm(&lock), and UnsetKeyOnExit::arm( is listed under the existing rule for calls. arm takes the held lock guard by reference, so an unlocked use doesn't compile; the scan still checks that the lock is a *_TEST_LOCK taken in the body.

The bare-name matching is gone: BracketContext, names_a_value, initializes_a_field, their tests and the docs paragraph. Every MUTATIONS entry ends in ( again. Before removing it, I found it also missed (r#struct, Some(UnsetKeyOnExit)), because the keyword guess read the raw identifier as struct. That miss goes away with the code.

Blocking: scanner branches with no test. Resolved by removal. The ;/} resets, the per-occurrence search and the blank-skipping no longer exist. Your three mutants did reproduce first, and the } reset was reachable through a raw identifier ending a block.

Blocking: description. Rewritten for the final scope, with counts refreshed: xtask 252 (46 env_mutation), rocm-core 455.

Docs list. The bare-name rules are gone from it. The blind-spot bullet now just says to list the call that builds a guard whose Drop mutates.

Test comment at :1089. Removed along with the test it belonged to.

Unwind test reason. Fixed in 622260f7 to cite your panicking() mutant. I re-ran it: the explicit-drop test passes and the unwind test fails.

Removing the lock from either test now fails a_test_mutating_the_environment_serializes_itself, naming UnsetKeyOnExit::arm.

@rominf

rominf commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

Rebased onto main 1e5aac8f, with no conflicts; the head is now 13292e5a. CI's Test (affected crates) builds rocm-core with #510's e2e-test-hooks feature, which a branch based on a main from before #510 lacks. The PR's diff is unchanged, and it still merges cleanly with current main.

Verified locally:

  • fmt
  • tests for rocm --bin (1035), rocm-core --features e2e-test-hooks (606), xtask (248) and e2e-cucumber lib (142), plus feature_naming
  • check --test e2e
  • both clippy invocations, on Linux and for x86_64-pc-windows-gnu

@rominf
rominf force-pushed the test/env-guard-restore-coverage branch from d7935ff to 13292e5 Compare October 13, 2026 08:28
@rominf
rominf dismissed siloteemu’s stale review October 13, 2026 09:20

This reviewed ef39e90. The head is now 13292e5: d7935ff builds the guard through UnsetKeyOnExit::arm(&lock) and removes the bracket-tracking scanner (the design route from the review), faa5e32 and 622260f address the remaining notes, and the branch is rebased onto main. All required checks are green on 13292e5.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · 13292e5

This automation never files a GitHub approval, so no approving review will
appear here whatever the outcome — the merge decision stays with a human
reviewer.

Review — no blocking findings

Full review of the whole change.
No ticket named. The description cites #337 only as where the restore guard came from, and claims to close nothing.

Blocking

  • standing — The test plan's counts do not match this head — PR body (Test plan)
    Standing from the review at ef39e90, which objected that the description left out most of the change and quoted stale test counts, and asked to refresh the test plan. This round's blind review did not re-find it. It was settled by putting it to a reviewer after the review ended, so this is an anchored check, not a blind finding.
    The scope part no longer holds: the description now covers everything in the diff. The counts part still holds. At 13292e5, cargo test -p xtask runs 248 tests, and the description says 252. cargo test -p rocm-core --lib runs 606 (605 passed, 1 ignored), and the description says 455.
    Fix: refresh the test-plan counts for this head.
    The blind review above found no new blocking finding, and the title is its outcome. This round still requests changes because part of this standing objection holds.
  • The other blocking objection from ef39e90, untested branches in the bare-type matcher, no longer holds: that code is removed. It was standing, not re-found blind, and was settled by putting it to the reviewer.

Non-blocking

  • new — A new doc comment says the restore guard has "the two callers". It has six call sites in four files — crates/rocm-core/src/test_env.rs:105-106
    RestoredEnvVar::set( is called in fix.rs:2182, lib.rs:7255, examine.rs:5504 and hardware_root.rs:160,173,182. All of these were already at the merge base. The code governs, because those are the real call sites.
    Confidence 85 · mechanical · Fix: say "its callers", or name the four modules.
  • new — The arm doc comment says the contract guard checks which lock is held. The guard's own doc says it does not — crates/rocm-core/src/test_env.rs:89-90, xtask/src/env_mutation_contract.rs:534-541
    test_env.rs says "which lock it is stays the contract guard's to check". The scanner says "The scan checks that A lock is held, not that it is THE lock every other mutator of the same key takes". The scanner's statement governs, because it describes the code: serializer_column accepts any *_TEST_LOCK with a .lock(, so arm(&OTHER_MUTEX.lock()) passes whenever some *_TEST_LOCK is taken elsewhere in the test.
    Confidence 85 · mechanical · Fix: "that some *_TEST_LOCK is taken stays the contract guard's to check."
  • new — Five commits add a mechanism that the last commit deletes in full — commits bf692f2, 732ad05, c426c6a, ad42990, 398ecbf, 13292e5
    Those five commits add bare-type-name matching to the scanner: type-position carve-outs, a cross-line bracket stack, struct-literal detection, fixtures, and a docs paragraph. 13292e5 removes all of it (+49/−383). The branch's net xtask diff is +60/−2.
    Two earlier messages also describe states that were later replaced: d15561b describes the bare UnsetKeyOnExit listing, and 9071c84 the panic = "abort" rationale that 443d3d9 retracts. AGENTS.md §11 asks for meaningful per-commit messages, and these commits do not describe the final tree.
    Confidence 85 · mechanical · Fix: fold bf692f2…398ecbf into 13292e5, or have d15561b use arm( from the start. This does not matter if the merge squashes.

Decisions for the author

  • The UnsetKeyOnExit::arm( entry in MUTATIONS only guards future code, and its link to the real constructor is unchecked — tradeoff
    • Today it catches nothing extra. Both real tests also call set_var/remove_var directly, so the whole-tree scan already flags them through those calls if the lock is dropped.
    • A rename goes unnoticed. The fixtures use the literal string, so renaming UnsetKeyOnExit or arm would leave a stale entry with every test green. RestoredEnvVar::set( already has the same property.
    • For keeping it: a later test that arms the guard without mutating directly stays visible to the scan.
    • Against: a private helper from one test module now sits in the scanner's global list, and that list's tests have to grow with it.
    • Either is defensible. Confirm the coupling is intended.

Positive signals

  • The unwind test carries the observed value in the resume_unwind payload. This closes the hole where catch_unwind would accept a panic raised by set itself before it planted anything. It also avoids swapping the process-wide panic hook.
  • arm(&MutexGuard) makes "hold a lock first" a compile-time requirement for the one way the guard is meant to be built. It does not depend on the text scan for that.

Deployment notes

None

What this covered

All three changed files were read completely: crates/rocm-core/src/test_env.rs, xtask/src/env_mutation_contract.rs (1658 lines) and the env-mutation section of docs/testing.md. Also read: every RestoredEnvVar caller, the workspace lint config and AGENTS.md. Diff reviewed: git diff prw-base...HEAD at 1e5aac8 (merge base)…13292e5a.

There were three parallel workers: the restore guard and its tests, the scanner with its docs, and the source-reading passes (instructions, history, comments, commits). Every test the PR adds or changes was then checked by mutation:

  • dropping_the_guard_restores_the_previous_state
    • Fails under: a no-op Drop, an empty-for-unset capture, a capture taken after the mutation, and a set that plants nothing.
    • Stays green under: a Drop that skips the restore while panicking. The unwind test catches that one.
  • unwinding_past_the_guard_restores_the_previous_state
    • Fails under: skip-while-panicking, a no-op Drop, a no-op set, and a late capture.
    • Stays green under: the empty-for-unset capture. The test above catches that one.
  • holding_a_mutating_guard_still_needs_the_lock and the new held loop in every_mutating_call_is_flagged_inside_an_unguarded_test
    • Both fail with UnsetKeyOnExit::arm( removed from MUTATIONS.
  • "Reverted" verdict: this change adds no production code. Every check above is a branch mutation of the existing RestoredEnvVar or MUTATIONS.
  • Commands run: cargo test -p rocm-core --lib test_env and cargo test -p xtask env_mutation, both passing at head.

CI at 13292e5:

  • Every completed check passed.
  • Skill checks and the Sphinx docs build were skipped.
  • E2E tests (MI300X) was still queued when read.

AGENTS.md §3: no regression test or scenario is owed. The change touches tests, tooling and docs only, and the description says so explicitly.

Leak scan of the diff is clean.

What did not run, and why:

  • Prior-changes pass: not run.
  • Design review (Step 7): the questions were answered in the driver from worker evidence, not by separate workers, so that part is not independent.
  • Workspace-wide test and clippy: not run locally, to keep build load down. CI's clippy and build-and-test passed.

Not reconciled against any discussion on the PR.

@rominf

rominf commented Oct 13, 2026

Copy link
Copy Markdown
Collaborator Author

On the automated review of 13292e5a, addressed in b92a03f6:

  • Test-plan counts (blocking): the description now quotes this head's counts: xtask 248 (46 env_mutation), rocm-core 605 passed with 1 ignored, and e2e-cucumber lib 142.
  • "The two callers": RestoredEnvVar's test doc now says its callers are in fix, lib, examine and hardware_root.
  • What the contract guard checks: arm's doc now says the guard checks only that some *_TEST_LOCK is taken, not which one, matching the scanner's own doc.
  • History: the repo squash-merges, so the superseded intermediate commits don't reach main.
  • Decision on the UnsetKeyOnExit::arm( entry: the coupling is intended. Today the real tests are also flagged through their direct set_var/remove_var. The entry keeps a later test that only arms the guard visible to the scan. A rename would leave a stale entry, as with RestoredEnvVar::set(. A test that each MUTATIONS entry names a real symbol would close that for both, and would make a reasonable follow-up.

@rominf
rominf dismissed siloteemu’s stale review October 13, 2026 17:55

This reviewed 13292e5. The head is now b92a03f: the description's test counts are refreshed for this head, and b92a03f corrects the two doc comments (callers in four modules; the contract guard checks that some *_TEST_LOCK is taken, not which). All required checks are green on b92a03f.

@rominf rominf changed the title test(core): prove the env-restore guard actually restores test(core): prove the env-restore guard actually restores (ROCMAI-536) Oct 26, 2026

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · b92a03f

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Ticket ROCMAI-536 named in the title and retrieved: "Prove the test env-restore guard actually restores the environment" (status: In Progress). This change implements exactly that — the two RestoredEnvVar tests plus the xtask scanner update needed to keep the new test-only cleanup guard inside the enforced env-mutation contract. The PR does not claim to close the ticket.

Blocking

  • UnsetKeyOnExit's cleanup-on-exit is new behaviour with no test verifying it actually does anything — crates/rocm-core/src/test_env.rs:97-104 (the Drop impl), crates/rocm-core/src/test_env.rs:117-192 (the two tests that construct it)
    I read every assertion in both dropping_the_guard_restores_the_previous_state and unwinding_past_the_guard_restores_the_previous_state; none inspects std::env::var_os(KEY) after _unset_on_exit drops out of scope — every assertion in both functions is about RestoredEnvVar's effect on KEY, never about UnsetKeyOnExit's. Mutating UnsetKeyOnExit::drop to a no-op (deleting its remove_var(KEY) call) would not fail either test: the first test's checks all happen before the function returns and _unset_on_exit drops; the second's checks are about the unwind payload and the post-restore value, not about the probe afterward. The PR's own stated reason for this type — "it removes the probe key on every exit, including a failed assertion; without it, a failing test would leak the probe" — is itself untested.
    Confidence 85 · logic · Fix: assert std::env::var_os(KEY).is_none() after the guard leaves scope — e.g. move the lock+guard+mutation into an inner block within one of the two tests and assert on KEY immediately after that block ends, or add a third test that panics inside catch_unwind on a false assertion and confirms the probe key is gone afterward.

Non-blocking

  • UnsetKeyOnExit::arm's doc comment implies more than the type enforces — crates/rocm-core/src/test_env.rs:88-94
    arm returns a zero-field Self and ties no lifetime to the &MutexGuard<'_, ()> it takes, so nothing stops a caller from ending the lock's scope and only then dropping the UnsetKeyOnExit value — its Drop would then call remove_var with no lock held, contrary to the struct's own doc ("Declare it after the lock so it drops while the lock is still held") and the Drop impl's SAFETY comment ("dropped inside the caller's TEST_ENV_TEST_LOCK scope"). arm's doc — "Takes the caller's lock guard by reference, so arming one without holding a lock does not compile" — reads as if that ordering were compiler-enforced; it is only enforced by convention. Both current call sites get the ordering right, so there is no live bug today.
    Confidence 70 · mechanical · Fix: soften arm's doc to say it only requires a lock guard in hand at the call site, not that the lock stays held for the guard's lifetime; or tie a lifetime to Self (e.g. struct UnsetKeyOnExit<'a>(PhantomData<&'a ()>), fn arm<'a>(_lock: &'a MutexGuard<'_, ()>) -> Self) so the borrow checker rejects dropping the lock first.

Decisions for the author

None

Positive signals

  • The unwind test carries the pre-panic observed value as the resume_unwind payload, closing the hole where catch_unwind would otherwise accept any panic — including one raised by set itself before it planted anything.
  • All 11 commits on this branch carry both a matching DCO Signed-off-by trailer and a verifiable SSH commit signature, and the commit messages are unusually specific — each names the exact mutant or false-positive/negative shape it closes, rather than a generic "fix"/"wip".
  • docs/testing.md, the xtask MUTATIONS doc comment, and the code agree on exactly what UnsetKeyOnExit::arm( must look like and why it must be a named call rather than a bare unit-struct construction.
  • The corrected doc comment ("its callers (in fix, lib, examine and hardware_root)") matches the real call sites exactly — I grepped all four and found no others.

Deployment notes

None

What this covered

Full review of the whole change, not narrowed (prw-base…HEAD, merge-base 1e5aac8…b92a03f6) — the PR has moved past every prior automated round, so nothing was assumed to still hold from them. All three changed files (crates/rocm-core/src/test_env.rs, xtask/src/env_mutation_contract.rs, docs/testing.md) were read completely, by two independent workers plus my own direct reads, along with the real call sites of RestoredEnvVar::set( (fix.rs, lib.rs, examine.rs, hardware_root.rs), the root AGENTS.md, Cargo.toml's unsafe_code lint configuration, and the full commit history and raw commit objects (signatures, sign-off trailers) on this branch.

Did not run: no Rust toolchain is available in this environment (cargo/rustc not found, no network path to install one), so no test or mutation was actually executed here; test-effectiveness verdicts above are from static reasoning over the code, not a run, and are capped accordingly. Read CI instead: every check that reported at this head (b92a03f) passed, including Test (affected crates), build-and-test, clippy, Commit signatures + sign-off, and Coverage (rocm-dash crates, ratcheted); Skill checks (skillscope) and Sphinx docs build (-W) were skipped, not failed. Prior review threads on this PR were read only to establish that no round had yet reviewed this exact head — not used to produce any finding above. Design review (Step 7) was folded into direct reasoning rather than run as separate dedicated workers, given the change's small size (200 lines across 3 files).

Not reconciled against any discussion on this PR.

@siloteemu siloteemu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · pr-review-watcher · b92a03f

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — no blocking findings

Full review of the whole change (prw-base...HEAD, 11 commits, 3 files). All of it is test, scanner and docs code, with no production change.
Titled "test(core): prove the env-restore guard actually restores (ROCMAI-536)". The description matches the diff, and its claim that no scenario is needed under AGENTS.md §3 holds because nothing user-observable changes.

Blocking

None

Standing objections from the review at 13292e5, none re-found blind this round. Each was settled by putting it to the reviewer after the review ended, so these are anchored checks, not blind findings:

  • standing (was blocking) — the test-plan counts. No longer holds: the description's xtask (248) and rocm-core (606 total) counts match this head. The e2e-cucumber counts were not re-run.
  • standing (non-blocking) — the "two callers" doc comment. No longer holds: the comment now names the four modules.
  • standing (non-blocking) — the arm doc saying the guard checks which lock is held. No longer holds: it now says the guard does not check which one.
  • standing (non-blocking) — intermediate commits that add the bare-type matcher and a later one that deletes it. The commits are unchanged. This is moot if the merge squashes; the repository also allows rebase merges.

Non-blocking

  • new — Two doc comments next to the new code are stale or hard-code a list — xtask/src/env_mutation_contract.rs:106-113 ("The two tests that use it"), xtask/src/env_mutation_contract.rs:780-781 ("a third call added to the const fails here"), crates/rocm-core/src/test_env.rs (the caller list "in fix, lib, examine and hardware_root" on dropping_the_guard_restores_the_previous_state).
    • RestoredEnvVar::set( has six call sites (hardware_root.rs:160,173,182, lib.rs:7255, fix.rs:2182, examine.rs:5504), not two.
    • The const now has four entries and the length-assertion test has a third held array, so "a third call" is wrong. The "third call" line was already off by one on prw-base (:760 there); the PR adds the fourth entry and a third array to that same test.
    • The caller list is accurate today (grep-checked) but goes stale when a caller is added elsewhere. Its line is 129 characters against 76 for its neighbours.
    • Governing side in each case: the code. Fix: drop the counts and the list ("a new call added to the const", "every caller relies on").
      Confidence 75 · mechanical · Fix: reword the three comments.
  • new — The docs/testing.md Drop-guard sentence sits in the wrong bullet — docs/testing.md:107-114. That bullet is about a #[test] delegating its mutation to a helper. The new sentence is about a guard that is held, which is a different shape. It is accurate but loosely placed.
    Confidence 70 · mechanical · Fix: a separate bullet.

Decisions for the author

  • Is UnsetKeyOnExit worth its cost? — non-blocking-improvement
    • What it buys: cleanup of a private probe key after a failed assertion. Nothing in the tree reads that key. The only vars_os() hits are a #[cfg(windows)] environment-block copy at lib.rs:1735 and a separate test binary at tests/e2e-cucumber/tests/e2e/tui_driver.rs:222. Both tests also set the key's starting state themselves (remove_var at test_env.rs:126, set_var at :135 and :176). Without the guard, a passing run would also leave the key set to "before", so the guard is general cleanup of a key nothing reads, not something specific to failures.
    • What it costs: a global MUTATIONS entry keyed on a private test-only name, an extra case in every_mutating_call_is_flagged_inside_an_unguarded_test, the new holding_a_mutating_guard_still_needs_the_lock test with its helper, and a docs paragraph that cites that private helper as its example.
    • Simpler alternatives, neither tried: end each test with an explicit remove_var(KEY) under the lock, which the scanner already sees and which needs no xtask change; or define the guard inside each test body. The second I did not verify against the brace counter.
    • Both sides are defensible: the guard does make a failed test leave its probe key unset, which is the stated aim. The simpler form does not.
  • The scanner entry is name-keyed, so it goes stale silently on rename — tradeoff
    • Renaming UnsetKeyOnExit or arm fails nothing. The fixtures use the string "UnsetKeyOnExit::arm", and nothing asserts that each MUTATIONS entry still appears in the tree.
    • The same is true of the existing RestoredEnvVar::set( entry, so the PR repeats an existing pattern instead of introducing one. It is documented in docs/testing.md:111-114.
    • Making MUTATIONS entries tree-checked would close it for both entries, but that is wider than this PR.
  • UnsetKeyOnExit can still be built bare — tradeoff
    • let _g = UnsetKeyOnExit; compiles inside mod tests. It skips the &MutexGuard argument and the arm( call text the scanner matches. The type's docs and docs/testing.md both say so, and the doc on arm claims only that arming through arm without a lock does not compile.
    • A private field inside a nested module would close it, at the price of extra ceremony for a test helper.
  • Tests, with both verdicts stated. Unit tests for the production code (RestoredEnvVar), run as single-branch mutations of test_env.rs:
    • dropping_the_guard_restores_the_previous_state and unwinding_past_the_guard_restores_the_previous_state each fail with Drop skipping the restore while thread::panicking() (the unwind test only), the restore-unset branch made a no-op (the drop test only), unset restored as empty (the drop test only), capture taken after set, set that plants nothing, and the restore-set branch made a no-op. Neither test passes with RestoredEnvVar broken or reverted.
    • What stays green: the unwind test does not exercise the None arm of Drop under unwind, and neither test uses a non-Unicode value. Both are low value, so neither is a finding.
    • Scanner tests: removing the UnsetKeyOnExit::arm( entry from MUTATIONS fails every_mutating_call_is_flagged_inside_an_unguarded_test and holding_a_mutating_guard_still_needs_the_lock. With the entry removed and the held case also removed from the first test, holding_a_mutating_guard_still_needs_the_lock alone still fails.
    • In the real tree, moving arm( above the lock in the unwind test fails a_test_mutating_the_environment_serializes_itself, naming crates/rocm-core/src/test_env.rs:170: UnsetKeyOnExit::arm.
    • A hits[0].call assertion and an arm( that lost its paren stay green by equivalence (the scanner trims the paren). Not a defect.

Positive signals

  • The unwind test carries the observed value in the resume_unwind payload. That stops catch_unwind accepting an unrelated panic, and it is what lets the test fail on a set that plants nothing.
  • The scanner tests build each fixture from the real MUTATIONS entry shape through a helper, and the length assertion forces a fixture whenever the const grows.
  • The branch dropped about 380 lines of cross-line bare-type scanning from earlier commits in favour of the arm( call. No leftovers remain in the final tree (checked in docs/testing.md, env_mutation_contract.rs and test_env.rs).

Deployment notes

None

What this covered

  • Read: every changed file completely (crates/rocm-core/src/test_env.rs, xtask/src/env_mutation_contract.rs, docs/testing.md env-mutation section), the callers of RestoredEnvVar and the mod test_env declaration, against prw-base...HEAD (merge base 1e5aac8f, head b92a03f64a5cba94bf66585b59f4187aa41ad816). Reading was split across independent workers: one code pass, one agent-instruction plus code-comment pass, one history plus prior-changes pass and one design pass. I ran the mutations myself.
  • Ran:
    • cargo test -p xtask env_mutation: 46 passed.
    • cargo test -p rocm-core --lib test_env: 2 passed.
    • The mutations listed above.
  • Not run: cargo clippy --workspace, the workspace test suite, and the Windows target. I relied on the CI rollup you gave me, where every completed check passed except Skill checks (skillscope) and Sphinx docs build (-W), which were skipped. A skipped check is not a pass, and docs/testing.md is in the diff, so the docs build has not been shown to be clean on this head.
  • Sign-off: check_signoff.py over prw-base..HEAD exited 0 with 11 commits checked and none missing. The project gates on DCO sign-off per AGENTS.md §2.
  • Leak scan: one worker ran the AGENTS.md §2 pattern over the diff and found nothing. The title carries a bare ticket ID, which AGENTS.md §2 permits (the older EAI-8397 in the scanner's module docs was already on prw-base). I did not rescan the diff myself or scan the PR body.
  • Prior changes: one worker read the review on the earlier PR that introduced these files (#337). Nothing in it applied here. It did not read this PR's comments or reviews.
  • Did not run: nothing else was skipped.
  • Not reconciled against any discussion here. The fixes were written by the author, not by this review.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review (Claude Code) · 2946fd4

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (1e5aac8f...2946fd41, 12 commits, 3 files: crates/rocm-core/src/test_env.rs, xtask/src/env_mutation_contract.rs, docs/testing.md).
Ticket ROCMAI-536 named in the title and retrieved: "Prove the test env-restore guard actually restores the environment" (status: In Progress). This change implements exactly that — the two RestoredEnvVar tests, the UnsetKeyOnExit cleanup guard and its matching xtask scanner entry. The PR does not claim to close the ticket, and the description's scope and test-plan counts match the diff.

Only one new commit (2946fd41) landed since the most recent prior round reviewed this PR at b92a03f6: it adds an assertion in dropping_the_guard_restores_the_previous_state that drops UnsetKeyOnExit explicitly while the probe key is still set and asserts the key is gone, and it ties UnsetKeyOnExit's lifetime to the borrowed lock guard (UnsetKeyOnExit<'lock>(PhantomData<&'lock ()>), arm(_lock: &'lock MutexGuard<'_, ()>)).

Blocking

  • Three hardware-runner E2E checks fail at this head, and two more have not reported — CI (get_check_runs at 2946fd41)
    E2E tests (Strix Halo, Ubuntu, release, current) and E2E tests (Strix Halo, Ubuntu, nightly, current) both fail with error: Failed to extract archive: triton-...-linux_x86_64.whl / No space left on device (os error 28) during SDK install on the physical runner — a disk-capacity failure on the install path, not anything this diff touches. E2E tests (Strix Halo, Windows, current) also fails; its log shows only post-job orphan-process cleanup, consistent with a runner-side timeout. E2E tests (MI300X, current) and E2E tests (W7900, current) are still queued and had not reported at review time. Every check that actually exercises this diff's content — Test (affected crates), build-and-test, windows-build-and-test, clippy, Commit signatures + sign-off, Coverage (rocm-dash crates, ratcheted) — passed; Sphinx docs build (-W) and Skill checks (skillscope) were skipped (not failed).
    This pattern (a different hardware E2E lane failing or queued at each round) recurs across this PR's whole review history and is consistent with shared-runner infrastructure flakiness (disk space, timeouts) rather than a regression from a test/tooling/docs-only change. It is reported here because a failing or non-reporting required check is taken at face value regardless of apparent relevance.
    Confidence 100 (fetched the job logs directly) · mechanical · Fix: confirm these three failures are infra (disk space / runner), not code-caused, and either re-run them or get the two queued hardware lanes to report before merging.

Non-blocking

None

Decisions for the author

  • Is the UnsetKeyOnExit cleanup guard worth its scanner-side cost? — non-blocking-improvement
    The guard exists to unset a private probe key (ROCM_CORE_TEST_ENV_RESTORE_PROBE) on any exit path; nothing else in the tree reads that key. It now requires a lifetime-bound wrapper type, an arm() constructor, a 4th MUTATIONS string entry in a whole-tree scanner, and a dedicated xtask test (holding_a_mutating_guard_still_needs_the_lock). An explicit remove_var(KEY) call under the existing lock at the end of each test would achieve the same cleanup and is already visible to the scanner through its existing remove_var( rule, with no xtask change needed. The chosen route generalizes to future guard types that mutate from Drop; the simpler route does not. Confirm the added machinery is deliberate.
  • The new MUTATIONS entry is a string literal with no check that it still names a real symbol — tradeoff
    "UnsetKeyOnExit::arm(" is matched as text; nothing in the xtask tests asserts that this literal still corresponds to an existing type/method in the tree. Renaming either half silently drops the enforcement with every test green. "RestoredEnvVar::set(" has the same property already, so this repeats an existing, documented pattern rather than introducing a new one. Not asking for a fix here, just flagging that the gap now has two instances instead of one.

Positive signals

  • The new assertion in dropping_the_guard_restores_the_previous_state (explicit drop(unset_on_exit) while the probe key is still set, then assert_eq!(std::env::var_os(KEY), None, ...)) genuinely discriminates a no-op UnsetKeyOnExit::drop: traced through the test, KEY is "before" immediately prior to that drop, so a no-op cleanup would leave it Some("before") against the expected None. This closes the blocking gap raised in the prior round (that type's cleanup-on-exit behavior was previously asserted nowhere).
  • The new 'lock lifetime on UnsetKeyOnExit<'lock>, tied via PhantomData<&'lock ()> to the &'lock MutexGuard<'_, ()> taken by arm, converts what was previously only a documented convention ("declare it after the lock") into a real borrow-checker constraint: a UnsetKeyOnExit value cannot outlive the borrowed lock guard it was armed with, so ending the lock's scope before the guard drops will not compile. This closes the non-blocking concern raised in the prior round about that doc comment overstating what the type enforced.
  • Doc comments that previously hard-coded counts or caller lists ("the two tests that use it", "a third call added to the const", the four-module caller list on RestoredEnvVar) were replaced with wording that does not go stale as the code grows, rather than being patched with new numbers that would drift again.

Deployment notes

None

What this covered

Read both changed Rust files completely at the head commit, the full accumulated diff against the merge base, and the single commit (2946fd41) that landed since the prior round, plus all six real call sites of RestoredEnvVar::set( across examine.rs, fix.rs, hardware_root.rs, and lib.rs (grepped directly). Reasoned through concrete mutations of the production code (no-op Drop for both RestoredEnvVar and UnsetKeyOnExit, late/empty capture, lifetime-bound borrow semantics) against the test assertions rather than asserting correctness. Read CI (get_check_runs) and the actual job logs for the three failing checks rather than assuming their cause. Retrieved ROCMAI-536 directly by key.

Two independent workers were used: one for the coupled code-correctness pass (completed as a single pass by the driver after a subagent-concurrency limit blocked a second spawn attempt — not independent for that portion, noted here rather than absorbed silently), one for agent-instruction adherence, history, and code-comment consistency (independent, found nothing at or above the confidence bar).

No Rust toolchain or network path to install one was available in this environment, so no test was actually executed; all test-effectiveness and lifetime-enforcement verdicts above are from static reasoning over the code, capped at confidence 85 rather than 100 where a run would otherwise have settled them outright.

Prior review threads on this PR were read only to establish that no round had yet reviewed this exact head and to identify what changed since the last round — not used to produce any finding above; this review was reached blind on the code itself before anything in those threads was consulted. Not reconciled against any discussion on this PR.


🤖 Generated with Claude Code

@r0x0r r0x0r left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving this, with two notes below that are worth a look but shouldn't block the idea.

This is test, tooling and docs only - every line added here lands inside #[cfg(test)], nothing in a production path.

I ran the five mutants and each one behaves exactly as the description claims, including the ones that actually discriminate: a Drop that skips the restore while std::thread::panicking() fails only the unwind test, and a no-op UnsetKeyOnExit::drop fails only the explicit-drop test. Dropping the new entry out of MUTATIONS fails both new xtask tests, and taking the lock away makes the whole-tree scan name UnsetKeyOnExit::arm. So the tests are pinned to the behaviour they claim to pin, rather than merely passing.

The lifetime-borrowed arm(&MutexGuard) is the nicest part of this - "armed without holding the lock" stops being a convention and becomes a compile error.

One design point I'd rather have you confirm than change: a file-private, test-only type's constructor now sits in the global MUTATIONS list. It's symmetric with RestoredEnvVar::set( and it's pinned by a test, so it isn't wrong - but it does set the precedent that every test-local drop guard ends up registered globally. Worth a conscious yes rather than drifting into it.

The two inline comments are about CI state and commit history rather than about the code itself.

Comment thread xtask/src/env_mutation_contract.rs Resolved
Comment thread xtask/src/env_mutation_contract.rs Resolved
@rominf

rominf commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator Author

Answering both automated reviews from @nowycondro.

Review of b92a03f6 (review): both findings were addressed in 2946fd41.

  • UnsetKeyOnExit's cleanup was untested (blocking): dropping_the_guard_restores_the_previous_state now drops the guard by hand while the probe is still set to "before" and asserts the key is gone. A no-op UnsetKeyOnExit::drop fails that assertion, and only that one.
  • arm's doc claimed more than the type enforced: rather than softening the doc, the type now carries the borrow: UnsetKeyOnExit<'lock>(PhantomData<&'lock ()>) with arm(_lock: &'lock MutexGuard<'_, ()>). Releasing the lock before the guard drops no longer compiles, so the doc's claim is now true.

Review of 2946fd41 (review):

  • Hardware E2E lanes red or not reporting (blocking): agreed that they are infrastructure, and I re-ran the failed jobs of the self-hosted run (run 41609102984, now attempt 2). From the attempt-1 logs: both Strix Halo Ubuntu lanes ran out of disk (No space left on device) while extracting a wheel during the SDK install; Strix Halo Windows had one unexpected failure, runtime-09 (runtime-resolve-canonical-release), because the runner installs from a package mirror, so the SDK preview names the mirror instead of the canonical release index as its source (its other three failures are expected-failure rows); W7900 was cancelled without getting a runner; the consolidated report failed because those lane reports were missing. MI300X and Strix Halo WSL2 passed. This change adds no lines outside #[cfg(test)] code, the xtask scanner and docs/testing.md, so none of those lanes runs anything it touches. I'll follow the re-run to completion.
  • Is UnsetKeyOnExit worth its scanner cost? Yes, deliberately. An explicit remove_var(KEY) at the end of each test runs only when every assertion before it passed; a failing restore assertion, which is exactly when someone would look at the probe, would skip it and leave the key set for the rest of the test binary. The guard runs on every exit, and since 2946fd41 it also cannot outlive the lock it was armed with. Nothing reads the probe key today, so a leak would be harmless now; the point is that a test about not leaking environment state should not leak its own. The scanner side is one MUTATIONS entry plus one test pinning it, under the rule docs/testing.md already states for any guard whose Drop mutates, so the next such guard follows the same route rather than needing new scanner code.
  • The MUTATIONS entry is a string with no check that the symbol still exists: correct, and as you note the existing RestoredEnvVar::set( entry has the same property. Closing it means checking every MUTATIONS entry against the tree, which belongs in its own change covering both entries rather than in this one.

I've merged current main into a scratch copy of this branch: it merges cleanly, and cargo test -p xtask env_mutation (46 passed) and cargo test -p rocm-core --lib test_env (2 passed) pass on the merged tree.

@rominf
rominf requested a review from volen-silo November 9, 2026 13:36
`RestoredEnvVar` exists to put an environment variable back when a test
unwinds, but nothing tested that it does: a no-op `Drop`, a capture that
treats an unset variable as empty, and a capture taken after the mutation
all left the rocm-core suite green.

Two tests now cover it. One drops the guard over a key that was unset and
over one that was set, and checks each comes back exactly as it was. The
other panics while holding the guard, inside `catch_unwind`, which is the
only path that catches a `Drop` skipping the restore while the thread is
panicking; the panic payload carries the value seen before the panic, so
an unrelated panic cannot pass it.

Both tests clean up their probe key on every exit, a failed assertion
included, through `UnsetKeyOnExit`. It is armed with `arm(&lock)` and
borrows the held lock guard, so arming it without the lock, or releasing
the lock before it drops, does not compile. The first test drops it by
hand and asserts the probe is gone.

Its `Drop` changes the environment where the env-mutation scan cannot
see it, so `UnsetKeyOnExit::arm(` joins `RestoredEnvVar::set(` in the
scan's `MUTATIONS` list, pinned by a new xtask test. docs/testing.md
records that rule for any guard whose `Drop` mutates, and that a
`*_TEST_LOCK` counts only when it is acquired with `.lock(`.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the test/env-guard-restore-coverage branch from 2946fd4 to 1e34b71 Compare November 10, 2026 12:22
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor
Security scan found no critical or high findings on lines changed by this pull request

View the security scan run for complete reports.

“New” means the finding is located in a changed file on an added or modified line; it is not a base-versus-head findings delta.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 1e34b715

Review — no blocking findings

Full review of the whole change (3 files, base 2d3e0cd5…head 1e34b715).
Title names ROCMAI-536; the ticket was not retrieved (no tracker access from this review context), so this is judged only against the PR's own description, which matches the diff — no overclaim or undisclosed scope found.

Blocking

None

Non-blocking

None

Decisions for the author

  • UnsetKeyOnExit can be bypassed within the same test module — non-blocking-improvement
    Because Rust privacy is module-scoped, another test added to the same tests module could construct UnsetKeyOnExit(PhantomData) directly instead of through arm(&lock), skipping both the lock-borrow and the MUTATIONS-matched text. This is already honestly disclosed in the type's doc comment and in docs/testing.md ("and nothing enforces this"), so it isn't a hidden defect — but a future hardening (e.g. moving the guard into its own private submodule) would close the gap rather than merely documenting it.
  • The env-mutation contract is a manually curated, text-matching scanner — tradeoff
    This PR extends the existing MUTATIONS list with one more entry (UnsetKeyOnExit::arm(), faithfully following the precedent set for RestoredEnvVar::set(. That's the right move within the existing design, but the design itself (a list that must be hand-updated every time a new restoring/Drop-based wrapper appears) is a known scaling limit the author/maintainers may want to keep an eye on as more such guards accumulate. Nothing to change in this PR.

Positive signals

  • Unusually rigorous self-documentation: every non-obvious scanner behavior and test-fixture choice is pinned by a named test or doc comment rather than left implicit.
  • The two new tests are a well-chosen pair — one exercises the plain-drop restore path, the other specifically exercises the unwind/panic path via catch_unwind/resume_unwind with a payload that distinguishes "guard restored" from "nothing was ever planted" — together they reproduce all three original mutant classes described in the PR (no-op Drop, unset-vs-empty conflation, late capture).
  • The known UnsetKeyOnExit bypass gap (above) is disclosed up front in the code and docs rather than glossed over.

Deployment notes

None — test-only, xtask-only, and docs-only change; no runtime or user-facing behavior changes.

What this covered

Full diff and full-file reads of all three changed files at head 1e34b71504f8a2470510da8b9af14c10f3dfd467 against base 2d3e0cd59f329d4d0d7a2623309c8a0b778aa7ad: crates/rocm-core/src/test_env.rs (including the pre-existing, unchanged RestoredEnvVar the new tests exercise), xtask/src/env_mutation_contract.rs (including the full scanner pipeline — scan, serializer_column, lock_name_column, statement_continuation, strip_literals_and_comments), and the relevant docs/testing.md section. Run via two independent fan-out passes (code-correctness; AGENTS.md-adherence + file history), plus direct spot-checks of my own (uniqueness of the new KEY/UnsetKeyOnExit identifiers across the tree, full text of the repo's root AGENTS.md — the only agent-instruction file in the tree — and the existing scanner test patterns the new fixtures mirror).

CI: all required checks green at this head; one self-hosted hardware E2E lane (E2E tests (W7900, current)) was still queued at review time rather than failed — not blocking, but not yet resolved.

Did not run as a separate pass: a dedicated "prior review comments on earlier changes to these files" check — folded into the history pass, since only one prior commit (the originating RestoredEnvVar/MUTATIONS guard) ever touched these files besides this PR. Step 7 design questions were answered directly by the consolidating reviewer rather than through further fan-out, given the narrow, single-purpose scope of the change. Not reconciled against this PR's lengthy prior review-comment history — that was deliberately not read, per the review method's independence rule.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · head 1e34b715

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (3 files: crates/rocm-core/src/test_env.rs, docs/testing.md, xtask/src/env_mutation_contract.rs).
Implements the task described in ROCMAI-536 ("Prove the test env-restore guard actually restores the environment", status: In Progress). The PR does not claim to close the ticket, and the description's scope matches what the diff actually does.

Blocking

  • CI: the self-hosted E2E consolidated report is currently red, caused by an unrelated hardware lane, not by this diff — E2E consolidated report (self-hosted) (run 42815174298, job 130543730901), E2E tests (W7900, current)
    Verified by reading the job log directly: the report-merge step fails because the e2e-gpu-w7900 lane's needs result is cancelled — ##[error]lane e2e-gpu-w7900 ran (needs result: cancelled) but uploaded no e2e-gpu-w7900-report, so it is missing from the consolidated report. This change touches only #[cfg(test)] code in rocm-core, an xtask static-analysis test, and docs — nothing that a GPU hardware E2E lane exercises. All other checks (changes, clippy, Windows build/test, Test (affected crates), CodeQL, security scans, every other E2E lane) are green on this head. For context, GitHub's own mergeStateStatus for the PR is UNSTABLE (not BLOCKED) with reviewDecision: APPROVED, which is consistent with these two checks not being configured as required merge gates — but the check is still red at the time of this review, so it's reported per the review's rule that a failing check is a blocking item until resolved (by rerun or explicit waiver), rather than assumed away.
    Confidence 100 · mechanical · Fix: rerun the e2e-gpu-w7900 job (or the whole self-hosted E2E workflow) once a W7900 runner is available, so the merge isn't made against a stale/unrelated red.

Non-blocking

None

Decisions for the author

  • MUTATIONS entries in xtask/src/env_mutation_contract.rs are plain string literals with no check that they still name a real symbol — non-blocking-improvement
    The new "UnsetKeyOnExit::arm(" entry (alongside the pre-existing "RestoredEnvVar::set(") is matched purely as text; nothing in the test suite fails if the named call is later renamed, leaving a stale, silently-inert entry that stops providing the coverage its comment claims. A test that asserts every MUTATIONS entry corresponds to a symbol actually present in the tree would close this for both entries at once. Not a defect in this change — it reproduces an existing, intentional pattern rather than introducing a new one — just worth confirming is an accepted tradeoff rather than an oversight.

Positive signals

  • The two new test_env.rs tests are constructed as real proofs rather than string-pinning: dropping_the_guard_restores_the_previous_state separately exercises the "was unset" and "was set" prior states (so a no-op Drop and an unset-vs-empty conflation are each caught by a distinct assertion), and unwinding_past_the_guard_restores_the_previous_state carries the observed value out through the panic payload specifically so a vacuous panic (e.g. from set itself) can't make the test pass for the wrong reason.
  • UnsetKeyOnExit::arm's lifetime signature (arm(_lock: &'lock MutexGuard<'_, ()>) -> Self<'lock>) makes the "armed only while a lock is held, and must drop before the lock does" invariant a compile-time property instead of a documentation-only convention.

Deployment notes

None

What this covered

Three independent fan-out passes read the full diff and the complete current text of all three changed files (plus the surrounding, unchanged parts of xtask/src/env_mutation_contract.rs needed to judge the scanner logic in place): a code-correctness pass (traced the scanner's matching logic and each new test's assertions against the three historical mutants the PR claims to catch), an agent-instruction-adherence pass (against this repo's AGENTS.md), and a history/prior-changes pass (git log/git blame on both files' pre-PR history). All three returned clean — no findings survived verification. I additionally checked independently: no new source files were added (so docs/architecture.md/check-architecture-doc does not newly apply here, and this head's clippy CI job, which runs that check, is green), the test's probe environment-variable key is read nowhere else in the tree, and the PR description's scope matches the full diff including the xtask scanner change.
Base 1e5aac8ffde6e7f91c54f87f958410b1058ccefe…head 1e34b71504f8a2470510da8b9af14c10f3dfd467.

Did not run: a dedicated design-review fan-out beyond the one decision noted above — the change is small and tightly coupled enough that this was judged directly rather than split across further workers. A prior human review thread on this PR was mentioned as possibly discussing a check-architecture-doc/property-test module-naming issue; no such thread was found in this PR's current (fresh) review-comment or issue-comment history, and this head's clippy check (which runs that gate) is green, so nothing currently blocks on it.
Not reconciled against this PR's existing review discussion — that was deliberately not read for finding-derivation purposes, per this review's blind-review method; CI status and git history were read directly from GitHub/git, not from the discussion.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Full review of the whole change. Implements test coverage for RestoredEnvVar's restore behavior (previously untested since #337) plus a matching scanner/docs update for a second Drop-mutating test guard; ticket ROCMAI-536 is named in the PR title, informational only — not retrievable from this context.

Blocking

  • One CI check is red on the head commit — E2E consolidated report (self-hosted) (run 42815174298). Verified by reading the job log directly: the e2e-gpu-w7900 lane was cancelled, and the consolidation step then errors because a cancelled lane uploaded no report ("lane e2e-gpu-w7900 ran (needs result: cancelled) but uploaded no e2e-gpu-w7900-report, so it is missing from the consolidated report"). Every other e2e lane (e2e-gpu, Strix Ubuntu/Windows/WSL, MI300X) succeeded; the combined legacy status is success; clippy, windows-build-and-test, both Test (affected crates) jobs, build-and-test, and Commit signatures + sign-off are all green. This reads as a self-hosted-runner availability issue (the W7900 lane getting cancelled, not failing its tests) rather than something this PR's content — a rocm-core test-only change — caused, but I did not re-run it myself and can't confirm it clears on retry.
    Confidence 85 · architectural (CI gate) · Fix: rerun the self-hosted E2E workflow for this commit; if the W7900 lane cancellation recurs independent of this branch, it's infrastructure, not this PR.

Non-blocking

  • A sibling Drop-mutating guard is still outside the contract scanner's enforcement — xtask/src/env_mutation_contract.rs:123-128 (MUTATIONS), crates/rocm-core/src/examine.rs:3074,3092,4572 (EnvVarGuard). Full detail in the inline comment on MUTATIONS. Pre-existing, not introduced by this PR — flagged because this PR is the first thing to name and fix this exact class ("a guard whose Drop mutates the environment, invisible to the text scan") for one instance (UnsetKeyOnExit), and EnvVarGuard has the identical shape and isn't listed, so a second unguarded call site to it would pass CI silently.
    Confidence 85 · architectural · Fix: add EnvVarGuard::set( to MUTATIONS with a fixture (mirroring held_mutating_guard), or document why it's deliberately exempt the way NAMED_SERIALIZERS already does for ScopedEnvVar.

Decisions for the author

None.

Positive signals

  • UnsetKeyOnExit::arm(_lock: &'lock MutexGuard<'_, ()>) ties the guard's lifetime to a borrow of the caller's lock guard, so "armed without holding a lock" and "lock released before the guard drops" are both compile errors rather than runtime hopes (crates/rocm-core/src/test_env.rs).
  • unwinding_past_the_guard_restores_the_previous_state's panic payload carries the environment value observed just before the panic, so catch_unwind can't be satisfied by an unrelated panic (e.g. from a broken set that plants nothing) — a deliberate, well-targeted test design, not just a happy-path check.
  • The docs/testing.md edit drops the hardcoded "Three known blind spots" count in the same change that adds a fourth bullet, instead of leaving a number that goes stale immediately (docs/testing.md).

Deployment notes

None — test-, tooling- and docs-only change; no production source file, CLI-visible behavior, or docs/architecture.md-tracked file is touched.

What this covered

Read the full diff (3 files, +225/-10) between merge-base 1e5aac8f and PR head 1e34b715, in a clean isolated clone checked out to the PR branch. Read the pre-existing RestoredEnvVar/Drop implementation and the env_mutation_contract.rs scanner's matching logic (serializer_column, NAMED_SERIALIZERS, MUTATIONS) to hand-trace that the new tests fail against each of the three mutants the PR describes (no-op Drop, unset/empty conflation, post-mutation capture), plus a no-op set. Searched the whole repository for every impl Drop that mutates an environment variable to check this PR's new blind-spot class against siblings. Checked all check-runs and the combined status for the head commit and read the one failing job's log.

Did not run: cargo test/cargo clippy — no Rust toolchain available in this environment, so the new-tests-fail-against-mutants claim is established by manual tracing against the real implementation, not by execution. Prior-changes pass — no earlier review rounds visible to compare against. Did not read the PR's existing review comments/discussion, per review-blindness practice. Single-pass review; the change is three small, tightly-coupled files, so no further fan-out was used.

Not reconciled against any discussion on this PR.


🤖 Automated by an AI agent

"set_var(",
"remove_var(",
"RestoredEnvVar::set(",
"UnsetKeyOnExit::arm(",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, pre-existing (not introduced by this PR), flagged for pattern relevance.

This PR documents and fixes exactly one instance of "a guard whose Drop mutates the environment, invisible to this text scan" (UnsetKeyOnExit). At least one sibling guard has the identical shape and is still unlisted here: EnvVarGuard in crates/rocm-core/src/examine.rs (struct + Drop impl around lines 3074/3092, one call site EnvVarGuard::set( around line 4572). Its one current call site does take PROCESS_ENV_TEST_LOCK immediately beforehand, so there's no live bug today, but since EnvVarGuard::set( isn't in MUTATIONS, a second unguarded call site would pass this scan silently — the exact regression class this guard exists to catch.

(ScopedEnvVar in apps/rocm/src/main.rs, the other Drop-mutating guard in the tree, is a different case: the NAMED_SERIALIZERS doc comment just above this one already explains why it was deliberately removed from auto-enforcement, so that one isn't an oversight.)

Now that the general rule is written down in docs/testing.md, worth either adding EnvVarGuard::set( to MUTATIONS (with a fixture, same shape as held_mutating_guard/holding_a_mutating_guard_still_needs_the_lock below) or recording the same kind of "deliberately exempt because…" reasoning NAMED_SERIALIZERS has for ScopedEnvVar.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed, and fixed in 4b2d114. EnvVarGuard now lives in crates/rocm-core/src/examine/framework.rs after the merge from main. Its constructor sets the variable and its Drop restores it, both outside the test body, so the scan saw nothing in the test that holds it.

EnvVarGuard::set( is now in MUTATIONS. every_mutating_call_is_flagged_inside_an_unguarded_test covers it as a held guard alongside UnsetKeyOnExit::arm, and the docs/testing.md blind-spot entry names it too. I checked that the guard bites on the real tree. With the PROCESS_ENV_TEST_LOCK acquisition removed from asking_to_skip_the_frameworks_is_recorded_as_skipped, a_test_mutating_the_environment_serializes_itself now fails at framework.rs:524: EnvVarGuard::set. With the lock back in place, it passes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
EnvVarGuard in the rocm-core examine framework tests sets a variable in
its constructor and restores it in Drop, both outside any test body, so
the scan saw nothing in a test that built one. Its one call site takes
PROCESS_ENV_TEST_LOCK today, but a second, unguarded one would have
passed the scan. List its constructor call the way UnsetKeyOnExit::arm
is listed, so dropping that lock now fails the build.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The wrapper and guard entries in MUTATIONS are matched as text, so
renaming one of those symbols left its entry matching nothing and the
scan green while the renamed call sat outside it. Check that every entry
is still called somewhere in the tree, on code with literals and
comments stripped so a doc-comment mention cannot keep it alive.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf added this pull request to the merge queue Nov 16, 2026
@rominf

rominf commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Answering the points in the review summaries from @nowycondro (2026-11-13) and @juhovainio (2026-11-16):

  • Red E2E consolidated report (self-hosted): both reviews traced it to the cancelled W7900 lane. ci: remove the W7900 lane from e2e-selfhosted #1066 has since removed that lane from e2e-selfhosted.yml, and I merged current main into this branch rather than rebasing it. On the new head (1cb6d9b) every check is green.
  • EnvVarGuard outside the scan: fixed in 4b2d114. Details are in the inline reply.
  • MUTATIONS entries are text that can go stale: fixed in 1cb6d9b, rather than accepted as a tradeoff. every_listed_mutation_is_still_called_in_the_tree fails when an entry no longer matches any call in the tree. It matches against code with literals and comments stripped, so a mention in a doc comment cannot keep a dead entry alive. I checked it by adding a fake entry for a renamed symbol: the test failed and named that entry.

Both commits are small xtask/docs additions made after r0x0r's approval of 1e34b71. The PR description lists them.

@rominf
rominf removed this pull request from the merge queue due to a manual request Nov 16, 2026

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$(cat /tmp/review_511_body.md)

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — no blocking findings

Full review of the whole change.
Adds test coverage, and a sibling-guard entry (EnvVarGuard::set(), to the env-mutation discipline introduced for RestoredEnvVar/UnsetKeyOnExit (ticket named in the PR title; not retrieved here, no tracker access in this context — the PR body stands in as the stated intent).

Blocking

None

Non-blocking

None

Decisions for the author

None

Positive signals

  • UnsetKeyOnExit::arm(&lock) borrows the caller's MutexGuard<'_, ()> for its own lifetime (PhantomData<&'lock ()>), so dropping the lock before the guard is a compile error rather than a convention someone has to remember.
  • unwinding_past_the_guard_restores_the_previous_state carries the observed value out through resume_unwind's payload instead of asserting inside the closure (which catch_unwind would swallow) or relying on a fixed string, so the test can't pass against a set that silently does nothing or panics on entry — both were checked.
  • The MUTATIONS entries for UnsetKeyOnExit::arm(/EnvVarGuard::set( are matched as call text rather than bare type names, sidestepping the fragile identifier/type-position heuristics an earlier round of this same PR had built and then removed; the new every_listed_mutation_is_still_called_in_the_tree test keeps the list itself honest against future renames.

Deployment notes

None

What this covered

Read crates/rocm-core/src/test_env.rs, xtask/src/env_mutation_contract.rs, and docs/testing.md in full, against 46b9e3b5...1cb6d9b6 (merge-base of origin/main and the PR head), plus the pre-existing surrounding context in each file (RestoredEnvVar, the scan/serializer_column/lock_name_column matching logic, and the one external call site this PR newly lists, EnvVarGuard::set( in crates/rocm-core/src/examine/framework.rs). Confirmed by reading the code that: the new MUTATIONS entries match real call sites and no others in the tree; holding_a_mutating_guard_still_needs_the_lock's guarded/unguarded fixtures exercise the scan's own line/column ordering rule correctly; and the EnvVarGuard::set( call site already takes PROCESS_ENV_TEST_LOCK before constructing the guard. No Rust toolchain was available in this environment to run the suite directly, so this is a traced-by-reading verification rather than an executed one.

CI: the primary CI workflow (build, clippy, Test (affected crates), windows-build-and-test, coverage) is green on this head. The separate E2E self-hosted workflow's MI300X/consolidated-report jobs failed on this head; the job log shows this is a GPU-hardware/torch-vLLM version-mismatch error during runtime install on the self-hosted runner, unrelated to anything this diff touches (test- and docs-only, no production code). Not treated as blocking.

Did not run: history pass and prior-changes pass as separate steps — the PR's own review threads (11, all already resolved by code changes on this head, most recently the EnvVarGuard::set( gap fixed in 4b2d114) were read afterward and are consistent with what this review found independently; agent-instruction adherence — no CLAUDE.md/AGENTS.md found in the touched directories, and the PR body cites AGENTS.md §3 for why no behavior scenario is needed, which matches a test/tooling/docs-only diff.
Not reconciled against the GitHub discussion UI beyond reading the review-thread text as part of the history/prior-changes check noted above.


🤖 Automated by an AI agent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants