Repository navigation
test(core): prove the env-restore guard actually restores (ROCMAI-536) #511
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
1e34b71
2439ed3
4b2d114
1cb6d9b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -64,3 +64,152 @@ impl Drop for RestoredEnvVar { | |
| } | ||
| } | ||
| } | ||
|
|
||
| #[cfg(test)] | ||
| mod tests { | ||
| use super::RestoredEnvVar; | ||
| use std::ffi::OsString; | ||
| use std::marker::PhantomData; | ||
| use std::path::Path; | ||
| use std::sync::MutexGuard; | ||
|
|
||
| /// Serializes the env mutation below, as the contract guard requires. | ||
| static TEST_ENV_TEST_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); | ||
|
|
||
| /// Used by nothing else in the tree, so no other test reads it mid-flight. | ||
| const KEY: &str = "ROCM_CORE_TEST_ENV_RESTORE_PROBE"; | ||
|
|
||
| /// Leaves [`KEY`] unset when a test exits, including by a failed assertion. | ||
| /// | ||
| /// It borrows the lock guard it was armed with, so it drops while that | ||
| /// lock is still held: releasing the lock first does not compile. A test | ||
| /// about not leaking environment state should not leak the probe when it | ||
| /// fails, which is exactly when someone would be looking at it. | ||
| /// | ||
| /// Holding one mutates the environment from a `Drop` impl the contract | ||
| /// guard cannot see, so the call that builds one, `UnsetKeyOnExit::arm(`, | ||
| /// is named in its `MUTATIONS` list: a test that arms one without taking a | ||
| /// `*_TEST_LOCK` is flagged like a direct call. Build it only through | ||
| /// `arm`: constructing it directly (`UnsetKeyOnExit(PhantomData)`) skips | ||
| /// both the lock borrow and the text the scan matches. | ||
| struct UnsetKeyOnExit<'lock>(PhantomData<&'lock ()>); | ||
|
|
||
| impl<'lock> UnsetKeyOnExit<'lock> { | ||
| /// Borrows the caller's lock guard for as long as the returned value | ||
| /// lives, so arming one without holding a lock, or releasing the lock | ||
| /// before this drops, does not compile. Any `Mutex<()>` guard | ||
| /// satisfies the type; that some `*_TEST_LOCK` is taken stays the | ||
| /// contract guard's to check (it does not check which one). | ||
| const fn arm(_lock: &'lock MutexGuard<'_, ()>) -> Self { | ||
| Self(PhantomData) | ||
| } | ||
| } | ||
|
|
||
| impl Drop for UnsetKeyOnExit<'_> { | ||
| #[allow(unsafe_code)] // std::env::remove_var is unsafe in edition 2024 | ||
| fn drop(&mut self) { | ||
| // SAFETY: dropped inside the caller's `TEST_ENV_TEST_LOCK` scope, | ||
| // and nothing else reads this key. | ||
| unsafe { std::env::remove_var(KEY) }; | ||
| } | ||
| } | ||
|
|
||
| /// The restore is this type's whole purpose, and it is what every caller | ||
| /// relies on to keep a panicking test from leaking a value into the next | ||
| /// one. | ||
| /// | ||
| /// Both prior states are needed, because they catch different mistakes: | ||
| /// | ||
| /// * *previously unset* must come back **unset**, not merely empty — which a | ||
| /// `var().unwrap_or_default()`-style capture would get wrong — and it also | ||
| /// catches capturing the previous value *after* the mutation, which would | ||
| /// "restore" the new value and leave the key set; | ||
| /// * *previously set* must come back to that exact value, which a `Drop` | ||
| /// that does nothing would leave as the planted one. | ||
| /// | ||
| /// It ends by dropping its [`UnsetKeyOnExit`] by hand, with the probe still | ||
| /// set, and asserting the probe is gone: the cleanup is a claim too. | ||
| #[test] | ||
| #[allow(unsafe_code)] // std::env::set_var/remove_var are unsafe in edition 2024 | ||
| fn dropping_the_guard_restores_the_previous_state() { | ||
| let lock = TEST_ENV_TEST_LOCK | ||
| .lock() | ||
| .unwrap_or_else(std::sync::PoisonError::into_inner); | ||
| let unset_on_exit = UnsetKeyOnExit::arm(&lock); | ||
|
|
||
| // SAFETY: serialized by the lock above, and nothing else reads this key. | ||
| unsafe { std::env::remove_var(KEY) }; | ||
| drop(RestoredEnvVar::set(KEY, Path::new("planted"))); | ||
| assert_eq!( | ||
| std::env::var_os(KEY), | ||
| None, | ||
| "a key that was unset must be unset again, not empty or still planted" | ||
| ); | ||
|
|
||
| // SAFETY: as above. | ||
| unsafe { std::env::set_var(KEY, "before") }; | ||
| let restore = RestoredEnvVar::set(KEY, Path::new("planted")); | ||
| assert_eq!( | ||
| std::env::var_os(KEY).as_deref(), | ||
| Some(std::ffi::OsStr::new("planted")), | ||
| "the guard must actually set the value it was given" | ||
| ); | ||
| drop(restore); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Both halves drive the restore with an explicit In practice
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added in |
||
| assert_eq!( | ||
| std::env::var_os(KEY).as_deref(), | ||
| Some(std::ffi::OsStr::new("before")), | ||
| "a key that was set must get its previous value back" | ||
| ); | ||
|
|
||
| drop(unset_on_exit); | ||
| assert_eq!( | ||
| std::env::var_os(KEY), | ||
| None, | ||
| "dropping the cleanup guard must leave the probe unset" | ||
| ); | ||
| } | ||
|
|
||
| /// The path the module docs promise: a test that panics while holding the | ||
| /// guard still gets its previous value back. An explicit `drop` runs the | ||
| /// same destructor, but only an unwind exercises it while the thread is | ||
| /// panicking: a `Drop` that skips the restore when | ||
| /// [`std::thread::panicking`] is true passes | ||
| /// `dropping_the_guard_restores_the_previous_state` and fails this test. | ||
| /// | ||
| /// The panic is raised with [`std::panic::resume_unwind`], which unwinds | ||
| /// without calling the panic hook, so a passing run prints no panic message | ||
| /// and no process-wide hook has to be swapped while other tests run. | ||
| /// | ||
| /// The payload carries the value observed just before the panic. Without | ||
| /// it, `catch_unwind` would accept any panic — including one raised by | ||
| /// `set` itself before it planted anything — and a `set` that planted | ||
| /// nothing would leave `"before"` in place for the final assertion to | ||
| /// pass on. An assertion inside the closure would not help: `catch_unwind` | ||
| /// swallows it and the result is still an `Err`. | ||
| #[test] | ||
| #[allow(unsafe_code)] // std::env::set_var is unsafe in edition 2024 | ||
| fn unwinding_past_the_guard_restores_the_previous_state() { | ||
| let lock = TEST_ENV_TEST_LOCK | ||
| .lock() | ||
| .unwrap_or_else(std::sync::PoisonError::into_inner); | ||
| let _unset_on_exit = UnsetKeyOnExit::arm(&lock); | ||
|
|
||
| // SAFETY: serialized by the lock above, and nothing else reads this key. | ||
| unsafe { std::env::set_var(KEY, "before") }; | ||
| let unwound = std::panic::catch_unwind(|| { | ||
| let _restore = RestoredEnvVar::set(KEY, Path::new("planted")); | ||
| std::panic::resume_unwind(Box::new(std::env::var_os(KEY))); | ||
| }); | ||
|
|
||
| assert_eq!( | ||
| unwound.unwrap_err().downcast_ref::<Option<OsString>>(), | ||
| Some(&Some(OsString::from("planted"))), | ||
| "the closure must have unwound, with the guard's value in place" | ||
| ); | ||
| assert_eq!( | ||
| std::env::var_os(KEY).as_deref(), | ||
| Some(std::ffi::OsStr::new("before")), | ||
| "unwinding past the guard must put the previous value back" | ||
| ); | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -131,7 +131,9 @@ The guard accepts `ScopedTestEnv` or **any** static named `*_TEST_LOCK` taken | |
| inside the test body. The suffix is the rule, so a new lock | ||
| is recognised the day it is declared rather than when someone remembers to add | ||
| it to a list. The exemption is per test function, not per file: one test taking | ||
| a lock does not cover its neighbours. | ||
| a lock does not cover its neighbours. "Taken" means acquired: a `*_TEST_LOCK` | ||
| counts only with a `.lock(` in the same statement, so `let _x = &SOME_TEST_LOCK;` | ||
| names a lock without holding it and does not satisfy the guard. | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not about this hunk. The line I mean is That bullet is the only place a contributor is told how to keep the guard honest, and it describes exactly one shape: "a wrapper hides the mutation until the wrapper itself is named in the scanner's
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done in 492cd82. The blind-spots bullet now says that a guard type whose |
||
|
|
||
| Two things the shape above gets right, both of which the guard checks only | ||
| partly: | ||
|
|
@@ -150,14 +152,19 @@ it reads what the panicking one left behind. `rocm-core`'s `RestoredEnvVar` | |
| unwinding runs it; prefer it where one exists. It restores but does not | ||
| serialize, so the lock is still yours to take. | ||
|
|
||
| Keep the mutation and the lock in the same test body. Three known blind spots, | ||
| all of which the scan reports nothing for: | ||
| Keep the mutation and the lock in the same test body. The scan reports nothing | ||
| for any of these known blind spots: | ||
|
|
||
| - A `#[test]` that delegates its mutation to a helper — the helper's body is a | ||
| different scope, so the scan cannot see the two together. It matches call | ||
| text, so a wrapper hides the mutation until the wrapper itself is named in the | ||
| scanner's `MUTATIONS` list, as `RestoredEnvVar::set(` is. Add a new restoring | ||
| helper to that list, or it is outside the guard. | ||
| - A guard held by the test whose `Drop` mutates the environment — the mutation | ||
| runs in the `Drop` impl, which the scan never enters. List the call that | ||
| builds the guard in `MUTATIONS`, as `UnsetKeyOnExit::arm(` and | ||
| `EnvVarGuard::set(` are, and build it only through that call: constructing | ||
| the type directly leaves no listed text to match, and nothing enforces this. | ||
| - A harness attribute that does not end in `test`, such as `#[test_case(..)]` or | ||
| `#[rstest]`. Neither is used in this tree. | ||
| - `#[cfg_attr(unix, test)]`, for the same reason: the attribute's path reads as | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -107,13 +107,33 @@ mod tests { | |
| /// an offense. Both directions have fixtures. | ||
| /// `RestoredEnvVar::set(` is here because the guard matches TEXT, and a | ||
| /// mutation spelled as a method on a restoring wrapper contains neither | ||
| /// direct call. The two tests that use it are the sanctioned exception this | ||
| /// change documents, and before this entry existed they sat outside the | ||
| /// guard entirely — deleting their lock acquisition left the scan green, | ||
| /// which is the opposite of what their comments told the next contributor. | ||
| /// direct call. The tests that use it are the sanctioned exception, and | ||
| /// before this entry existed they sat outside the guard entirely — | ||
| /// deleting their lock acquisition left the scan green, which is the | ||
| /// opposite of what their comments told the next contributor. | ||
| /// Listing the helper keeps the escape hatch narrow: it is still the lock | ||
| /// that is required, the wrapper just stops hiding the requirement. | ||
| const MUTATIONS: [&str; 3] = ["set_var(", "remove_var(", "RestoredEnvVar::set("]; | ||
| /// | ||
| /// `UnsetKeyOnExit::arm(` (`crates/rocm-core/src/test_env.rs`) is here | ||
| /// for the same reason, from the other side: `UnsetKeyOnExit` is a guard | ||
| /// whose `Drop` removes a variable, so the mutation happens in an impl the | ||
| /// scan never enters. What the test body does contain is the call that | ||
| /// builds the guard, and that call is what is listed. A guard constructed | ||
| /// directly from its type, rather than through that call, would leave | ||
| /// nothing for this list's text to match; build such a guard only through | ||
| /// the listed call. | ||
| /// | ||
| /// `EnvVarGuard::set(` (`crates/rocm-core/src/examine/framework.rs`) is the | ||
| /// same shape: its constructor sets the variable and its `Drop` restores | ||
| /// it, both in impls the scan never enters, so the test that builds one | ||
| /// shows neither call. | ||
| const MUTATIONS: [&str; 5] = [ | ||
| "set_var(", | ||
| "remove_var(", | ||
| "RestoredEnvVar::set(", | ||
| "UnsetKeyOnExit::arm(", | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Non-blocking, pre-existing (not introduced by this PR), flagged for pattern relevance. This PR documents and fixes exactly one instance of "a guard whose ( Now that the general rule is written down in
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Confirmed, and fixed in 4b2d114.
|
||
| "EnvVarGuard::set(", | ||
| ]; | ||
|
|
||
| /// Named helpers that serialize env mutation for their whole scope. | ||
| /// | ||
|
|
@@ -489,6 +509,41 @@ mod tests { | |
| ); | ||
| } | ||
|
|
||
| /// Every entry in [`MUTATIONS`] still names a call made somewhere in the | ||
| /// tree. | ||
| /// | ||
| /// The entries are text, so renaming a wrapper or guard leaves its entry | ||
| /// matching nothing: the scan stays green while the renamed call is | ||
| /// outside it, which is the silent failure this module is written | ||
| /// against. Matched on code with literals and comments stripped, so a | ||
| /// mention in a doc comment does not keep a dead entry alive, and this | ||
| /// file is skipped because its fixtures exist to spell the calls. | ||
| #[test] | ||
| fn every_listed_mutation_is_still_called_in_the_tree() { | ||
| let this_file = Path::new(env!("CARGO_MANIFEST_DIR")).join("src/env_mutation_contract.rs"); | ||
| let code: Vec<String> = workspace_rust_sources() | ||
| .iter() | ||
| .filter(|path| **path != this_file) | ||
| .map(|path| { | ||
| let text = std::fs::read_to_string(path) | ||
| .unwrap_or_else(|e| panic!("reading {}: {e}", path.display())); | ||
| strip_literals_and_comments(&text) | ||
| }) | ||
| .collect(); | ||
|
|
||
| let stale: Vec<&str> = MUTATIONS | ||
| .iter() | ||
| .copied() | ||
| .filter(|call| !code.iter().any(|text| text.contains(call))) | ||
| .collect(); | ||
| assert!( | ||
| stale.is_empty(), | ||
| "MUTATIONS lists calls nothing in the tree makes any more, so the \ | ||
| scan no longer covers whatever replaced them -- rename the entry \ | ||
| with the symbol, or drop it: {stale:?}" | ||
| ); | ||
| } | ||
|
|
||
| /// A mutating call, assembled at runtime. | ||
| /// | ||
| /// Spelling `env::set_var` literally in a fixture would make this file's own | ||
|
|
@@ -518,6 +573,15 @@ mod tests { | |
| format!("let _restore = crate::test_env::{path}(\"KEY\", &value);") | ||
| } | ||
|
|
||
| /// A guard whose `Drop` mutates, armed by a test that never calls a | ||
| /// mutation itself. | ||
| /// | ||
| /// Assembled rather than written out for the same reason as | ||
| /// [`mutation_call`]. | ||
| fn held_mutating_guard(constructor: &str) -> String { | ||
| format!("let _held = {constructor}(&lock);") | ||
| } | ||
|
|
||
| fn unguarded_test(body: &str) -> String { | ||
| format!( | ||
| "#[cfg(test)]\nmod tests {{\n #[test]\n fn t() {{\n {body}\n }}\n}}\n" | ||
|
|
@@ -546,14 +610,15 @@ mod tests { | |
| /// `remove_var` inside a `#[test]` was flagged at all. | ||
| /// | ||
| /// The length assertion is what stops the next entry arriving untested: a | ||
| /// third call added to the const fails here until it gains a case. | ||
| /// new call added to the const fails here until it gains a case. | ||
| #[test] | ||
| fn every_mutating_call_is_flagged_inside_an_unguarded_test() { | ||
| let kinds = ["set_var", "remove_var"]; | ||
| let delegated = ["RestoredEnvVar::set"]; | ||
| let held = ["UnsetKeyOnExit::arm", "EnvVarGuard::set"]; | ||
| assert_eq!( | ||
| MUTATIONS.len(), | ||
| kinds.len() + delegated.len(), | ||
| kinds.len() + delegated.len() + held.len(), | ||
| "a call was added to MUTATIONS without a case here" | ||
| ); | ||
| for kind in kinds { | ||
|
|
@@ -572,6 +637,14 @@ mod tests { | |
| "an unguarded {path} inside a #[test] must be flagged" | ||
| ); | ||
| } | ||
| for name in held { | ||
| let source = unguarded_test(&held_mutating_guard(name)); | ||
| assert_eq!( | ||
| env_mutations_in_unserialized_tests(&source).len(), | ||
| 1, | ||
| "a test holding an unguarded {name} must be flagged" | ||
| ); | ||
| } | ||
| } | ||
|
|
||
| /// A restoring wrapper does not exempt the test that drives it. | ||
|
|
@@ -601,6 +674,33 @@ mod tests { | |
| ); | ||
| } | ||
|
|
||
| /// A guard whose `Drop` mutates does not exempt the test that holds it. | ||
| /// | ||
| /// The mirror of | ||
| /// [`a_mutation_delegated_to_a_restoring_wrapper_still_needs_the_lock`] | ||
| /// for a guard: the test body only calls the guard's constructor, so the | ||
| /// scan enforces the lock only while that call is in [`MUTATIONS`]. | ||
| /// Unguarded is an offense, guarded is not. | ||
| #[test] | ||
| fn holding_a_mutating_guard_still_needs_the_lock() { | ||
|
rominf marked this conversation as resolved.
Resolved
|
||
| let held = held_mutating_guard("UnsetKeyOnExit::arm"); | ||
|
|
||
| let hits = env_mutations_in_unserialized_tests(&unguarded_test(&held)); | ||
| assert_eq!(hits.len(), 1, "holding the guard mutates on drop: {hits:?}"); | ||
| assert_eq!( | ||
| hits[0].call, "UnsetKeyOnExit::arm", | ||
| "the offender is named by the constructor call" | ||
| ); | ||
|
|
||
| let guarded = unguarded_test(&format!( | ||
| "let lock = SOME_TEST_LOCK.lock().unwrap();\n {held}" | ||
| )); | ||
| assert!( | ||
| env_mutations_in_unserialized_tests(&guarded).is_empty(), | ||
| "the lock above is what makes this shape acceptable" | ||
| ); | ||
| } | ||
|
|
||
| /// The qualifier is a matter of how the file imports, not of what the call | ||
| /// does. Keying the match on `env::` made a test that had written | ||
| /// `use std::env::set_var` invisible to the scan while doing exactly the | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This moves a
remove_varout of a test body, where the contract guard watches it, into aDropimpl, where it does not. The whole-tree scan stops at the end of the test function, andUnsetKeyOnExitis not in the guard'sMUTATIONSlist, so any test that uses this helper mutates the environment invisibly. I wrote one — a#[test]holdingUnsetKeyOnExitand no lock at all — andcargo test -p xtask env_mutationstayed green.That is the case the docs on
RestoredEnvVarthirty lines above already call out: "wrapping a mutation in a method hides it by default — a restoring helper added later is outside the guard until it is listed too." This is such a helper. It is also the mirror image of theScopedEnvVarmistake the guard's ownNAMED_SERIALIZERScomment records, where a save-and-restore type handed out an exemption it could not back.Correct today, since both call sites lock first. But what keeps it correct is now the doc comment "Declare it after the lock", not the guard — and this PR is about making that contract enforceable rather than assumed. Adding
"UnsetKeyOnExit"toMUTATIONSinxtask/src/env_mutation_contract.rsrestores enforcement; note that list has a sibling test asserting every entry is detected, so it wants a fixture case alongside.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, this is the helper the
RestoredEnvVardocs warn about. In59a1193d,UnsetKeyOnExitis inMUTATIONS, with no(because a unit struct is only named, never called. The every-entry fixture has a matching case, built at runtime so the xtask file doesn't match its own scan. Your experiment now fails: a#[test]holdingUnsetKeyOnExitwith no lock failscargo test -p xtask env_mutationat that line. With the entry removed, the whole-tree scan goes green again and the every-entry fixture fails, so the listing can't be dropped silently. Both real call sites lock first, and the whole-tree scan still passes.