Audit tests: remove change-detectors, fix slow lanes

[cbusillo]

Assign to: Claude Fable (owner's instruction). Earlier passes by other models did not flag these patterns, so do not assume the existing tests were already reviewed for this.

Problem

Across these repositories, tests have been blocking intended changes (version bumps, toolchain bumps, workflow edits, dependency updates) without catching real defects, and CI is slow. Many were written by coding agents to lock in their own change, which produces tests that restate the implementation. The quick scan below shows few markers here, so this may be a short audit-and-confirm. Confirm rather than assume.

Evidence for this repo (grep-level scan, 2026-09-19 - indicators, not proof)

Measure	Value
Test files / test lines	8 / 965
Test files that read workflow YAML	0
64-hex (hash) literals in tests	0
Version-like string literals in tests	0
Slowest of the last 20 successful CI runs	1 min

This repository began as a fork, but upstream is abandoned and this is now the maintained project (published to PyPI). Audit the whole test suite and CI as first-party code, including whether the inherited tests cover the behaviour the PyPI package promises.

Phase 1 - rubric pass

A test stays only if it fails when the product is broken and passes when someone makes an intended change.

• Tests that assert literals defined elsewhere (versions, build numbers, toolchain, hashes): replace with agreement against one source of truth, or delete.
• Tests that assert workflow or config text: delete. Enforce the rule where it executes (the workflow itself, a helper script with its own unit test, or actionlint).
• Verification or loading code must not depend on the state of the working tree; check live state only on the path that acts on it.
• Generated inventories or snapshots must not require a regeneration commit for unrelated changes.
• Keep byte-exact and hash gates on real artifacts and on immutable evidence.

Speed

• Measure first: time per CI lane and the 20 slowest tests.
• Run lanes only when the files they cover change; do not run the full product suite for docs-only or evidence-only changes.
• Cache toolchains and dependencies.
• Set a pull-request time budget and have CI report when it is exceeded.

Phase 2 - deeper test and code audit (as rate limits allow, after Phase 1)

Answer "are we testing the right things?" with history, not opinion:

• Friction ledger: over recent history, classify test failures as real defect caught versus intended change blocked; list tests edited in lockstep with the code they cover.
• Risk map: name the ways this product actually fails for users and check that each has a test that would catch it; list high-risk code with no meaningful test.
• Tests that have never failed, duplicate coverage, and tests that mock away the thing they claim to test.
• Output: a keep / rewrite / delete / add list with reasons, then small pull requests.

Done when

• Every removed assertion is listed with its category in the pull request.
• Planted faults (deliberately broken product behaviour) still fail the suite.
• A version bump or toolchain bump needs no test edits.
• CI time is reported before and after.
• A guard rule is added to AGENTS.md / CLAUDE.md: no test may assert a literal defined elsewhere or assert workflow text; verification code must not depend on working-tree state.