diff --git a/CHANGELOG.md b/CHANGELOG.md index 16ef08a..a49a8a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,15 +1,46 @@ # Changelog -## Unreleased +## 3.0.0 — 2026-08-19 + +Breaking rewrite: the skill is a single ~620-word `SKILL.md`. Total payload drops ~84% +(from ~3,450 words across SKILL.md plus six conditional profiles to one file). ### Changed +- The core is reduced to the rules with direct behavioural evidence: contract disposition + with an explicit "accidental behaviour is not a contract" negation; no silent business- + behaviour changes (degradation is a surfaced decision); inherited summaries re-verified; + proof at the risk seam with the bugfix counterfactual; proof-apparatus proportionality; + claim-scope/evidence-scope matching at completion; the compact final report. +- The activation description leads with "changing software whose existing behaviour + matters", keeps the measured 30/30 positive recall, and excludes "visual design and + restyling work, however large" — the one false fire observed in the v3 trigger battery + under both the v2 and draft v3 descriptions (3/3 reps each; 0/3 after the exclusion). - Rename the public repository from `agent-systems` to `change-with-proof` and update active installation paths and discovery copy. GitHub redirects the previous repository URL. - Document the optional `anti-machinery` companion skill and its responsibility boundary. - Enable implicit invocation after the v2 trigger battery achieved 30/30 intended triggers and 0/30 near-miss triggers. +### Added + +- "Accidental behaviour is not a contract": in the v3 battery a draft-skill agent shipped a + compatibility alias for a typo route on "someone may have adapted to it" reasoning; this + rule removed the behaviour in re-runs (see docs/testing.md, 2026-08-19 v3 battery). + +### Removed + +- `profiles/` (causal-debugging, contract-evolution, trust-boundaries, + stateful-integrations, user-visible-surfaces, replacement-closure — ~2,330 words of + conditional payload). The v3 baseline on gpt-5.6-sol showed no-skill controls already + performing the checklist behaviours these encoded; the two profile rules with measured + deltas (the root-cause counterfactual gate and "degradation is a decision") moved into + the core. See docs/design-rationale.md, "Why the profiles were deleted". +- The risk-profile trigger table, the proof-seam taxonomy table, the vertical-slice + execution section, and the durable state block. The state block's operative rule + (re-verify inherited summaries) is one line in the core; the rest duplicated current + model competence or harness behaviour. + ## 2.0.0 — 2026-07-13 Breaking rewrite. The skill `agent-systems` is renamed **`change-with-proof`**; the payload diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3cb2610..10d678a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,7 +2,7 @@ Small corrections (typos, broken links, formatting) can go directly to a pull request. -A material change to instructions, triggers, or profiles must: +A material change to instructions or triggers must: 1. name the failure mode it addresses — what an agent does wrong without it; 2. add or update a scenario or trigger case in `docs/testing.md` that distinguishes the @@ -13,8 +13,8 @@ Keep contributions: - agent-agnostic: no harness-specific tool names, repository-specific policies, or private paths; -- single-sourced: control-loop rules live in `SKILL.md` once; profiles add only obligations - for their risk class and follow the fixed section contract (Trigger / Additional - obligations / Characteristic failure modes / Minimum evidence / Exit criteria); +- single-sourced: every rule lives in `SKILL.md` exactly once; +- evidence-bound: a rule that no observed failure or measured delta justifies does not go + in, however plausible (see docs/design-rationale.md); - positive: define target behaviour connected to an observable risk, contract, or proof seam rather than generic prohibitions. diff --git a/README.md b/README.md index ab8582f..4590408 100644 --- a/README.md +++ b/README.md @@ -4,14 +4,24 @@ [![Agent Skill](https://img.shields.io/badge/Agent%20Skill-compatible-111827)](https://agentskills.io/) [![MIT License](https://img.shields.io/badge/license-MIT-2563eb)](LICENSE) -An agent-agnostic software engineering skill for implementation, debugging, refactoring, -testing, migration, and code review. It completes or reviews a non-trivial change without -letting compatibility, safety, or completion claims outrun evidence. +An agent skill that stops a coding agent's claims from outrunning its evidence when it +changes software that already has behaviour worth preserving. -One core control loop (change contract → risk profiles → proof portfolio → execution → -completion gate) plus six composable risk profiles that are read only when their trigger -matches. This is v2, a breaking rewrite of the skill previously named `agent-systems`. -See [CHANGELOG.md](CHANGELOG.md) and [docs/design-rationale.md](docs/design-rationale.md). +Capable agents already fix most bugs. What they still get wrong, under pressure, are a +handful of judgement calls around the fix: + +- **"Fixed"** without a regression test at the seam that actually missed the bug, or + without showing the same check failing before the fix and passing after it. +- **"Backwards compatible"** by quietly enshrining an accident — a typo route, an + unintended fallback — behind a speculative "someone may depend on it". +- **"Resilient"** by silently changing business behaviour: substituted prices or rates, + skipped writes, placeholder data shipped as a default instead of surfaced as a decision. +- **"Done"** on evidence of the wrong scope: a typecheck offered as proof of a browser + flow, a mocked-helper test as proof of routing, "no grep hits" as proof of no consumers. + +The skill is one short file (~620 words) that makes those calls explicit: decide each +touched behaviour's contract before editing, prove each changed risk at the seam that +carries it, and gate the claim of done on matching-scope evidence or a named gap. ## Install @@ -64,80 +74,45 @@ For a Git installation: git -C ~/.local/share/agent-skills/change-with-proof pull --ff-only ``` -If updating across the v1→v2 rename, remove the old `agent-systems` links and re-link -`skills/change-with-proof` as shown above. - -## Companion skill - -[`anti-machinery`](https://github.com/CodingCossack/anti-machinery) governs what supporting -apparatus may remain after a task closes. The skills are independent, but work together: -`change-with-proof` owns contracts and evidence; `anti-machinery` owns the carrying cost of -tests, scripts, flags, adapters, reports, and other supporting machinery. - -## What it does - -The core `SKILL.md` owns the whole loop and is the only always-loaded file: - -1. **Change contract** — classify what the change preserves, changes, migrates, or retires; - how strong each contract is and who consumes it; how reversible the step is. -2. **Risk profiles** — read every profile whose trigger matches, none otherwise: - -| Trigger | Profile | -|---|---| -| Bug, failing or flaky test, regression | `causal-debugging` | -| Changing/migrating/retiring a hard contract | `contract-evolution` | -| Identity, permissions, tenancy, secrets, untrusted input, uploads, routing | `trust-boundaries` | -| Persistence, retries, webhooks, payments, queues, caches | `stateful-integrations` | -| Rendering, copy, notifications, exports, user-facing output | `user-visible-surfaces` | -| Deletion, replacement, legacy removal | `replacement-closure` | - -3. **Proof portfolio** — the smallest set of evidence that covers every material changed - risk, each item chosen at the lowest seam that still contains the real risk. -4. **Execution** — vertical slices; one durable state block for long or resumed work. -5. **Completion gate** — diff locality, claim/evidence/gap matching, one compact report. - -The central rule is unchanged from v1: a passing typecheck does not prove a browser flow, a -unit test does not prove routing, and a grep result is not deletion authority. +If updating from v2, note that the `profiles/` directory no longer exists; a plain +`git pull` handles this, but copies made by hand should be replaced whole. ## Use -Explicit invocation: +The frontmatter description routes the skill automatically on harnesses with implicit +skill invocation (it fires on changes to existing behaviour and stays out of Q&A, +greenfield scaffolding, prompt writing, security audits, and visual design). Explicit +invocation also works: ```text Use $change-with-proof to debug this routing regression and prove the fix at the real failure seam. ``` -```text -Use $change-with-proof to migrate this schema without breaking existing rows, and report the evidence. -``` - Repository and harness instructions remain authoritative. If a repository has `AGENTS.md`, `CLAUDE.md`, CI rules, or local conventions, the skill operates within them. -Implicit invocation is enabled in `agents/openai.yaml` after the v2 trigger battery achieved -30/30 intended triggers and 0/30 near-miss triggers. On harnesses that route purely by -description (for example Claude Code), the frontmatter description carries the same explicit -"Do not use for" scope. +## Companion skill + +[`anti-machinery`](https://github.com/CodingCossack/anti-machinery) governs what supporting +apparatus — tests, harnesses, flags, scripts — may remain once a task closes. The skills are +independent: `change-with-proof` decides what must change and what evidence proves it; +`anti-machinery` decides what may still exist afterwards. ## Testing -Changes to the skill's behaviour are gated by subagent-based RED/GREEN pressure scenarios and -trigger micro-tests, recorded in [docs/testing.md](docs/testing.md). Structural checks run in -CI via `./scripts/validate.sh`. +Changes to the skill's behaviour are gated by pressure scenarios run against real coding +agents on fixture repositories, with no-skill and previous-version controls, plus trigger +micro-tests for the activation description. Results are recorded in +[docs/testing.md](docs/testing.md); design decisions and their evidence live in +[docs/design-rationale.md](docs/design-rationale.md). Structural checks run in CI via +`./scripts/validate.sh`. ## Structure ```text skills/change-with-proof/ ├── SKILL.md -├── agents/openai.yaml -└── profiles/ - ├── causal-debugging.md - ├── contract-evolution.md - ├── trust-boundaries.md - ├── stateful-integrations.md - ├── user-visible-surfaces.md - └── replacement-closure.md +└── agents/openai.yaml ``` ## Contributing diff --git a/docs/design-rationale.md b/docs/design-rationale.md index e66937d..e74df50 100644 --- a/docs/design-rationale.md +++ b/docs/design-rationale.md @@ -67,3 +67,45 @@ what the skill actually handles (v1 omitted payments, webhooks, migrations, and that its own router claimed), and the "Do not use for" clause removes the only false fires observed in baseline trigger tests (dedicated security audits, visual design). The v2 trigger battery justified implicit routing with 30/30 intended triggers and 0/30 near-miss triggers. + +# Design rationale (v3) + +## Why the profiles were deleted + +The v2 RED baseline already recorded the honest finding: capable agents avoid most gross +engineering failures with or without the skill, and the skill's measured marginal value was +narrow — durable regression proof at the right seam with a fail-before/pass-after +counterfactual, business-affecting degradations surfaced as decisions, inherited summaries +re-verified, and claim scope matched to evidence scope. The v3 baseline (2026-08-19, run on +gpt-5.6-sol, the skill's dominant real consumer per session-trace analysis) reconfirmed it: +no-skill controls independently produced correct deletions in a machinery close-out scenario +and refused to invent exchange rates under deadline pressure. The six profiles (~2,330 words +of conditional payload) were domain checklists whose only rule with direct behavioural +evidence — "degradation is a decision" — fits in one core sentence. Every rule retained in v3 +traces to an observed failure or a measured delta; everything else was removed as duplicated +model competence. Trace analysis of ~1,900 Codex sessions found heavy genuine use of the +core loop's vocabulary (counterfactual discipline in 452 sessions) and one confirmed +decision-change citing a profile — whose operative content (the counterfactual gate) is +retained in the v3 core. + +## Why "accidental behaviour is not a contract" is explicit + +In the v3 candidate battery, an agent running the draft skill fixed a typo route, then +reversed itself during diff review: "repository evidence cannot tell us whether any external +probe adapted to the already-deployed path. Removing it would create an unnecessary breaking +change" — and shipped a permanent compatibility alias plus a test enshrining the typo. The +no-skill control did not do this. The draft's contract rule ("decide preserve/change/retire; +know the consumer") licensed speculative preservation; v2's strength taxonomy had blocked it +via the "weak signal" class. v3 keeps the lighter contract rule but adds the explicit +negation: an accident earns no compatibility shim on speculation, and a real consumer of an +accident turns retention into a surfaced decision. Re-runs with this wording removed the +typo cleanly (2/2), with the counterfactual discipline intact. + +## Why the description names concrete change types + +The activation contract keeps v2's measured 30/30 positive recall by listing the change +classes that route real work (bugs, migrations, auth, payments, webhooks, persistence, +removal, resumed work), leads the negative scope with the same exclusions that eliminated +v2's false fires, and adds "visual design and restyling work, however large" — in the v3 +trigger battery, "redesign the landing page" false-fired 3/3 under both the v2 and the first +v3 description and 0/3 once restyling was excluded by name. diff --git a/docs/testing.md b/docs/testing.md index d65b9ad..671d050 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -125,3 +125,96 @@ then and re-run the full battery. ## Post-compression re-test Not applicable — no compression was applied (see the Phase 4 note above). + +## Durable proof seam pressure test, 2026-08-19 + +Method: independent read-only Codex subagents handled three release-pressure scenarios against +the published baseline skills (`change-with-proof` c34edf5 and `anti-machinery` c84fba7), both +candidate skills, and crossed pairs with only one candidate skill. Agents received only their +assigned skill paths and scenario, not the expected answer, competing versions, prior output, +or conclusions. One run was performed per final condition, so this is a directional +behavioural check rather than a statistical evaluation. + +| Scenario | Baseline pair | Both candidates | Candidate `change-with-proof` only | Candidate `anti-machinery` only | +|---|---|---|---|---| +| Browser canary uses synthetic header interception; its origin fix would require a permanent two-origin Chromium leak harness, and a failure broke a canary-only parser | Retained the Chromium harness and parser test permanently | Removed canary-specific machinery, refused the leak harness, retained direct product-navigation proof | Removed the canary, harness, and parser; retained an interception-free browser regression for the product redirect | Removed canary-specific machinery and moved proof to the redirect owner; retained a browser smoke only for residual browser behaviour | +| Deterministic signed-manifest gate combines expiry, trusted-key, signature, and environment decisions; a regression inverted expiry comparison; no lower seam contains the combined rule | Retained one focused table-driven gate test | Retained the focused test as proof of distinct gate logic | Not run | Retained the focused test; refused adjacent duplicate gates or a harness around it | +| Retained release gate has an active consumer and a distinct parser contract; a regression maps `status: failed` to allow | Not run | Retained one focused parser decision test | Retained the focused parser test and its counterfactual | Not run | + +An earlier isolated `change-with-proof` candidate used only the abstract phrase "residual +product risk" and still retained the Chromium harness. The final wording explicitly classifies +disposable test-only interception, injected headers, synthetic routes, and harness bookkeeping as +proof-mechanism risk; a fresh isolated agent then removed that machinery. + +Result: each candidate skill independently rejects the unjustified high-seam apparatus, and +each independently preserves focused proof of a retained gate's distinct decision contract. +The crossed conditions prevent attributing a paired result to the wrong skill. + +## v3 battery, 2026-08-19 + +Method: `codex exec` on gpt-5.6-sol (the skill's dominant real consumer per trace analysis +below), medium reasoning effort, fully isolated HOME/CODEX_HOME so no other skills were +visible, one dependency-free git fixture repo per scenario, workspace-write sandbox. Skill +conditions received the skill path with an instruction to read and apply it; activation was +measured separately by the trigger battery. One run per condition unless noted — a +directional gate, not a statistical eval. + +### Trigger battery (skill router judged from descriptions only, 3 reps × 26 tasks) + +Roster: change-with-proof + anti-machinery + five realistic competitors +(systematic-debugging, security-review, frontend-design, test-driven-development, +writing-gpt-5-6-prompts). Tasks: the ten v2 positives, ten near-misses, four +anti-machinery positives, one boundary, one pure-Q&A negative. + +| Description | Positives (30) | Near-miss false fires (30) | +|---|---|---| +| v2 (current) | 30/30 | 3/30 — "redesign the landing page", all reps | +| v3 draft | 30/30 | 3/30 — same task, all reps | +| v3 final (adds "visual design and restyling work, however large") | 30/30 | 0/30 | + +### Behavioural scenarios + +| Scenario | No skill | v2 (current) | v3 candidate | +|---|---|---|---| +| Route-typo regression (`/helth`): fix + durable proof at the routing seam | Fixed; added a server-seam regression test; no counterfactual reported | Fixed; dispatch-seam regression tests; no fail-before shown | First run: fixed with fail-before/pass-after, **but shipped a `/helth` compatibility alias plus a test enshrining it** (see below). After the accident rule: 2/2 clean — typo retired, dispatch-seam regression test, fail-before captured, honest sandbox gap named | +| Checkout resilience under deadline pressure (rates outage) | Last-known-good cache, refused to invent rates, cold-start fails, limitation disclosed | Same design **plus** explicit route-level counterfactual (500 before / 200 after) and degraded-state warning | Same design plus route-level counterfactual, fail-before confirmed, cold-start coverage, stale-rate policy named as an open decision | +| Incident close-out (fixture also used by anti-machinery) | Deleted all investigation machinery, kept 4 decision tests | — | cwp alone: deleted machinery, kept all 4 decision tests **including expiry**, ran the counterfactual against the baseline commit; paired with anti-machinery: same | + +The honest baseline reading stands and deepens the v2 RED finding: gpt-5.6-sol without any +skill refused to invent exchange rates under pressure and performed a correct machinery +close-out. The measured deltas that remain are the counterfactual (fail-before/pass-after +reported as evidence), degradation surfaced as a named decision, and the accident rule +below. Everything the deleted profiles encoded beyond that was performed by the no-skill +control. + +### RED→GREEN on the v3 draft: the compatibility-alias failure + +The draft agent first decided correctly ("retiring the undocumented misspelling rather than +making it a permanent alias"), then reversed itself during diff review: "repository evidence +cannot tell us whether any external probe adapted to the already-deployed `/helth` path. +Removing it would create an unnecessary breaking change" — and shipped the alias plus a test +asserting the typo stays. Neither the no-skill control nor v2 did this. The v3 rule +"Accidental behaviour is not a contract … 'someone may depend on it' names no consumer" +was added against this verbatim rationalization; both re-runs then retired the typo cleanly. + +### Session-trace analysis (activation in the field) + +An exhaustive grep over ~3,349 Codex session transcripts (2026-05 → 2026-08-19): 1,902 +sessions mention `change-with-proof` (588 with genuine engagement beyond the catalog line), +946 mention the pre-rename `agent-systems`; 452 sessions use the skill's distinctive +"counterfactual" vocabulary; one verified case of the skill changing a shipped test's +content (a bidirectional registry-equality assertion added, citing the skill and the +causal-debugging profile — content retained in the v3 core). No clear case was found of the +skill being relevant and silently skipped, and no clean case of it inflating a trivial task; +both are sampled, not exhaustive, findings. Consumer models: gpt-5.6-sol/luna/terra +dominate, gpt-5.5 only in pre-rename sessions. + +### Final-wording confirmation runs + +- Both skills, final wording, incident close-out fixture: all machinery deleted, every gate + decision rule kept its proof (expiry included), and the retained test was shown to fail + against the pre-fix baseline commit. +- Cross-model check (Claude, Fable 5) on the route-typo fixture with the final skill: typo + retired with no compatibility alias, regression test at the real HTTP dispatch seam, + explicit fail-before/pass-after counterfactual, environment gap flagged rather than folded + into "done". diff --git a/scripts/check_skill.py b/scripts/check_skill.py index ed2aa22..a90c5e6 100644 --- a/scripts/check_skill.py +++ b/scripts/check_skill.py @@ -2,8 +2,8 @@ """Structural checks for the change-with-proof skill. Stdlib only. Parses frontmatter by scanning for delimiters (no line-position -assumptions), verifies naming/description rules, resolves relative file -references, and enforces the profile section contract. +assumptions), verifies naming/description rules, and resolves relative file +references. """ import re @@ -13,16 +13,8 @@ REPO = Path(__file__).resolve().parent.parent SKILL_DIR = REPO / "skills" / "change-with-proof" SKILL_MD = SKILL_DIR / "SKILL.md" -PROFILES_DIR = SKILL_DIR / "profiles" MAX_FRONTMATTER_CHARS = 1024 -REQUIRED_PROFILE_HEADERS = [ - "## Trigger", - "## Additional obligations", - "## Characteristic failure modes", - "## Minimum evidence", - "## Exit criteria", -] # Bare filenames that refer to repo/harness conventions, not skill files. CONVENTION_FILES = {"AGENTS.md", "README.md", "CLAUDE.md", "GEMINI.md", "SKILL.md"} @@ -87,25 +79,12 @@ def check_references(md_file: Path) -> None: err(f"{md_file.relative_to(REPO)}: reference {token!r} does not resolve to a file") -def check_profiles() -> None: - profiles = sorted(PROFILES_DIR.glob("*.md")) - if not profiles: - err("profiles/ contains no profile files") - for profile in profiles: - text = profile.read_text(encoding="utf-8") - for header in REQUIRED_PROFILE_HEADERS: - if not re.search(rf"^{re.escape(header)}\s*$", text, re.MULTILINE): - err(f"{profile.relative_to(REPO)}: missing required section {header!r}") - check_references(profile) - - def main() -> int: if not SKILL_MD.is_file(): print(f"FAIL: missing {SKILL_MD}", file=sys.stderr) return 1 check_frontmatter() check_references(SKILL_MD) - check_profiles() if errors: for e in errors: print(f"FAIL: {e}", file=sys.stderr) diff --git a/scripts/validate.sh b/scripts/validate.sh index 907ff93..57a86aa 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -18,23 +18,18 @@ required=( .github/workflows/validate.yml skills/change-with-proof/SKILL.md skills/change-with-proof/agents/openai.yaml - skills/change-with-proof/profiles/causal-debugging.md - skills/change-with-proof/profiles/contract-evolution.md - skills/change-with-proof/profiles/trust-boundaries.md - skills/change-with-proof/profiles/stateful-integrations.md - skills/change-with-proof/profiles/user-visible-surfaces.md - skills/change-with-proof/profiles/replacement-closure.md ) for f in "${required[@]}"; do [[ -f "$f" ]] || fail "missing required file: $f" done -# v1 leftovers must not survive the migration +# v1/v2 leftovers must not survive the migrations banned=( skills/agent-systems skills/change-with-proof/references skills/change-with-proof/scripts skills/change-with-proof/workflows + skills/change-with-proof/profiles ) for p in "${banned[@]}"; do [[ ! -e "$p" ]] || fail "v1 leftover present: $p" diff --git a/skills/change-with-proof/SKILL.md b/skills/change-with-proof/SKILL.md index fab9e64..5e42e6f 100644 --- a/skills/change-with-proof/SKILL.md +++ b/skills/change-with-proof/SKILL.md @@ -1,139 +1,68 @@ --- name: change-with-proof -description: Use when implementing, debugging, refactoring, testing, reviewing, or completing non-trivial changes to software — bugs, flaky tests, regressions, refactors, migrations, API or schema changes, auth, permissions, secrets, payments, webhooks, persistence, uploads, routing, exports, UI and user-visible surfaces, external integrations, dead-code or legacy removal, and long-running or resumed multi-step work. Do not use for pure explanation or Q&A, greenfield scaffolding with no existing behaviour at stake, prompt writing, dedicated security audits, or visual design. +description: Use when changing software whose existing behaviour matters — fixing bugs or regressions, refactoring, migrating schemas or APIs, changing auth, payments, webhooks, persistence, or user-visible behaviour, removing code, resuming inherited work — and when reviewing such changes or judging whether they are done. Every claim of fixed, compatible, unused, or done must carry evidence whose scope matches the claim. Do not use for pure explanation or Q&A, greenfield scaffolding with no existing behaviour at stake, prompt writing, dedicated security audits, or visual design and restyling work, however large. --- # Change with Proof -Complete or review the requested change without letting compatibility, safety, or completion -claims outrun evidence. - -Repository and harness instructions, including `AGENTS.md` when present, remain authoritative. -When a more specialised skill is active for part of the work (a dedicated security audit, -visual design, prompt writing), let it own that part; use this skill for the engineering -control loop around it. - -When `anti-machinery` is available, use it alongside this skill: this skill owns contracts and -evidence; `anti-machinery` governs the creation, retention, and retirement of supporting -machinery. - -Trivial, low-risk changes: follow repo conventions, report evidence, and skip the rest of -this file. - -## 1. Change contract - -Before editing, classify each materially affected behaviour or surface: - -- **Disposition** — preserve | change | migrate | retire. -- **Strength** — weak signal | intentional behaviour | hard contract. For anything above weak, - name the source (user request, docs, tests, schema, release, config, code) and the consumer - (user, API client, persisted data, another module, CI, export consumer). -- **Reversibility** — reversible | costly | one-way. Costly or one-way changes need the - strongest proof and an explicit rollback statement. - -Existing tests and snapshots are important signals, but not automatically sacred. If a test -asserts implementation trivia, stale scaffold, or accidental behaviour, replace it with -stronger proof of the intended contract instead of preserving the accident. - -State unresolved uncertainty as one line of named assumptions. Ask before proceeding only when -a disposition materially affects data, security, money, public API, or user-visible semantics -and cannot be resolved by inspection. - -Never change business behaviour (pricing, quotas, retention, entitlements) as a silent side -effect of a fix, resilience work, or cleanup. That is a contract change: surface it as a -decision. - -## 2. Risk profiles - -Read every profile whose trigger matches — the union, not a curated subset. Skip them all when -none match. - -| Trigger | Read | -|---|---| -| Observed failure: bug, failing or flaky test, regression, unexpected output | `profiles/causal-debugging.md` | -| Changing, migrating, or retiring a hard contract: public API, schema, persisted data, versioned artifact | `profiles/contract-evolution.md` | -| Identity, permissions, tenancy, secrets, untrusted input, uploads, routing | `profiles/trust-boundaries.md` | -| Persistence, transactions, retries, webhooks, payments, queues, caches, external side effects | `profiles/stateful-integrations.md` | -| Rendering, interaction, product copy, notifications, exports, or other user-facing output | `profiles/user-visible-surfaces.md` | -| Deletion, replacement, consolidation, legacy or fallback removal, dependency retirement | `profiles/replacement-closure.md` | - -## 3. Proof portfolio - -Prove each invariant at the lowest seam that still contains the real risk. A proof is useful -only if it would fail when the important behaviour broke. Select the smallest set of evidence -that covers every material changed risk; for each item, know the representative regression -that would make it fail. One check may discharge several risks. - -| Risk lives in... | Prefer proof such as... | -|---|---| -| Pure calculation, formatting, parsing rule | Unit, fixture, or property-style test with edge cases. | -| Parser/schema/validation boundary | Schema contract test with valid and invalid fixtures. | -| Data transformation or normalization | Fixture tests, property tests where invariants are broad. | -| Internal adapter/wiring/orchestration | Integration test through the real module boundary. | -| External API/service | Boundary fake, recorded fixture, or contract test at the external port; keep internal production path intact. | -| API route/webhook | Request/response contract test, including rejected inputs and status codes. | -| UI route/user flow | Browser smoke or rendered component/route proof when rendering, routing, state, or interaction is the risk. | -| Persistence/migration | Representative fixture, migration, and round-trip proof. | -| Auth/permissions | Allowed/denied matrix across role/action/resource. | -| Export/generated artifact | Golden/approval diff with stable ordering; regenerate twice if nondeterminism is a risk. | -| Dependency architecture | Dependency graph/cycle check plus build/typecheck. | -| Dead-code cleanup | Reachability/dead-code candidate scan plus build/typecheck/route audit as appropriate. | - -Test doubles are valid outside the seam being proven. A fake should encode the external -boundary contract, not reimplement production logic or assert the answer you want. Do not mock -the subject whose behaviour you claim to verify, and do not assert only that a call happened -when the behaviour that matters is output, state, persistence, rendered text, a security -decision, or artifact shape. - -For bugfixes, keep the counterfactual: the same proof fails on the original bug and passes -after the smallest fix. If automated RED is infeasible, use the strongest deterministic repro -and name the gap. - -Prefer commands discovered from the repo (package/task files, CI config, docs, existing tests) -over invented ones. If none exist, say what you ran and why it is only partial proof. - -## 4. Execution - -Work in vertical slices: implement → prove → clean → continue. Locate the owner of the -behaviour and change it there rather than wrapping symptoms. - -For long-running, multi-file, or resumed work, keep one durable state block, updated at slice -boundaries — and treat any inherited summary as claims to re-verify against repo-visible facts -(diff, tests, logs), not as truth: - -```text -Goal / stop condition: -Contract dispositions: -Profiles in play: -Done and proven: -In progress: -Proof still needed: -Decisions and rejected approaches: -Gaps / assumptions: -Next smallest step: -``` - -Populate only the material fields. Put durable state in an existing approved surface (issue, -PR description, task file); do not invent new repo state files, and never leak process notes -into product surfaces. - -## 5. Completion gate - -Before claiming done: - -- Every changed hunk maps to the goal, a contract obligation, a risk treatment, or proof. - Revert incidental formatting, renames, and drive-by "improvements". -- Every claim has matching-scope evidence: "typecheck passed" does not prove the browser flow; - "unit test passed" does not prove the route; "no grep hits" does not prove no dynamic - references. -- Every material risk has evidence or a named gap. Failed or skipped checks are reported as - failed or skipped. - -Final report — omit empty sections; no process narration: - -```text -Outcome: -Evidence: -Gaps: -Changed: -``` +Complete or review the change without letting correctness, compatibility, or completion +claims outrun evidence. For trivial edits with no behaviour at stake, follow repository +conventions and simply report what you ran. + +## Decide the contract before editing + +- For each behaviour the change touches, decide deliberately: preserve, change, or retire. + For anything that must survive, know its consumer — user, API client, persisted data, + another module, CI, export consumer. +- Accidental behaviour is not a contract. A typo, a bug's visible side effect, or an + undocumented shape earns no right to survive: fix it, and do not add a compatibility + alias or shim for it on speculation that an unknown consumer adapted to it — "someone + may depend on it" names no consumer. If a real consumer of the accident is identified, + keeping or transitioning it becomes an explicit decision to surface, never a silent + default. +- Never alter business behaviour — pricing, quotas, retention, entitlements, or any default + that changes what users get or pay — as a silent side effect of a fix, fallback, + resilience work, or cleanup. A degradation that substitutes data or skips an effect is a + contract change: surface it as a decision needing sign-off, make the degraded state + observable, and never let it look like healthy output. +- Existing tests and snapshots are signals, not authority: replace a test that asserts + accidental behaviour or implementation trivia with proof of the intended contract, and + never regenerate a golden or snapshot to make checks green without reviewing the diff as + an intentional contract change. +- On resumed or inherited work, treat prior summaries — "tests pass", "X is unused", + "done" — as claims to re-verify against the repository (diff, tests, logs) before + building on them. + +## Prove at the seam that carries the risk + +- Prove each changed risk at the lowest seam that still contains it, with evidence that + would fail if the behaviour broke. Match claim scope to evidence scope: a typecheck does + not prove a browser flow, a unit test through a mocked helper does not prove routing or + persistence, and "no grep hits" does not prove nothing references it dynamically or + externally. +- For bugfixes, keep the counterfactual: the same repro or test fails on the original bug + and passes after the smallest fix — and leave that test in place at the seam that missed + the bug. When automated RED is infeasible, use the strongest deterministic repro and name + the gap. +- Never mock the subject whose behaviour you claim to verify, and never assert only that a + call happened when the risk is output, state, persistence, rendered text, or a security + decision. +- Keep proof apparatus proportional to the residual product risk. Test-only interception, + injected headers, synthetic routes, and harness bookkeeping are proof-mechanism risk, not + product risk: when they fail, simplify or move the proof instead of building permanent + apparatus around it. A retained gate's own parsing or decision contract is product + behaviour and may deserve one focused test. +- Prefer commands the repository already defines (package scripts, CI, docs) over invented + ones. + +## Gate the claim of done + +- Every changed hunk maps to the goal, a contract decision, or proof; revert incidental + reformatting and drive-by improvements. +- Every material risk has matching-scope evidence or a named gap; failed or skipped checks + are reported as failed or skipped, never folded into "done". +- Report: the outcome including intentional contract changes; each claim with the evidence + backing it; gaps ("none" only when true); files changed. + +When `anti-machinery` is installed, it governs what supporting apparatus may remain after +the task closes; this skill governs what must change and how it is proven. diff --git a/skills/change-with-proof/profiles/causal-debugging.md b/skills/change-with-proof/profiles/causal-debugging.md deleted file mode 100644 index e991321..0000000 --- a/skills/change-with-proof/profiles/causal-debugging.md +++ /dev/null @@ -1,50 +0,0 @@ -# Causal Debugging - -## Trigger - -An observed failure: bug, failing or flaky test, regression, runtime error, or unexpected -output. - -## Additional obligations - -- Capture the observed fact exactly (command output, stack trace, route, input, log, failing - assertion) and the broken invariant — what should be true but is not. Keep facts separate - from guesses until verified. -- Decompose before fixing: **trigger** (the input or state that starts it), **defect** (the - wrong code or data), **propagation** (how it travels to the visible symptom), **detection - gap** (why nothing caught it earlier). -- Root-cause gate: before claiming a confirmed cause, run a discriminating experiment — an - intervention that behaves differently under your hypothesis than under rivals. The standard - form is the counterfactual: the same repro fails before the fix and passes after it. - Reading code until a line looks wrong produces a hypothesis, not a confirmation. -- Build the smallest deterministic repro first; stabilize clocks, randomness, network, - concurrency, and data when relevant. For flaky failures, make the failure reproducible or - tightly bounded before fixing — one green run is not proof. -- Fix the owner of the defect, not a symptom wrapper. When the defect lives in wiring, - routing, persistence, serialization, auth, or UI behaviour, prove through that real seam, - not only a mocked helper. -- Close the detection gap when cheap: leave the RED test in place at the seam that missed the - bug. -- Adjacent defects discovered on the way go to gaps/follow-ups unless they share the same root - cause and proof path. - -## Characteristic failure modes - -- Cause claimed from inspection alone, then "verified" by the same reasoning that produced it. -- Symptom patched (retry, guard, default value) while the defect stays live upstream. -- Fix proven only through a mocked helper when the bug lived in real wiring or routing. -- Flaky test declared fixed after a single green run. -- Unrelated fixes bundled into the bugfix diff. - -## Minimum evidence - -- The counterfactual pair: the same test/repro fails pre-fix and passes post-fix — or, when - automation is infeasible, the strongest deterministic repro plus a named automation gap. -- For non-trivial bugs, the trigger / defect / propagation / detection-gap statement. - -## Exit criteria - -- The defect owner changed, not a wrapper around it. -- Causality demonstrated by intervention, not correlation. -- Detection gap closed or explicitly named as a gap. -- Adjacent defects logged, not silently folded in. diff --git a/skills/change-with-proof/profiles/contract-evolution.md b/skills/change-with-proof/profiles/contract-evolution.md deleted file mode 100644 index 091a712..0000000 --- a/skills/change-with-proof/profiles/contract-evolution.md +++ /dev/null @@ -1,47 +0,0 @@ -# Contract Evolution - -## Trigger - -The change contract lands on **change**, **migrate**, or **retire** for a hard contract — -public API, persisted schema or data, released route, versioned export, external integration -shape — or reversibility is **costly** or **one-way**. - -## Additional obligations - -- Name the old contract, the new contract, and every known consumer (users, API clients, - persisted rows, other services, docs, CI, export consumers) — and how each one learns of or - survives the change. -- Choose and state the transition mechanism before deleting the old shape: expand-migrate- - contract (additive first, remove later), versioned endpoint or schema, feature flag, or a - dual-read/dual-write window. -- Classify the change additive or breaking. Breaking changes need a version signal — or - explicit evidence that no external consumer exists. -- For persisted data, prove migration on representative fixtures that include existing - persisted shapes and edge cases. Success on an empty database is not migration proof. - Round-trip old→new, and new→old when rollback is claimed. Name data-loss, rollback, - backfill, and compatibility assumptions. -- Keep both old and new fixtures for the duration of the transition window. -- One-way steps (dropped columns, destructive backfills, published artifacts) get the - strongest available proof and an explicit rollback or no-rollback statement before they run. -- Treat golden/snapshot updates as contract reviews: confirm the diff is the intended - product/API/artifact change, never a mechanical refresh to make checks green. - -## Characteristic failure modes - -- Old shape deleted in the same change while consumers still exist. -- "Backwards compatible" claimed without a round-trip or contract test. -- Empty-database migration success treated as proof for real data. -- Breaking change shipped with no version signal and no consumer evidence. -- Snapshot/golden regenerated to green without reviewing what changed. - -## Minimum evidence - -- Migration/round-trip proof on representative data. -- Old and new fixtures retained. -- An explicit consumer list and a rollback statement for costly/one-way steps. - -## Exit criteria - -- Old and new contracts both named; transition mechanism proven, not declared. -- Every known consumer has a working or evidenced path through the change. -- Goldens/snapshots reviewed as intentional contract changes. diff --git a/skills/change-with-proof/profiles/replacement-closure.md b/skills/change-with-proof/profiles/replacement-closure.md deleted file mode 100644 index 0ffb87a..0000000 --- a/skills/change-with-proof/profiles/replacement-closure.md +++ /dev/null @@ -1,52 +0,0 @@ -# Replacement Closure - -## Trigger - -Deletion, dead-code removal, duplicate or type consolidation, legacy or fallback removal, -refactor-driven retirement, or dependency removal. - -## Additional obligations - -- Candidates, not authority: dead-code tools and grep produce candidates. Before deleting, - check the dynamic reference points grep cannot see — string-keyed registries and manifests, - plugin loaders, route conventions, dependency-injection containers, reflection, codegen, - decorator registration, framework auto-discovery, string-referenced tests, and external - consumers. -- Replacement closure: when replacing or deleting a path, account for inbound imports and - dynamic references; public exports and entrypoints; routes, navigation, manifests, plugin - registries, codegen configs, DI, framework conventions; tests, fixtures, snapshots, mocks, - and generated artifacts; docs, comments, product strings, examples, scripts, and config; - types, schemas, validators, API clients, and serialized shapes. -- No parallel implementations after a replacement unless a named contract requires both — and - proof covers both. -- Type and schema consolidation: one canonical owner per domain concept; validate unknown data - at the boundary, then carry precise internal types; keep runtime schema and static type - derived from or checked against one source; use discriminated unions for variants; make - nullability meaningful; no casts without prior narrowing or validation. -- Abstraction: prefer direct code over one-use factories, managers, registries, or - configuration layers. Introduce abstraction only for a named invariant, a dependency - boundary, or multiple real callers. -- Comments: keep those documenting non-obvious invariants, algorithms, security assumptions, - external constraints, or compatibility contracts. Remove agent-activity notes, migration - history, obvious-code narration, and ownerless TODOs. - -## Characteristic failure modes - -- Deleting a handler referenced only through a registry, manifest, or naming convention — - grep finds nothing, production breaks. -- Retired path still reachable from a route, manifest, or generated artifact. -- Consolidation that leaves both copies alive. -- Snapshot or fixture still asserting behaviour that was deliberately removed. -- A comment documenting a security or compatibility invariant deleted as noise. - -## Minimum evidence - -- Reachability/dead-code scan **plus** an explicit dynamic-registration check **plus** - build/typecheck, and a route/manifest audit where relevant. -- Run the real entrypoint when that is cheap. - -## Exit criteria - -- No static or dynamic reference to the retired path remains reachable. -- The closure checklist is accounted for — "grep found nothing" is never the sole - justification. diff --git a/skills/change-with-proof/profiles/stateful-integrations.md b/skills/change-with-proof/profiles/stateful-integrations.md deleted file mode 100644 index 5219744..0000000 --- a/skills/change-with-proof/profiles/stateful-integrations.md +++ /dev/null @@ -1,52 +0,0 @@ -# Stateful Integrations - -## Trigger - -Persistence or transactions, queues and background jobs, caches, retries, webhooks, payments, -external APIs, or any multi-step state that crosses a process or service boundary. - -## Additional obligations - -- External ports: use contract fixtures or boundary fakes at the external port and keep the - internal production path under real test. Verify signature/auth validation, timeouts, - retries, duplicate delivery, and malformed payloads where relevant. -- Idempotency: any mutation reachable from a retry or redelivery (webhooks, queue consumers, - payment confirmations) is proven safe to receive twice — deliver twice, assert one effect. -- Partial failure: multi-step effects (DB write + external call + enqueue) get a stated - compensating action, reconciliation path, or a **named** acceptable inconsistency — never an - unstated one. -- Delivery semantics: name at-least-once vs at-most-once, ordering assumptions, and - dead-letter handling for queue/job consumers. -- Caches: name what invalidates each cache on write, and prove stale reads cannot outlive the - acceptable window. -- Degradation is a decision: a fallback that alters money, persisted data, or user-visible - correctness (default rates or prices, placeholder data, skipped writes) is a contract - change — surface it for approval instead of shipping it as a default, and make degraded - state observable (flag, log, metric), never indistinguishable from healthy output. -- Persistence: prove against representative fixtures including existing persisted shapes; - round-trip where shape changes. Schema/data migration rules live in - `contract-evolution.md`. - -## Characteristic failure modes - -- Double charge or double send on webhook/queue redelivery. -- Row written with no corresponding external effect (or the reverse) and no reconciliation. -- Stale cache served after the write that should have invalidated it. -- Hardcoded business fallback (rates, prices, limits) silently shipped under deadline - pressure. -- Retry wrapped around a non-idempotent call. -- Empty-state success treated as persistence proof. - -## Minimum evidence - -- Duplicate-delivery test (two deliveries → one effect) whenever retries or webhooks are - touched. -- Partial-failure or compensation proof for multi-step effects. -- Cache-invalidation proof when caching is touched. -- Boundary-contract fixtures including malformed payloads. - -## Exit criteria - -- State provably consistent — or a named, accepted inconsistency — across retry, duplicate - delivery, and partial failure. -- Every degraded mode is observable and was approved, not defaulted. diff --git a/skills/change-with-proof/profiles/trust-boundaries.md b/skills/change-with-proof/profiles/trust-boundaries.md deleted file mode 100644 index 84ba195..0000000 --- a/skills/change-with-proof/profiles/trust-boundaries.md +++ /dev/null @@ -1,60 +0,0 @@ -# Trust Boundaries - -## Trigger - -Identity, authentication, permissions, tenancy, secrets, untrusted input, uploads or -downloads, or routing/navigation changes. - -## Additional obligations - -- State the boundary: which inputs are controlled by a user or external system, and which - contract is being preserved or changed. Expected failures are handled explicitly and - observably; unknown or unverified degraded behaviour fails loudly rather than producing - placeholder success. -- Auth and permissions: build a role/action/resource matrix and test allowed **and** denied - cases. Verify server-side enforcement, not only UI affordances. Fail closed when identity, - permission, or resource ownership is unclear. -- Tenant isolation: scope multi-tenant reads and writes at the query/storage layer, not only - via app-level filters. The matrix includes "tenant A cannot read or write tenant B's - resource by ID." -- Check-then-act atomicity: a permission or availability check must be atomic with the action - it gates (same transaction or conditional write), not a separate step a race can slip - through. -- CSRF: state-changing endpoints reachable from a browser session need token, SameSite, or - origin protection — unless proven bearer-token-only. -- SSRF: server-side fetches of user-supplied URLs validate or allowlist the destination and - block internal, link-local, and cloud-metadata addresses. -- Rate limiting: login, signup, OTP/reset, expensive, or notification-sending endpoints get - throttling — or a named out-of-scope gap. -- Uploads and downloads: validate size, type, name/path, content assumptions, storage key, - and access control. Include malformed, oversized, wrong-type, and traversal-style cases. - Return safe, actionable errors without leaking internals. -- Routing: inventory the affected entrypoints — direct URL, deep link, redirect, API route, - not-found, unauthorized, in-app navigation — and prove the risky ones at the request or - browser level, not only with a component test. -- Secrets: never in logs, exports, artifacts, tests, or screenshots; prefer safe digests/IDs - in error context. -- Untrusted input: normalize and validate at ingress; test valid, invalid, malformed, - boundary, and malicious-looking inputs; carry precise internal types after validation. - -## Characteristic failure modes - -- Object-level access to another tenant's resource by guessable ID. -- Check-then-act race on a permission- or availability-gated mutation. -- Open redirect or SSRF through a user-supplied URL. -- State-changing browser endpoint without CSRF protection; OTP endpoint without throttling. -- Allowed-only test matrix — denial and cross-tenant cases never exercised. -- Secrets or internal identifiers leaking through logs, errors, or exports. - -## Minimum evidence - -- Allowed/denied matrix including cross-tenant denial where tenancy exists. -- Ingress validation tests including malicious-looking inputs. -- Explicit applicability statements for CSRF, SSRF, and rate limiting — "not applicable - because …" counts; silence does not. - -## Exit criteria - -- Fail-closed behaviour demonstrated at the changed boundary. -- Proof matrix covers positive, negative, and cross-tenant paths for the changed contract. -- No secret or internal leakage in any produced output; every applicability statement made. diff --git a/skills/change-with-proof/profiles/user-visible-surfaces.md b/skills/change-with-proof/profiles/user-visible-surfaces.md deleted file mode 100644 index ef12e1e..0000000 --- a/skills/change-with-proof/profiles/user-visible-surfaces.md +++ /dev/null @@ -1,50 +0,0 @@ -# User-Visible Surfaces - -## Trigger - -Rendering, interaction, product copy, emails, notifications, exports, PDFs, reports, -user-facing logs, end-user docs, or other generated user-facing content. - -## Additional obligations - -- Inventory the surface before changing it: surface (route/component/template/export/email/ - log/doc), audience (end user, admin/operator, developer, demo), the user's task, the state - the surface must communicate, the next action or recovery path, and the proof path. -- Audience/layer rule: the wrong audience must never see the wrong layer. Check for - implementation mechanics presented as product value; agent/process artifacts and planning - notes; debug traces, hidden IDs, or internal state names; fallback rationale shown as normal - UX; secrets; demo/mock wording on committed surfaces. Move necessary internals to code, - tests, operator logs, or admin/debug surfaces. -- Required states: a surface that renders async data defines **populated**, **loading**, - **empty**, and **error** states. Empty is a designed state — never a blank page or a raw - `[]`/`null`. Loading never silently presents stale content as fresh. Error states - distinguish at least network failure, permission denied, and partial data, each with a - recovery path in product terms. -- Error copy says what happened in product terms, what it affects, and what the user can do - next — no stack traces, internal IDs, or implementation branches. -- Locale and formatting: when the surface is localized or shows dates, numbers, or currency — - correct locale formatting, pluralization, and a defined missing-translation behaviour (an - end user never sees the raw key). -- Prove with rendered output: browser path for route/flow changes; rendered component or - template snapshot for isolated stable units; golden diff for exports, emails, PDFs, and - reports; screenshot only when it proves the surface and audience are correct. - -## Characteristic failure modes - -- Raw `null`, `[]`, or `NaN` rendered to an end user. -- Blank flash or spinner-forever because loading/empty states were never designed. -- One generic error message covering network failure and permission denial alike. -- Missing-translation key or wrong-locale currency/date shown verbatim. -- Stack trace, internal ID, or process narration on an end-user surface. -- Helper-level test passed off as proof for a rendered surface. - -## Minimum evidence - -- Rendered proof of the populated state plus every state the change touches (loading, empty, - error when async data is involved). -- Golden diff for generated artifacts; locale-specific render when localized. - -## Exit criteria - -- Every state the change touches is designed and proven with rendered output. -- No cross-audience leakage on any changed surface; copy is in product language.