Skip to content

Grade a skill's cases without the skill, and report which ones still need it - #35

Open
johnl-amd wants to merge 17 commits into
mainfrom
skill-staleness
Open

johnl-amd wants to merge 17 commits into
mainfrom
skill-staleness

Conversation

@johnl-amd

@johnl-amd johnl-amd commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #6.

skillscope behavioral tells you a skill's cases pass. It cannot tell you they
pass because of the skill.

skillscope staleness runs a skill's existing behavioral cases twice, once with
the skill installed and once with nothing installed, and puts the two runs side
by side. It reads the same evals.json, so there is no new file to write and no
new assertion to learn.

What it reports

What each half spent. An agent holding the skill goes straight to the work;
an agent without it explores. Run on rocm-doctor:

Half Wall time Model calls Tokens Cost
with the skill 1m40s 12 12,576 $0.72
nothing installed 9m58s 85 35,995 $3.13

Six times the wall clock and four times the money, to fail both cases.

A verdict per case.

Verdict Means
earns_it passed with the skill, failed without it
harmful failed with the skill, passed without it
no_signal came out the same both ways
ungraded one side or both could not be graded

The pass counts and an advice column sit beside each verdict, so a no_signal
where both sides passed is told apart from one where both sides failed.

What the skill stopped. A guard rail the baseline broke and the skill held
is credited to the skill. On rocm-doctor the unaided agent attempted a sudo
write under /opt/rocm/share on a case the skill kept read-only.

Running it

skillscope staleness --skill <a small skill> --max-cases 2

Two agent passes, and the baseline is the slow one, so --timeout defaults to
twice the usual and --max-cases and --concurrency are there to keep a run
affordable. Needs ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN on the default
engine: each half gets its own throwaway CLAUDE_CONFIG_DIR, which only works
when auth comes from the environment. Claude Code only for now.

Reporting only. A verdict never fails a run, though an expired deadline or a
refused run still exits non-zero.

Effect on the existing commands

structural, routing, behavioral and select behave the same, and nothing
new runs in CI: staleness is absent from selection.plan() and from every
workflow.

One change to this repo's own CI, which does not reach a consumer's.
.github/workflows/selftest.yml adds F841 beside F401 on the ruff step,
and the job is renamed to match. F841 finds a local nothing reads, which is
the same signal as an unused import one scope down: a review found a list
still being built on every run after the report section that consumed it was
deleted, and neither F401 nor a green suite could see it. The tree is clean
under both rules.

One behaviour change beyond this command. engine/judge.py returned a failed
check when the sandbox would not list or the grader answered in prose, which is
the run failing rather than the agent. It now raises, checks carry a graded
flag, and the case comes out ungraded. behavioral gets that fix too.

behavioral's JSON gains activated, contaminated and attempt per case and
graded per check, and inspect logs gain the skill, the side and the skill's
path. All additive.

--from-logs, --run-id and --unlabelled-logs rebuilt a comparison from logs
on disk. They are scoped out of this round and will come back on their own.

@johnl-amd
johnl-amd requested a review from rominf October 29, 2026 14:08
@volen-silo

Copy link
Copy Markdown
Collaborator

Thanks for this. The with/without-skill comparison answers a question that our behavioral pass rates can't. Contamination detection and voiding verdicts asymmetrically are well thought out.

A heads-up, not a blocker. #33 (the skillscope/harness/ contract) has merged, and #34 (Copilot adapter plus --harness) is open. Together they touch a few places this PR also touches, so a rebase is needed regardless. These are the spots that would need to go through the harness:

  • install flag: claude_code_no_sandbox is now a thin wrapper over harness_solver, so the flag needs to be forwarded there.
  • Credentials and isolation in cli.py: the staleness path uses routing.can_isolate_config(), routing.ENV_CREDENTIALS, and a throwaway CLAUDE_CONFIG_DIR. The harness exposes the equivalents as can_isolate_room, credentials, and host_env/env, so the same code works for any harness.
  • args.engine == "claude-code-no-sandbox": after Add GitHub Copilot CLI as a harness, with routing that needs no model credential #34, engine names are resolved with resolve_leg, so this comparison would need to use the resolved leg.
  • .claude/skills/<name> in engine/behavioral.py: the contamination check could use the harness's skills_dir instead of the literal path.
  • Error text: the messages that mention Anthropic credentials would be wrong for another harness.

None of this changes your design. It's only about where the Claude-specific knowledge lives. I'm happy to help with the rebase or review it once it's out of draft. If you'd rather land this first and have #34 adapt to it, tell me. I just want to avoid us both rewriting _prepare_graded_run.

@johnl-amd
johnl-amd marked this pull request as ready for review November 2, 2026 13:18
@johnl-amd

Copy link
Copy Markdown
Collaborator Author

Rebased on main. The only conflict was no_sandbox.py, and it was the install flag, so that one is handled: install is a parameter of harness_solver now and claude_code_no_sandbox passes it through. The baseline half stages nothing whichever harness runs it.

I also did the contamination one. _skill_marks was spelling .claude/skills/<name> in by hand, and it asks harness.skills_dir now. There is a note on it saying it asks the default harness, and that the staged mark needs to come off the log the way SKILL_DIR_KEY does once a run can name a harness, because a reader on a different harness than the run is not evidence about the run.

The other three I cannot do from here. can_isolate_room, credentials and resolve_leg are on your branch, not on main, so there is nothing to write against yet.

On who adapts to whom: I would rather #34 land first and then rebase this on it. The credential check and the engine comparison in _prepare_graded_run are both mine to convert, and converting them against the real API in one pass beats guessing at it now and you fixing it later. Nothing in #34 needs to know this exists, so it should not be blocked on this. Happy to go the other way if that suits your timing better, just say so.

@nowycondro nowycondro left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 5fa09b0e

Review — needs work

Full review of the whole change (base 39eff043…head 5fa09b0e). Implements the staleness check requested in #6 ("How do we check whether a skill is stale"); #6 is still open, which is expected since it closes on merge rather than on PR open.

Blocking

  • A tool-call shape _inspect_room cannot walk is silently reported as a clean room instead of UNKNOWN_ROOM — skillscope/engine/behavioral.py:359-375.
    The function's own docstring states the intended contract: "a shape this cannot walk is an unknown room, not a failed case." But the except Exception branch at line 373-374 returns the exact same expression as a walk that completed cleanly and found nothing (why or (UNKNOWN_ROOM if unknown else None)), and unknown is set once before the walk starts (marks is None, line 358) — it is never updated when the walk is cut short. _activation_event (skillscope/engine/routing.py:240, return _activation_event_from(call.function, call.arguments or {})) accesses call.function with no getattr guard, unlike every other accessor in this same call chain (_read_the_skill, _tool_calls), so a call object of an unexpected shape (plausible from --from-logs reading an older/foreign log, which this exact code path exists to tolerate) raises partway through the loop. Any contamination evidence on a later tool call is never inspected, and the room is reported as un-contaminated — silently restoring a no_signal/harmful verdict the module is otherwise careful to void whenever the baseline can't be vouched for.
    Confidence 70 · logic · Fix: in the except branch, treat an interrupted walk as UNKNOWN_ROOM unconditionally (e.g. return fired, why or UNKNOWN_ROOM), rather than reusing the "walk completed, nothing found" fallback.

  • No test exercises the harmful verdict under baseline contamination — tests/test_skillscope.py:6283-6337 (TestADirtyBaselineCannotShowWhatAnAgentDoesUnaided).
    VOIDED_BY_CONTAMINATION (skillscope/staleness.py:165) names both no_signal and harmful as voided when the baseline is dirty, and this class tests the no_signal case and the earns_it-survives case, but never pairs a contaminated baseline with a harmful result. I confirmed by reading classify() that the current code handles it correctly, so this is a coverage gap rather than a live bug — but it means a future regression narrowing the voided set to just ("no_signal",) would pass the whole suite while silently reporting an actively harmful skill as standing evidence from a baseline that wasn't clean.
    Confidence 75 · logic · Fix: add a case pairing a contaminated baseline with a harmful result and assert it comes out ungraded.

  • cmd_staleness's documented non-zero exit on an expired deadline is untested at the command level — tests/test_skillscope.py:7179-7233 (TestTheSecondPassGetsABudgetOfItsOwn), skillscope/cli.py:946-947.
    The command's own help text says a completed run always exits 0 but "an expired deadline or a refused run still exits non-zero." Refusal is tested via assertRaises(SystemExit) and the happy path is tested to return 0, and _fail_if_expired() has a generic unit test elsewhere, but nothing drives cmd_staleness through to its final _fail_if_expired() gate with an expired/interrupted deadline to confirm the command-level exit code the help text promises.
    Confidence 70 · logic · Fix: add a test that mocks the deadline as expired/interrupted and asserts cmd_staleness returns non-zero.

Non-blocking

None

Decisions for the author

  • Contamination on the graded ("with-skill") side is recorded but never consulted by classify()/the voiding logic, which only inspects without.contaminated — non-blocking-improvement. In principle an earns_it verdict could be credited to the wrong skill if an unrelated other: skill also fired on the graded side, but the harness's isolation design (temp config_dir, the sandboxed claude-code guest having no user-level skills) makes this close to unreachable given the engines this PR wires up, and the docstrings scope the contamination check specifically to the baseline's unique claim of being empty. Worth a one-line docstring note if this is intentional, otherwise worth symmetric handling.

Positive signals

  • _mentions's boundary-matching (skillscope/engine/behavioral.py:223-240) and its docstring calling out the exact prior bugs it fixes (/run/user/1000 matching a skill mark, foo-extra matching foo) is a good example of a fix that documents the regression it closes, not just the new behavior.
  • The contamination-voiding asymmetry (earns_it/fails_either_way survive, no_signal/harmful are voided) is implemented exactly as the PR describes and is directly covered by tests (tests/test_skillscope.py:6296-6307).

Deployment notes

None — staleness is opt-in, confirmed absent from selection.plan() and .github/workflows/reusable.yml/selftest.yml/skill-evals.yml, so no downstream repo starts running it as a side effect of this merge.

What this covered

Every changed file read in full against the PR head (skillscope/staleness.py, skillscope/engine/behavioral.py, skillscope/behavior.py, skillscope/cli.py, skillscope/engine/no_sandbox.py, skillscope/engine/verify.py, skillscope/datasets.py, tests/test_skillscope.py in full — 7238 lines, README.md, action.yml, docs/usage.md, docs/authoring-evals.md, pyproject.toml), split across four independent fan-out workers grouped by coupling (staleness engine core; CLI+engine wiring; test suite; docs+config), plus my own consolidation, a light history pass on the flagged region, and a direct source check confirming each worker's candidate findings against skillscope/engine/routing.py and the test file. All 8 CI checks on this commit are green (selftest workflow, incl. unit tests on Linux/Windows × Python 3.10/3.12, sandbox machinery, action-against-a-throwaway-repo).

Did not run: agent-instruction-adherence pass (no CLAUDE.md/AGENTS.md in this repository); prior-changes pass (no related earlier PRs on these specific files were checked in depth, since staleness.py is new and this is its first review round).
Not reconciled against any review discussion — none exists on this PR as of this head commit.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@johnl-amd

johnl-amd commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review — needs work

Full review of the whole change (39eff043…5fa09b0e, PR #35: 13 files, +2811/-65). Closes #6 ("Add skill staleness test: how do we check whether a skill is stale"). The PR description's claims — the six-verdict table, the new eval-log metadata keys, the asymmetric contamination-voiding rule, and "not wired into CI by design" — all check out against the code; no mismatch found between what it says and what it does.

Blocking

  • A guards_only case with a contaminated baseline gets a false "it rests on the baseline failing" claim in the report — skillscope/staleness.py:141 (the guards_only branch in classify() returns before any contamination check runs), skillscope/staleness.py:165 (VOIDED_BY_CONTAMINATION's own reasoning only covers no_signal/harmful/earns_it/fails_either_way — guards_only isn't considered), skillscope/staleness.py:614-641 (render_markdown's dirty/stood split, built from summary["cases"] filtered only by baseline_contaminated, not by verdict).
    Comparison.baseline_contaminated is set unconditionally from the baseline outcome regardless of verdict (staleness.py:205,220), so a guards_only case (no positive claims at all — only guard-rail checks) can still have a dirty baseline. render_markdown then puts it in stood and prints: "because they rest on the baseline failing, and it failed while holding more than it should have" — but a guards_only case has no positive claim that could fail or pass; the sentence's premise is false for it. No test combines a guards_only verdict with a contaminated baseline (tests/test_skillscope.py:6247-6255 only exercises guards_only with a clean baseline).
    Confidence 85 · logic · Fix: give guards_only its own neutral phrasing in the stood text (e.g. "asserts nothing positive, so a dirty baseline does not change its verdict"), or exclude guards_only cases from stood entirely.

  • engine/verify.py's install=False path — the sandboxed engine's half of the baseline run — is never exercised by a test that actually calls it — skillscope/engine/verify.py:215-293 (build_task), :296-362 (run).
    Every call to engine_verify.build_task/engine_verify.run in the test suite uses the default install=True (tests/test_skillscope.py:5542,5552,5621 etc.); the one place that documents the gap (tests/test_skillscope.py:6635-6641) checks only inspect.getsource(engine_verify.build_task) for a substring, which doesn't execute the install=False branch. A regression dropping either if install else [] / if install else {} guard, or flipping not install, would make the --engine claude-code baseline pass silently still stage and install the skill — defeating the entire measurement this PR exists to make — with nothing in the suite to catch it.
    Confidence 85 · architectural · Fix: add a test that calls build_task(..., install=False) directly and asserts task.metadata[behavioral.BASELINE_KEY] is True and that the solver chain received an empty skills mapping/list, mirroring the existing SKILL_DIR_KEY test's pattern but as a real call rather than a source-text check.

  • cli._run_behavioral's per-engine dispatch — the code that actually turns install into the right call for each engine — is never exercised by an unmocked test — skillscope/cli.py:700-759.
    The one test that invokes cmd_staleness for real, _staleness_harness (tests/test_skillscope.py:6075-6113), patches cli._run_behavioral out entirely (mock.patch.object(cli, "_run_behavioral", record)), so it verifies only what cmd_staleness asks _run_behavioral to do, never that _run_behavioral turns install=False into baseline=not install for no_sandbox.claude_code_no_sandbox or into install=install for verify.run. The remaining assertions about this function (tests/test_skillscope.py:2387) are source-text substring checks. A one-token inversion of baseline=not install (cli.py:749) would silently swap which half of every claude-code-no-sandbox staleness run is stamped as the baseline, inverting every downstream verdict, undetected.
    Confidence 85 · architectural · Fix: a focused test calling the real _run_behavioral with stub verify/no_sandbox modules, asserting the actual baseline/install values each engine branch receives, for both args.engine values.

Non-blocking

None

Decisions for the author

  • docs/usage.md's per-command flag summary doesn't mention staleness's extra flags — non-blocking-improvement. docs/usage.md:55-60 ("Beyond those, the graded commands take --skill and --only... Routing adds --jobs, --case-timeout,...") lists routing's extra flags but has no analogous clause for staleness's --from-logs/--unlabelled-logs, even though both are fully documented in the dedicated staleness section further down. A one-clause addition there would make the summary complete; not wrong as it stands, just incomplete as a summary.

Positive signals

  • classify()'s six verdicts are produced by mutually exclusive, exhaustive branches, and the asymmetric contamination-voiding rule (VOIDED_BY_CONTAMINATION) is non-obvious but explicitly justified in a comment and tested both ways it can go.
  • The contamination detector's boundary-aware path matching (_mentions/_skill_marks) is unusually rigorous for this kind of check, and several of its rules are explained by name against a real false positive/negative they were written to fix.
  • _run_behavioral's extraction as the single seam both cmd_behavioral and cmd_staleness now share is a faithful, behavior-preserving refactor.
  • Across behavioral.py/verify.py/no_sandbox.py, the sandbox spec, model, effort, limits and hook wiring are kept completely independent of the install/baseline flag — confirmed the two passes of a staleness run differ by only that one flag at the engine-construction level, which is the property the whole feature depends on.

Deployment notes

Opt-in only: absent from selection.plan() and reusable.yml, so no existing workflow's behavior changes by merging this — it only runs where a job names command: staleness or it's invoked directly. The default engine (claude-code-no-sandbox) refuses to start without ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN in the environment (needed to keep the baseline's config dir isolated). Budget roughly twice behavioral's wall-clock and cost per invocation — --timeout defaults to 2x for that reason.

What this covered

Every file in the diff, read completely, split across three independent workers grouped by coupling: (1) skillscope/staleness.py + behavior.py + the new test classes directly exercising the comparison/verdict/contamination logic; (2) skillscope/engine/behavioral.py, engine/verify.py, engine/no_sandbox.py, datasets.py + the new engine-integration test classes; (3) skillscope/cli.py, action.yml, docs/usage.md, docs/authoring-evals.md, README.md, pyproject.toml + the remaining new test class. Each worker also read the other groups' core files as calling/contract context. 39eff043463b774e86e78af8b90637eca66d3e08…5fa09b0eae839bc4972a7d98b5de4d5cbf610c0e. All findings above were independently re-verified against the source (file:line traced directly) rather than taken on a worker's word. CI: all 8 checks green on the head commit (unit tests on 4 OS/Python combinations, unused-imports, sandbox machinery, action-against-a-throwaway-repo) — not re-run locally.

Did not run: agent-instruction adherence — no CLAUDE.md/AGENTS.md in this repository. Prior-changes — skillscope/staleness.py is a new file with no earlier review to carry forward; history pass was limited to scanning recent commit titles touching the modified existing files (behavioral.py, verify.py, no_sandbox.py, behavior.py, datasets.py, cli.py), nothing in them changed this review's conclusions. Not reconciled against this PR's existing discussion — that was deliberately not read, per this method.


🤖 Automated review by an agent.
Model claude-sonnet-5[1m] · Not a human review. Reply here if a finding is wrong.

@johnl-amd johnl-amd added the agent-hub-reviewed agent-hub has reviewed this label Nov 4, 2026

@johnl-amd johnl-amd left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head 9f4527f8

Review — needs work

Full review of the whole change (base 39eff043…head 9f4527f849e2ce463980f6ea13eb23165e94e5f7, +4526/-66 across 13 files, 25 commits), split across four independent fan-out workers grouped by coupling (the staleness engine core; CLI/engine wiring, selection and CI gating config; the test suite; docs and config), each blind to the others and to this PR's existing comment thread. Closes #6 ("Add skill staleness test" — how do we check whether a skill is stale). The description accurately describes what the diff implements, including its own disclosed gaps (the sandboxed claude-code leg wasn't exercised against a live agent), and the change does close what #6 asks for.

Blocking

  • A guard rail broken with the skill installed can go completely unreported when --attempts > 1 — skillscope/staleness.py:724-729 (summarize's broken/guard_checks computation), skillscope/staleness.py:195-224 (_decisive), skillscope/staleness.py:410 (checks_with=list(here.checks) if here else [])
    The module's own stated purpose for this is unambiguous: "Guard rails, counted separately. A broken one is the serious finding here: a skill that made the agent do the forbidden thing" (also stated in docs/usage.md: "a guard broken with the skill installed is the serious finding"). But summarize() only inspects checks_with, which pair() populates from _decisive(these) — "one that passed, else the first" — selected purely on positive claims (expected_behavior/logs_contain/files_exist) and firing, with no awareness of guard-rail (unexpected_behavior) results. Verified directly by constructing two synthetic attempts for one case (attempt 1: positive check passes, guard passes; attempt 2: positive check fails, guard fails — "ran sudo!") and running pair() + summarize(): the result reports verdict: earns_it, guards_broken_with_skill: 0, even though the skill broke a guard rail on one of its attempts with the skill installed. The contamination-voiding logic in the same file correctly scans across the relevant attempts for its own purpose, which makes this an inconsistency within the new code rather than a deliberate simplification.
    Confidence 90 (confirmed by direct execution against the real pair/summarize functions) · logic · Fix: have summarize() check guard rails across every with-skill attempt for a case, not only the decisive one — e.g. by having Comparison/pair() retain each attempt's full check list rather than collapsing to the decisive attempt before the guard scan runs.

Non-blocking

  • docs/usage.md overstates "It reports and never gates" — docs/usage.md:395-399
    The doc's blanket claim, read immediately after a paragraph about --timeout truncation, omits that an expired deadline or a refused run still exits non-zero — which the CLI's own --help text (skillscope/cli.py's staleness_parser description: "a verdict here never fails a run, though an expired deadline or a refused run still exits non-zero") and cmd_staleness's own docstring both state explicitly, and which _fail_if_expired()'s return value confirms in code. A reader of only this doc (not --help) would reasonably conclude a staleness run can never produce a non-zero exit, which is false for a timed-out or refused run.
    Confidence 80 · logic · Fix: add the expired-deadline/refused-run exception to the "It reports and never gates" sentence, matching the CLI's own --help text.

Decisions for the author

None

Positive signals

  • The verdict classification (earns_it/harmful/unattributed/no_signal/fails_either_way/guards_only/ungraded) was directly mutation-tested against five of its most load-bearing branches (the firing guard, the contamination-voiding set, the discriminating-share exclusion of guards_only, the --from-logs newest-run selection, and the attempt-collision key) — each mutation was caught immediately by the specific test the code's own docstrings/history point to, with no false-green behavior found.
  • _mentions's path-boundary matching was exercised directly against 16 cases (boundary at string start/end, sibling-name false positives like foo-extra/foo.bak/myfoo, overlapping occurrences, empty text/mark) — all correct, no false positives or infinite loops.
  • The contamination-voiding asymmetry (no_signal/harmful voided by a dirty baseline, earns_it/fails_either_way not) is implemented exactly as documented and is directly tested.
  • Baseline cleanliness for the default engine is real, not assumed: both halves run against a throwaway CLAUDE_CONFIG_DIR, gated behind routing.can_isolate_config() requiring ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN before a claude-code-no-sandbox run is allowed to start at all.

Deployment notes

  • GitHub reports this PR's mergeable_state as dirty (conflicts with current main) at the time of this review — a rebase will be needed before merging regardless of this review's findings.
  • staleness is confirmed opt-in and not wired into CI: absent from selection.plan() and from .github/workflows/reusable.yml, so no downstream repo starts running it as a side effect of this merge.
  • No CI checks are recorded against this exact head commit (gh/the GitHub API report 0 check runs for 9f4527f8) — unlike the other PRs reviewed in this batch, there is no CI-level confirmation that the 25-commit, 4500+-line change passes anywhere other than the author's own local run described in the PR body.

What this covered

Every changed file read in full against the PR head by one of four independent fan-out workers grouped by coupling: skillscope/staleness.py (the new ~1282-line module, core verdict/contamination/log-pairing logic), skillscope/cli.py/skillscope/selection.py/.github/workflows/skill-evals.yml (CLI wiring, gate reporting, CI annotation), tests/test_skillscope.py (the ~2100-line test diff, verified via direct mutation testing in a disposable scratch clone), and README.md/docs/usage.md/docs/authoring-evals.md/pyproject.toml/tools/benchmark_engines.py (docs and config, cross-checked against the actual CLI help text and flag defaults, including running --help against an extracted copy of the PR tree). Findings were consolidated and re-verified against source directly before inclusion in this report.

Did not run as a separate pass: agent-instruction adherence (no CLAUDE.md/AGENTS.md exists anywhere in this repository). History and prior-changes were not run as fully separate dedicated passes this round beyond what each coupled-file worker incidentally checked (e.g. git log context on the touched modules); no contradiction with history was found where it was checked. Not reconciled against this PR's existing comment thread (3 comments) or any earlier review round on a prior head of this same branch — this pass was kept blind to those by design, per the review method's independence rule.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@johnl-amd

johnl-amd commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

Review — needs work

Full review of the whole change (29 commits, 7a8c00b9...4b99e511, the current PR head at review time). Implements the full ask of issue #6 ("Add skill staleness test" / "how do we check whether a skill is stale") via a new skillscope staleness command, plus a richer verdict taxonomy than the ticket's minimal framing.

Blocking

  • A test named for cwd-independence can never fail, regardless of whether that property holds — tests/test_skillscope.py::TestTheRoomIsJudgedWhereTheRunWasNotWhereTheReaderIs::test_the_verdict_is_the_same_from_any_directory
    The test's own stated purpose (per its docstring and the class docstring) is to guard against a specific historical bug: _skill_marks/_inspect_room (skillscope/engine/behavioral.py) resolving the skill's path fresh from the reader's cwd instead of using the recorded SKILL_DIR_KEY from the log, so a rebuild run from a different directory could silently flip a verdict. But the test computes own once, outside the os.chdir loop, and passes it as skill_dir=own (i.e. recorded) to both calls — so _skill_marks never reaches the cwd-dependent fallback branch (if home is None: ... resolve here) in either call. On top of that, the chosen argument ({"file_path": "run/SKILL.md"}) is not a substring of either mark (the absolute skill directory, or the staged .claude/skills/run path) under any boundary-matching implementation, so _read_the_skill returns None for both calls regardless. The assertion is None == None under the correct implementation and under the regression it claims to catch — I confirmed this by tracing the code directly, and a reverted/weakened implementation of the recorded-path override leaves this test green. No other test in the file exercises the skill_dir=None fallback from two different working directories. This is exactly the "check whose name claims a branch it cannot fail on" case called out in review convention — worse than no test, because it reads as covered.
    Confidence 90 · test-coverage · Fix: change the argument to something that actually matches a mark (e.g. the staged-copy mark, or the real absolute path), and/or drive the skill_dir=None fallback path itself (not just the recorded-override path) from two different cwds so the property that was actually broken historically gets exercised.

Non-blocking

  • _worst_guards's order-independence ("worst showing wins", not "last attempt wins") has no test that can distinguish the two — skillscope/staleness.py:239-255, test at tests/test_skillscope.py:6270 (test_a_guard_broken_on_an_attempt_that_was_not_decisive_still_counts)
    The implementation correctly lets a failure replace a recorded pass but never lets a later pass overwrite a recorded failure (order-independent "worst wins", per the function's own docstring). The one multi-attempt test happens to put the failing attempt last in the list, so a simpler "last attempt wins" implementation would pass the same test. I confirmed this by tracing the attempt order in the test fixture (attempt=1 passes, attempt=2 fails) against the seen is None or (seen["passed"] and not guard["passed"]) condition.
    Confidence 75 · mechanical · Fix: add (or reorder) a case where the failing attempt comes first and a later attempt passes, asserting the guard is still reported broken.

  • Two doc claims in README.md are made inaccurate by this PR's own new content. (1) The PR adds "Staleness" as a fourth bullet under "Skillscope enables four main types of tests," but an untouched sentence further down still says pointing a workflow at the reusable workflow "will triger all graders to run directly from your repo" — not true for staleness, which the PR's own docs/usage.md states explicitly runs in no workflow ("no workflow runs it for you"), confirmed against .github/workflows/reusable.yml, which only wires structural/routing/behavioral. (2) README states staleness "refuses on the same terms" as routing's harness-neutral check, but routing.py:143 refuses per-harness based on that harness's own credentials (Harness.can_isolate_room(), which Copilot already implements), while cmd_staleness (skillscope/cli.py) instead blanket-refuses any non-default --harness outright — a stricter, Claude-Code-specific check, not an equivalent one.
    Confidence 80 · mechanical · Fix: update the "all graders" sentence to exclude staleness, and reword "same terms" to describe the actual (temporarily Claude-Code-only) restriction.

  • A pre-existing docstring in a touched file is made stale by this PR, without being updated. skillscope/engine/no_sandbox.py's harness_solver docstring states, as a general rule, that config_dir is "optional because a behavioral case ... grades what the agent produced, so a stray user-level skill is at worst noise" and that "[a behavioral] leg passes no config_dir." This PR adds a new behavioral-style caller — the staleness baseline leg, via _run_behavioral(..., config_dir=isolated, ...) in cli.py — that does pass config_dir into this exact solver, precisely because for that caller a stray skill is emphatically not "at worst noise" (the PR's own new comment in cli.py says so: "the baseline being empty is the entire measurement"). The paragraph wasn't touched to note the exception.
    Confidence 85 · mechanical · Fix: add a sentence noting the staleness baseline as the one behavioral-style caller that does pass config_dir, and why.

Decisions for the author

  • cmd_staleness's harness restriction is hand-rolled rather than reusing the existing generic abstraction — tradeoff
    routing.py already isolates per-harness via Harness.can_isolate_room(), which Copilot implements (its own credential tuple and its own config_dir/COPILOT_HOME wiring already exist). cmd_staleness instead hardcodes routing.can_isolate_config() (Claude-Code-specific) and separately blanket-refuses any --harness other than the default before that check is ever reached. The code's own error message is honest that this is scoped as "no staleness leg yet" rather than a bug, so this isn't a defect — but it means a future Copilot staleness leg can't just drop the blanket refusal and reuse what's already built; it will need to swap in the generic check too. Worth confirming this was a deliberate "ship Claude-Code-only first" choice rather than an oversight.

  • Both halves of one staleness invocation share a single CLAUDE_CONFIG_DIR temporary directory — tradeoff
    cmd_staleness runs the graded (with-skill) half and then the baseline (without-skill) half inside the same tempfile.TemporaryDirectory, reusing one isolated config dir for both. The contamination detector (_inspect_room) only inspects what the agent did per sample (tool-call arguments); it has no visibility into whether the real claude CLI itself persists any global, non-project-keyed state into that shared directory across the two halves. I could not determine from the code or tests whether Claude Code's CLI ever does this, so I'm not asserting a defect — but since the baseline being a genuinely empty room is this command's entire premise, it seems worth the author explicitly confirming (or giving each half its own fresh config dir, which costs nothing extra to isolate against).

Positive signals

  • _inspect_room's two-branch design deliberately keeps "another skill fired" and "skill under test fired" as separate, non-elif-chained assignments specifically to avoid a documented historical bug where a later contamination hit got misread as the skill having activated — the corresponding tests are well-targeted and mutation-confirmed.
  • The --from-logs rebuild path's "a rebuild is never more permissive than the run it rebuilds" principle (unlabelled logs opt-in only, collision vs. legitimate-repeat distinguished, ambiguous log buckets treated conservatively rather than guessed at) is implemented correctly and thoroughly exercised — every claimed property was independently confirmed via direct mutation testing.
  • The asymmetric contamination-voiding rule (dirty_on_pass/_voided_if_dirty: a dirty baseline voids no_signal/harmful but not earns_it/fails_either_way) is correctly implemented and its rationale holds up under adversarial tracing.

Deployment notes

  • staleness costs roughly twice a behavioral run, and the baseline half can run far longer than the graded half (the PR cites 2m23s vs. 30+ minutes unfinished on a two-case skill); --timeout defaults to twice the shared bound for this reason. Confirmed nothing in .github/workflows/ or skillscope/selection.py invokes staleness — it is opt-in only, matching the PR description.
  • On --engine claude-code-no-sandbox, the baseline half requires ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN in the environment and refuses to run otherwise, since the isolation mechanism only works when auth doesn't live in the redirected config directory.

What this covered

Every changed file read completely across six parallel, independent fan-out workers plus direct verification of the findings above: skillscope/staleness.py (new, 1315 lines, split by verdict-logic vs. log-rebuild/report-rendering halves), skillscope/cli.py (full 1936 lines), skillscope/behavior.py, skillscope/datasets.py, skillscope/engine/behavioral.py, skillscope/engine/verify.py, skillscope/engine/no_sandbox.py, README.md, docs/authoring-evals.md, docs/usage.md, action.yml, pyproject.toml, and all new/changed test classes in tests/test_skillscope.py relevant to this feature (~55 test methods examined test-by-test for "would it fail on revert" and "does it exercise the specific branch its name claims", several confirmed by live mutation on a disposable, non-git-tracked export of the PR's content — never on a checkout of the repository itself). Diff resolved via git fetch origin +pull/35/head:refs/remotes/pr/35 against origin/main (merge-base confirmed equal to the base tip, 29 commits, first-parent count matches total — no stray merge-in from another branch). Ran the full unit-test suite (664 tests) against the actual PR head content and confirmed all pass; independently confirmed via the GitHub API that all 8 CI checks on the current head commit (4b99e511) are green.

Did not run: a "prior changes" pass in the strict sense (review comments on other PRs that touched these files) — no such retrievable record beyond in-repo pattern comparison, which was done. No CLAUDE.md/AGENTS.md exists in this repository, so agent-instruction-file adherence does not apply. One fan-out worker tier failed twice on an unrelated model-deployment error and was successfully re-spawned on the default model before completing.

Not reconciled against any prior review or PR discussion — this is a from-scratch blind review of the current, full diff.


🤖 Automated review by an agent.
Model claude-sonnet-5[1m] · Not a human review. Reply here if a finding is wrong.

@johnl-amd johnl-amd left a comment •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — needs work

Full review of the whole change (base 7a8c00b9…head 4b99e511, +4643/-65 across 13 files, 29 commits).
Implements the full ask of issue #6 ("Add skill staleness test" — how do we check whether a skill is stale), which remains open (expected: it closes on merge, not on PR open). The change goes beyond the ticket's minimal framing with a seven-verdict taxonomy, contamination detection, and a --from-logs offline-rebuild path.

Blocking

  • A tool-call shape _inspect_room cannot walk is silently reported as a clean room — skillscope/engine/behavioral.py (_inspect_room, the except Exception branch), skillscope/engine/routing.py (_activation_event/_activation_event_from).
    The function's own docstring states the contract: "a shape this cannot walk is an unknown room, not a failed case." But unknown is computed once before the walk starts, from whether _skill_marks resolved a path — never updated if the walk itself is cut short. The except Exception branch then returns fired, why or (UNKNOWN_ROOM if unknown else None), the identical expression a clean, completed walk that found nothing would return. _activation_event does call.function/call.arguments with no getattr guard, unlike every other accessor in this chain, so a call object of a shape it doesn't expect (plausible on --from-logs reading an older or foreign log — exactly the input this code exists to tolerate) raises mid-loop, and any contamination evidence on a later tool call is never inspected; the room is reported un-contaminated instead of UNKNOWN_ROOM. No test constructs this case (marks resolved, walk raises partway) — the one existing UNKNOWN_ROOM test only covers the pre-walk "marks never resolved" path. Because _inspect_room now runs on every behavioral/verify sample, not only staleness ones, this silently corrupts the new contaminated/activated fields on any run, and directly defeats staleness's no_signal/harmful voiding logic on exactly the runs most in need of it.
    Confidence 90 · logic · Fix: return fired, why or UNKNOWN_ROOM unconditionally in the except branch.

  • Several new tests are structurally unable to fail under the regression each claims to guard against — tests/test_skillscope.py:

    • TestTheRoomIsJudgedWhereTheRunWasNotWhereTheReaderIs::test_the_verdict_is_the_same_from_any_directory passes the same resolved skill_dir=own to both compared calls, so _skill_marks's cwd-dependent fallback is never reached by either call; the chosen argument ({"file_path": "run/SKILL.md"}) also cannot match either mark under any implementation, so both calls return None regardless. assertEqual(None, None) passes whether or not the cwd-independence property holds.
    • TestADirtyBaselineCannotShowWhatAnAgentDoesUnaided::test_an_earns_it_survives_its_own_contamination sets up a baseline that fails, so dirty_on_pass (which requires a contaminated attempt that passed) is false regardless of whether earns_it is correctly excluded from VOIDED_BY_CONTAMINATION — a tuple that wrongly included "earns_it" would pass this test unchanged.
    • TestGuardRailsAreNotJudgedAsIfTheyDiscriminated::test_a_guard_holding_on_both_sides_does_not_make_a_case_no_signal varies only the positive claim (which alone decides the verdict); guard-state parity is inert in the fixture, so it can't distinguish correct guard-exclusion from a bug that folds guards into the verdict.
    • TestTheCostControlsBindBothHalvesEqually::test_both_halves_get_the_same_set asserts f(x) == f(x) on a pure function without driving a real two-halves cmd_staleness comparison.
    • TestAFlagThatCannotDoAnythingIsRefused::test_the_default_harness_is_not_refused always supplies --from-logs, which the harness-refusal check is unconditionally gated behind; the test silently exercises an unrelated "spent flags" refusal instead of the one it names.
    • TestTheSecondPassGetsABudgetOfItsOwn (whole class) is named for per-pass budgeting, but its own docstring says "two passes under one bound," and every one of its five tests only checks argparse default/help-text wiring — none exercises or could distinguish a shared Deadline from an independent one per pass.
      Each is a distinct authoring mistake, but the recurrence (at least six instances across the PR's ~27 new test classes) across otherwise well-targeted, well-documented tests suggests new tests in this PR were not routinely run against a deliberately-broken implementation before being accepted as coverage.
      Confidence 85-90 (each traced directly against the production code it claims to cover) · test-coverage · Fix: for each, either change the fixture/argument to one that actually reaches the differentiating branch, or rename/re-scope the test to what it actually checks.
  • staleness's own report never discloses whether the run was sandboxed/isolated — skillscope/staleness.py (render_markdown), contrast skillscope/behavior.py's _isolation_note.
    cmd_staleness's summary meta spreads **_sandbox_meta(args) in, identically to cmd_behavioral's — but unlike behavior.py's render_markdown, which calls _isolation_note(meta) to print "Cases ran isolated" or warn "Cases ran unsandboxed", staleness.py's render_markdown never reads or prints that data anywhere. For a command whose entire premise is "the baseline was a genuinely empty, trustworthy room," omitting whether the room was even sandboxed understates exactly the risk the feature exists to surface, and reads as a stronger guarantee than the run actually gives.
    Confidence 80 (confirmed by source cross-check; not exercised by an actual run with missing isolation) · architectural · Fix: call an isolation-note helper from staleness.render_markdown too.

Non-blocking

  • docs/usage.md overstates staleness's harness refusal as a "stronger version of the same reason" routing refuses — contrast cmd_staleness's actual check. Routing's refusal is credential-scaled per harness; staleness's is an unconditional block on any non-default --harness, regardless of what credentials are present. A reader could wrongly conclude a correctly-credentialed non-default harness would be let through.
    Confidence 80 · mechanical · Fix: state plainly that staleness has no leg for any non-default harness yet, rather than framing it as the same check scaled up.
  • cmd_staleness's live-run meta dict is built by hand instead of reusing _leg_meta(args) — skillscope/cli.py, contrast cmd_behavioral's **_leg_meta(args). This omits the harness/harness_version keys from staleness's JSON report entirely and skips _engine_label()'s translation. Inert today, since staleness force-restricts to the default harness — but the schema gap becomes a live inconsistency the moment that restriction is loosened for another harness.
    Confidence 80 · mechanical · Fix: use **_leg_meta(args) in cmd_staleness the same way the other graded commands do.

Decisions for the author

  • Both halves of one staleness invocation share a single temporary CLAUDE_CONFIG_DIR — tradeoff. The contamination detector only inspects what the agent's tool calls did, not whether the underlying CLI process itself persists any non-project-keyed state into that shared directory across the two halves. Worth confirming this is safe, or giving each half its own fresh directory, which costs nothing extra.
  • cmd_staleness hand-rolls a Claude-Code-specific isolation check rather than the generic Harness.can_isolate_room() abstraction engine/routing.py already uses for the identical concern — tradeoff. Functionally equivalent today, since staleness is restricted to the default harness regardless. But a later harness generalization can't just drop the blanket non-default-harness refusal and reuse what already exists; it will need to swap this check out too. Worth confirming this was a deliberate staged rollout.
  • The judge-availability guard in _prepare_graded_run (scoped to command == "behavioral") was not extended to staleness, unlike the sibling _require_hook_support guard in this same diff (which was) — non-blocking-improvement. Currently masked by staleness's own, unrelated non-default-harness refusal, which happens to produce the same practical effect today for a different reason. Worth confirming this omission is intentional.

Positive signals

  • _mentions's boundary-aware path matching explicitly documents and defends against two real historical false-positive bugs (a bare mark matching inside an unrelated longer path; a sibling skill name matching this one's) — a model of a fix that names the regression it closes.
  • The contamination-voiding asymmetry (no_signal/harmful voided by a dirty baseline, earns_it/fails_either_way not) is implemented correctly, and the fired/why conflation it depends on is defended by a genuinely well-targeted test class.
  • _worst_guards scans every answered attempt for a side rather than only the one the headline verdict rests on, so a guard broken on a non-decisive attempt is still surfaced in the report.
  • A guards_only case (no positive claim to credit or fail) is carved out of the report's "stood despite a dirty baseline" narrative into its own, accurately-worded bucket, rather than being told it "rests on the baseline failing" when it asserts nothing that could.
  • The --from-logs rebuild path correctly withholds model/engine/effort from its report meta, since a rebuild reading old logs did not itself run anything on them.

Deployment notes

  • staleness is opt-in only: absent from selection.plan() and from all three workflow files (reusable.yml, selftest.yml, skill-evals.yml) — confirmed directly, no downstream repo starts running it as a side effect of this merge.
  • The default engine (claude-code-no-sandbox) refuses to start without ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN in the environment, since its isolation mechanism only works when auth comes from there rather than the runner's own config.
  • Budget roughly twice a behavioral run's wall-clock and cost per the command's own documented defaults and example.
  • All 8 CI checks are green on the current head commit (4b99e511): unit tests on Linux/Windows × Python 3.10/3.12, sandbox machinery, unused-imports, and the action run against a throwaway repo.

What this covered

Every file in the diff read completely at the PR head, split across five independent fan-out workers grouped by coupling: skillscope/staleness.py + skillscope/behavior.py (core verdict/contamination logic); skillscope/cli.py (CLI wiring); skillscope/engine/behavioral.py + engine/verify.py + engine/no_sandbox.py + datasets.py (engine integration); tests/test_skillscope.py (itself further split by that worker into four independent sub-workers across its ~27 new test classes); README.md + docs/usage.md + docs/authoring-evals.md + action.yml + pyproject.toml (docs/config). All workers were blind to the PR's description, comments and each other's findings. The highest-impact candidates were then independently re-traced directly against the source at 7a8c00b9…4b99e511 before scoring.

Did not run: agent-instruction adherence (no CLAUDE.md/AGENTS.md in this repository); a dedicated "prior changes" pass beyond git log/git blame on the flagged regions, which surfaced nothing that changed this round's conclusions. Not reconciled against this PR's existing review/comment thread — this was a from-scratch blind pass over the code at its current head.


Automated review by an agent.

@siloteemu

Copy link
Copy Markdown

Thanks John, and thanks for the thorough write-up. The concern is real: we shouldn't ship skills that do nothing. I'd like to raise a higher-level question before we invest further, because "does the skill beat no skill, per case" seems to me a narrow slice of what skill value is. This is about intent and scope rather than the implementation.

1. Value isn't a boolean. A skill that is 90% unnecessary is a problem too, and earns_it can't see that. Reporting "earns it" for a skill that is mostly dead weight gives false confidence. How much of the skill matters seems as important as whether it has any effect.

2. Our evals weren't designed to cover a skill's value. Skill behavior is too open-ended for an eval set to cover fully, and many cases exist for routing or guard rails rather than for what the skill adds. Running every case twice spends most of the compute on cases never meant to show value, and no_signal can't tell us which kind we have.

3. What is the bar? The PR defines it as pass/fail, and with --attempts > 1 a baseline passes if any attempt does. But "the AI manages without it" is not obviously the right bar. Alternatives: same result in the same time, at the same cost, or equally reliably. The strongest signal in the PR may be one it doesn't use: the baseline was still running after 30 minutes while the skill run finished in under 3. "The no-skill run takes materially longer or costs more" looks like a cheaper and more meaningful check than re-grading every case.

4. Part of a skill's value is doing things safely. A skill may reach the same outcome as no skill, but make sure the agent doesn't cause harm on the way: no destructive commands, no leaked credentials, no changes to the wrong environment. Outcome comparison can't see that. The PR flags a guard broken with the skill, but doesn't credit a skill for guards the baseline would break, and guards_only cases are explicitly given "nothing to credit". Harm is also rare and hard to elicit: an unaided agent often just doesn't do the dangerous thing, so "baseline didn't break it" is weak evidence the skill is unnecessary, and reproducing the risky situation in an eval sandbox is often impractical. So "baseline also passes" shouldn't be read as "the skill adds nothing" for this class of value.

5. Skills that exist for environment coverage. Some skills (rocm-doctor is the example I have in mind) make a straightforward task work across many environments, GPUs and failure states. A single baseline in a clean sandbox can't exercise that, so the verdict says little about their value. I'd want to know whether we can even set up those environments for evals.

6. Continuous cost for a mostly-true answer. I'd expect this check to almost always come back "yes, the skill matters", and it is least informative for exactly the safety and environment-coverage cases above. Running it continuously seems a poor use of effort compared with other ways of evaluating skill quality and value.

So my concern isn't the implementation. I'd rather first decide what we want to learn about skill value and whether this is the way to learn it. If we keep it, I'd suggest a much thinner version (time/cost to completion, opt-in, run in the skill's own repo) rather than the full verdict taxonomy. Happy to discuss on a call.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right idea, but it does too much at once. One real gap and the scope question need settling before merge.

Direction: grading cases against a no-skill baseline is a useful addition, and keeping it opt-in and out of CI is right. But this is about 1.3k lines of new module plus 2.1k lines of tests for one opt-in command, and the question raised in the thread above (what do we actually want to learn about a skill's value?) is still unanswered. I'd answer that first and land a smaller core: the paired run, the verdicts, --attempts and contamination voiding. --from-logs, --run-id and --unlabelled-logs can follow. Reporting graded-vs-baseline time and cost costs almost nothing and covers that comment's point 3.

Before merge:

  • A judge that could not grade counts as a real failure, and staleness turns that into a verdict. engine/judge.py:235 and :280 return False with no error when the sandbox listing fails or the judge returns no JSON. answered() keeps those attempts, so a judge hiccup on the baseline gives earns_it and one on the graded side gives harmful. In behavioral this only shows up as a red case. Here it produces the headline verdict. Treat those checks as unanswered (so the case comes out ungraded), or add a graded flag to checks, the way #26 does for the legacy path.
  • #34 has merged. On Nov 3 you planned to convert the credential and engine checks to can_isolate_room, credentials and resolve_leg once it landed. What landed instead is a blanket refusal of any non-default harness (cli.py:990). That's fine as a staged step, but say so in the PR and track it in an issue. Right now docs/usage.md:467 and README.md:53 call it "the same terms" as routing, and it isn't.
  • #39 rewrites the room check and the judge path in cli.py, engine/behavioral.py, engine/no_sandbox.py and docs/usage.md, so agree on a merge order with volen-silo. #29 touches the same engine files.

AgentHub check (latest review):

  • Blocking #1 (_inspect_room's except branch): the mechanism is real (behavioral.py:373), but calling it blocking overstates it. The calls come from inspect's message.tool_calls (routing.py:354), which always have .function, so there's no realistic trigger and no evidence it corrupts every run. Take the one-line fix anyway. It was first raised on Nov 4.
  • Blocking #2 (tests that can't fail): overstated as blocking.
    • Two really can't fail, so fix them: the cwd test (test_skillscope.py:7261) and the default-harness test (:6776), because --from-logs skips the refusal at cli.py:991.
    • The earns_it-survives, guard-parity and same-set tests check properties that hold by construction, since earns_it needs a failing baseline and dirty_on_pass only fires on a passing one. They're harmless.
    • The point on TestTheSecondPassGetsABudgetOfItsOwn is wrong. Its tests do check the doubled timeout and the help text. Only the class name is off.
  • Blocking #3 (no isolation note): a parity nit, not blocking. Whether the baseline is empty depends on the isolated config dir and the contamination check, not on the filesystem sandbox. Printing _isolation_note is still cheap.
  • Judge guard not extended to staleness: moot. Staleness refuses non-default harnesses earlier, and #39 deletes that guard.
  • Shared CLAUDE_CONFIG_DIR across halves: I can't confirm or rule out a leak, because workspaces are per sample. Two temp dirs cost nothing, so I'd just use them.
  • Hand-built meta instead of _leg_meta: confirmed, but has no effect today. Optional.

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Automated review · 4b99e51

🟡 Blocking: yes · Non-blocking: yes · Decisions: yes

Review — needs work

Full review of the whole change: PR #35 head 4b99e511 against the main tip 7a8c00b9, three-dot diff, 13 files, 4643 added lines.
Closes #6, but the issue is a single vague sentence, so the change cannot be said to under- or over-implement it. The description matches the code except for the Copilot credential claim in docs/usage.md, which is under Blocking.

Blocking

  • The sandbox (--engine claude-code) baseline has no test. — skillscope/engine/verify.py:252-293, skillscope/engine/verify.py:337-353
    Each of these mutations left the full suite green: skills=[skill_dir] if install else [] changed to [skill_dir]; always staging the skill; BASELINE_KEY: not install; the task name and tags; install=install not forwarded from run(); epochs=attempts; max_samples. The first two make the baseline identical to the graded run, which defeats the whole feature. The PR text admits the leg was never run live, and cli.py steers users to this engine. The suite already fakes inspect_swe via sys.modules (tests/test_skillscope.py:3848), so a test that records the claude_code(skills=...) kwargs is practicable.
    Confidence 90 · logic · Fix: test install=False and install=True, and that run() forwards install, run_key, epochs and max_samples.

  • The pairing keys never reach Task metadata in any test, and the test that claims to guard this cannot fail. — skillscope/engine/behavioral.py:120-128, skillscope/engine/behavioral.py:626-656, skillscope/engine/verify.py:254-266, tests/test_skillscope.py:7245-7253
    Dropping run_key from the metadata, dropping SKILL_KEY, or dropping the tags all stay green. test_the_other_engine_records_it_too asserts "SKILL_DIR_KEY" in inspect.getsource(engine_verify.build_task). Deleting the real line behavioral.SKILL_DIR_KEY: ... still passes, because the comment above it contains the string. It certifies a gap as covered, which is worse than having no test.
    Confidence 95 · mechanical · Fix: build the task with a faked inspect_swe and assert on task.metadata and tags.

  • The rebuild (--from-logs) dies on one unreadable log. — skillscope/staleness.py:600-601
    from_logs loops for info in list_eval_logs(log_dir): log = read_eval_log(info.name) with no try. The docstring promises logs that are not skillscope's own are passed over and that failing on a neighbour's file "would make this unusable". Pointing it at a directory with one truncated .eval raised ValueError: EOCD not found and aborted. cmd_staleness does not catch it. The loop also reads every log in full before checking BASELINE_KEY, while engine/behavioral.py:545 already uses header_only=True for this check.
    Confidence 100 (ran it) · logic · Fix: read the header first, skip on any read error with a one-line note, and fully read only logs that carry BASELINE_KEY.

  • A failed room walk is reported as a clean baseline, contradicting its own docstring. — skillscope/engine/behavioral.py:373-375, docstring at :347-348
    The code is except Exception: return fired, why or (UNKNOWN_ROOM if unknown else None). The docstring says a shape this cannot walk is "an unknown room, not a failed case". With known marks, an exception mid-walk discards the remaining calls and returns why=None, which dirty_on_pass reads as clean. A baseline that was not inspected is credited as clean, which is the dangerous direction for this feature. Dropping the UNKNOWN_ROOM fallback from the except branch left the suite green.
    Confidence 70 (the exception was not triggered with any shape tried) · logic · Fix: return fired, why or UNKNOWN_ROOM on exception, and test it with a call that raises.

  • Git-Bash-style host paths slip past the contamination check on Windows. — skillscope/engine/behavioral.py:156-158, skillscope/engine/behavioral.py:170-214
    With the recorded skill dir C:\Users\me\repo\skills\foo, all of these came out (None, None), meaning clean: cat /c/Users/me/repo/skills/foo/SKILL.md, ls /c/Users/me/repo/skills/foo, find /c/Users/me/repo -name SKILL.md. C:\... and C:/... are caught. On Windows, Claude Code's Bash tool is Git Bash, so /c/... is the natural spelling. A missed contamination turns ungraded into a confident no_signal.
    Confidence 85 (ran it) · logic · Fix: have _normalise also map /<drive>/ to <drive>:/ on Windows.

  • Two report sentences state things the code did not establish. — skillscope/staleness.py:932-951, skillscope/staleness.py:1111, skillscope/staleness.py:1150-1155
    (a) "Some cases ran on one side only" fires for a case whose every graded attempt errored, because per_side is built from graded attempts only. Ran: attempts_with=0, errored_with=1 printed "ran on one side only" for a case that ran on both. (b) voided = [c for c in dirty if c["verdict"] == "ungraded"] prints "a baseline that reached another skill ... cannot show". Ran: a case ungraded because the graded side hit API Error 504, with a contaminated baseline, was blamed on contamination.
    Confidence 100 (ran both) · logic · Fix: (a) count attempts + errored, or say "graded on one side only"; (b) record whether the contamination did the voiding, for example voided_by_contamination on Comparison.

  • staleness is documented as refusing Copilot "on the same terms", but it refuses it outright. — docs/usage.md:463-467 against skillscope/cli.py:975-991
    The docs name COPILOT_GITHUB_TOKEN, GH_TOKEN and GITHUB_TOKEN and say the host leg "refuses a staleness run on the same terms". The code refuses any --harness other than the default ("has no staleness leg yet"), and I ran it and got that error. The code governs. The same limit is missing from the "still assume Claude Code" list in docs/harnesses.md:134-137.
    Confidence 90 · mechanical · Fix: say staleness is Claude Code only and add it to the harnesses.md list.

  • Dead field degraded in the rebuild. — skillscope/staleness.py:503-507
    degraded=routing_core.is_provider_error(error) is never read in staleness.py; consumers exist only in behavior.py and routing.py. The comment says leaving it False would report a provider outage as a skill failure, but any outcome with error set is already excluded by answered(). Mutating it to False left every test green.
    Confidence 85 · mechanical · Fix: remove the field and its comment from the rebuild.

  • Untested behaviour the change introduces. — one finding for several gaps
    Each of these was mutated to if False or the equivalent and stayed green:

    • The outcomes_from_log wiring of SKILL_DIR_KEY (staleness.py:474,520-522, mutated to None; 192 selected tests stayed green). TestTheRoomIsJudgedWhereTheRunWasNotWhereTheReaderIs calls _inspect_room directly and _StubLog never sets the key, although its docstring says it guards a rebuilt verdict.
    • _mentions before-boundary and the ~ expansion branch (behavioral.py:301-307), plus the first-reason-wins if why is None guard (:367).
    • The live-path attempt=int(getattr(sample, "epoch", 1) or 1) (behavioral.py:433). Without it every repeated attempt would collide in _no_duplicates.
    • Render and summary branches: the --max-cases caveat (staleness.py:971), the errored-attempts note (:1074), the unknown-room note and baseline_unknown (:1094, :793), the guards_only plus dirty-baseline note (:1174), the "and N more" suffix (:1024), the \| escape (:904), and _why_dirty's known-beats-unknown rule (:374-375).
    • On the CLI side: the harness refusal (test_the_default_harness_is_not_refused uses --from-logs, which skips the guard), max_cases < 0, --from-logs --concurrency, the config_dir handed to each half, baseline=not install, and the attempts pass-through to verify.run.
      Confidence 100 (mutated) · mechanical · Fix: one small test per branch.

Non-blocking

  • --from-logs silently accepts --model, --engine, --effort and --skip-preflight — skillscope/cli.py:1052-1075
    The spent list refuses --attempts, --max-cases, --concurrency, --no-extended and --harness on the stated rule "refuse rather than ignore". Ran with those four flags: accepted with no complaint. The same flags are ignored at the live path's rebuild. Confidence 90 · mechanical · Fix: add them to spent.
  • Wrong error for a bad --from-logs directory — skillscope/cli.py:1125-1146
    An empty directory plus --skill foo reports "nothing in matches --skill foo". A nonexistent path reports the "no staleness logs, check STALENESS_RUN_KEY" explanation. --attempts 0 --from-logs gets a live-run message. All three were run. Confidence 90 · mechanical · Fix: check is_dir() and "no logs" before narrowing, and validate --attempts after the from_logs branch.
  • The free isolation refusal runs after the paid preflight — skillscope/cli.py:1197, skillscope/cli.py:1214
    _require_hook_support is deliberately placed before the model probe because it costs nothing, but routing.can_isolate_config() runs after _prepare_graded_run. Confidence 85 · logic · Fix: move it ahead of _prepare_graded_run.
  • README and usage.md disagree about CI — README.md
    It says "four main types of tests" and that skillscope-evals "will trigger all graders", while usage.md:551 and usage.md:405-407 say no workflow runs staleness. Confidence 70 · mechanical · Fix: say in the README that staleness is not in the workflow.
  • Doc and error text drift — docs/usage.md:329-335, skillscope/cli.py:1062-1064
    The --from-logs refusal list omits --harness. The doc says plain behavioral logs "rebuild as nothing", but the code exits 1 with "no skillscope staleness logs". The cli.py error calls behavioral "harness-neutral" while docs/harnesses.md:136 says it is refused on other harnesses. The key requirement is stated as unconditional in usage.md:262-278 and README.md:45-54, but cli.py:1214 checks only the host engine and runs after the "nothing gradable" return at :1203. Confidence 85 · mechanical · Fix: reconcile each against the code.
  • Table-cell injection — skillscope/staleness.py:1311-1312
    text[:70].replace('|', ...) does not strip newlines, and case['id'] is unescaped, although _detail strips them. Ran: an expectation of "line1\nline2" split the row. Confidence 100 · mechanical · Fix: one escape helper for |, \n and \r.
  • Docstring and constant drift — skillscope/staleness.py:18, skillscope/engine/behavioral.py:102-105
    "Five things a case can turn out to be" while VERDICTS has seven; the build_task docstring says baseline "only names the task" but it also stamps BASELINE_KEY and a tag. Confidence 85-100 · mechanical · Fix: correct both.
  • Comments narrate the history of code that never existed on main — skillscope/staleness.py:419-423,566-574,917-921,1005-1009,1233-1235,1279-1284; skillscope/cli.py:1191-1194
    "used to say" and a commit hash in a comment go stale after a squash merge. Confidence 85 · mechanical · Fix: state what the code does now and why.
  • Advice wrongly fires for an all-fails_either_way suite — skillscope/staleness.py:1050
    "No case in this suite shows the skill doing anything ... stopped earning its place" also fires when every graded case failed on both sides, where the right advice is to fix the cases. Confidence 65 · logic · Fix: split the branch.
  • Pairing-evidence loss is not reported per skill — skillscope/staleness.py:1014-1020
    read = set(used.values()) is global across skills, so one skill's skipped run is not listed if another skill read it. Confidence 60 · logic · Fix: list per (skill, run).
  • Partial epochs vanish — skillscope/engine/behavioral.py:447-450
    graded is keyed on case id, so a case with --attempts 3 and one cancelled epoch is reported as present. Confidence 55 · logic.
  • Smaller items — _bundled_skills (behavioral.py:309-314) is a magic staging-directory name with no cited source and is global rather than per skill (confidence 70); the "no logs" message says both halves carry STALENESS_RUN_KEY even under --unlabelled-logs (60); created timestamps are compared as strings (60); UNKNOWN_ROOM is imported lazily in three places; the getattr(args, "attempts", 1) fallbacks in _run_behavioral and _gradable_cases are a hidden contract with the other parsers' defaults (60); verdicts are bare strings, where a Literal or Enum would make a typo fail at definition rather than as a KeyError (70).

Decisions for the author

  • The sample-to-outcome mapping is copied between the live path and the rebuild — non-blocking-improvement
    staleness.py:456-532 against engine/behavioral.py:378-440. The comments keep saying "exactly what the live path sets", and a test (test_the_live_path_and_the_rebuild_word_it_the_same_way) pins the copy by grepping source text. Drift already exists: the live path turns a log with no samples into one failed outcome per case carrying log.error.message, while the rebuild uses log.samples or [] and yields nothing. A shared helper in engine/behavioral.py would remove the copy and the grep test.
  • staleness exits 0 when every case errored — tradeoff
    The description states this is deliberate ("No verdict fails a run"), and a nondeterministic agent justifies not failing on a verdict. The other side: a run where both halves hit infrastructure errors still exits 0 with an all-ungraded report, which routing_gate-style callers would treat as failure. Confirm that is wanted, or return non-zero when zero case pairs were graded.
  • --timeout defaults to 1800s, "twice the usual", while the docstring calls twice "the floor, not the figure" — tradeoff
    On the measured case the baseline was still running past 30 minutes, so the default still truncates it. The parser description reads both "more than twice" and "defaults to twice". Truncating a baseline is cheaper than an open-ended run; the choice is yours, but the wording contradicts itself.
  • contaminated is also set on the graded leg's JSON — tradeoff
    The description acknowledges this. The graded agent legitimately reads its own skill, so the field is routinely set in ordinary behavioral output, and the name reads like a problem. Confirm the shape for external consumers.
  • The change bundles a feature, a reader (--from-logs), contamination detection and two engine changes (27 commits, 4643 added lines) — non-blocking-improvement
    It reviews as one coupled unit, but the live and rebuild paths and the engine metadata could have landed separately.

Positive signals

  • Verdicts are asked over the whole side, not off one attempt. The attempt order cannot change the activation verdict.
  • _no_duplicates errors both colliding logs instead of choosing one, and keys on (id, attempt) so --attempts repeats are not mistaken for collisions.
  • Contamination voiding is driven from VOIDED_BY_CONTAMINATION, so the report text follows the rule.
  • UNKNOWN_ROOM refuses to claim a clean baseline where the room cannot be judged.
  • The default is threaded into _add_timeout_argument, so --help states the value in force, and a test checks the rendered text.
  • harness_solver(install=False) is tested by driving the solver and observing whether install_skill is reached, not by reading a signature.
  • _gradable_cases is shared by both commands, and the cap applies once and per skill so both halves grade the same cases.
  • The --harness refusal fails loudly rather than reporting a baseline it never checked.

Deployment notes

  • No packaging, version or action.yml change is needed. The version is 0.1.3 everywhere, the new module sits inside the package, and the pyproject.toml change is a comment only.
  • behavioral JSON gains three per-case keys (activated, contaminated, attempt), and every behavioral and verify task now records skill, side and skill dir in log metadata and passes epochs (1 on an ordinary run). Additive, as the description says.
  • behavioral.py now walks the tool calls of every sample on all behavioral and verify runs, not only staleness runs. It is local work, and per the first Blocking item on the room walk it can return a wrong answer rather than failing.
  • staleness needs ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN on the host engine.

What this covered

Every changed file was read completely by four workers split by coupling: staleness.py with its tests; cli.py and its CLI tests; engine/behavioral.py, verify.py, no_sandbox.py, behavior.py and datasets.py with their tests; and the non-code files checked claim by claim against the code. The range is 7a8c00b9…4b99e511 (head 4b99e511ec38c4a9176c2731671bcc68201ca4af).
Scope was resolved from refs/remotes/pr/35, fetched with the recipe's flags into the shared clone's remote-tracking refs only, with the base read as origin/main after a fetch. The PR was open, and git log --left-right showed only this branch's commits (27 by one author, one merge from main). Workers read a temporary detached worktree outside the clone and made their own worktrees for runs and mutations. The review ran as a fan-out of five sonnet workers, one pass each, not as a single context.

Test verdicts:

  • Revert verdict: with the production change reverted, the staleness, contamination, cost-control and build_task tests fail on import or attribute errors. Exceptions: the CLI-argument tests exercise cli.py rather than staleness.py, and test_the_other_engine_records_it_too passes with its key line deleted (first two Blocking items).
  • Mutation verdict: roughly 50 single-branch mutations in total. The survivors are listed under Blocking. About 20 branches were killed by existing tests, among them verdict preference, the duplicate guard, the contamination rule per verdict, --attempts in spent, and the no_sandbox if install: gate.
  • Verdict basis: both were established by mutation in worker-owned worktrees, with __pycache__ cleared and PYTHONDONTWRITEBYTECODE=1 set, not by literal revert.

Test runs:

  • Windows ran 775 tests with one failure: TestTheHostLegActuallyKillsTheProcess.test_the_work_after_the_decision_never_happens (1.41s against a 1.4s bound), and again alone (1.66s). This is the known flaky bound on that machine, unrelated to the change, and the PR does not touch the code it covers. It was not investigated further.
  • WSL Ubuntu ran the same 775 tests from a copy with all passing, including that test.
  • Each run printed skillscope.__file__ and the path pointed into its own tree.

CI: on the head commit all 8 checks succeeded (Unused imports, unit tests on ubuntu and windows with Python 3.10 and 3.12, sandbox machinery on both, and the action check). The base-branch check-runs were all success, so no failing check needed the base exception. The /status surface was not read, and the rollup showed only CheckRun rows. The CI read did not run tests beyond that. statusCheckRollup rows were taken at face value.

Did not run: the history pass (staleness.py is a new file; the other workers' history reads found nothing relevant), the prior-changes pass (no earlier review was read, by design), and the agent-instruction adherence pass (the repository has no CLAUDE.md or AGENTS.md). A live staleness run was not made, so the API-key refusal, timeout and cost figures are traced rather than run. inspect_swe is not installed in the review environment, so what claude_code(skills=[]) does versus skills=None and whether _bundled_skills exists could not be checked. config.py, deadline.py and usage.py were read only where cli.py calls them. ai-instruction-quality was not activated because no AI instruction file is in the diff. .github/workflows and examples/ were grepped for staleness, not read.

Not reconciled against any discussion here. The description, and issue #6, were read only after the findings were settled.

This watch does not file change requests, so the blocking findings above are published as a comment-type review and gate nothing on their own.

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

johnl-amd added a commit that referenced this pull request Nov 12, 2026
The first real run of this command found something the report would not say.
On `rocm-permission-denied-kfd` the unaided agent attempted a `sudo` write to
/opt/rocm/share/rocdxg/dids.conf; the agent holding the skill kept to
read-only diagnostics. Both sides failed the case. Every verdict was
identical. The only difference between the two runs was the one the report
did not mention.

Guards were counted in one direction only, so a reader saw "none was broken
with the skill installed" and learned that the skill did no harm, never that
it stopped any. That is a whole class of a skill's value the pass/fail
comparison cannot see: both sides can reach the same answer with only one of
them wrecking something on the way.

The data was already there -- both sides' guards are graded on every run --
so this is a second `_worst_guards` call, a count, and a sentence.

A guard the baseline was never held to is not credit. The baseline has to
have been asked the question and failed it; a case it never reached says
nothing about what it would have done.

Raised by siloteemu on #35, who said outcome comparison could not see this
and was right. It turned out not to need the sandbox I claimed it would: the
situation reproduced on its own, on the first skill we pointed this at.
@johnl-amd

Copy link
Copy Markdown
Collaborator Author

Thanks both. Taking the scope question first, because it decided what I did next.

Teemu, points 1, 2 and 6: conceded. earns_it cannot see a skill that is 90% dead weight. Running every case twice does spend most of the compute on cases written for routing or guard rails that were never meant to show value. And no_signal really was two findings sharing a name.

One correction on 6: staleness is already opt-in only. It is absent from selection.plan() and from all three workflows, so nothing runs it unless somebody asks. There is no continuous cost to weigh, and I agree there should not be one.

Point 3 was right, and it is now the headline. The 2m23s-against-30-minutes gap was sitting in the run and the report discarded it. The report now leads with what each half spent.

Point 4 was right too, and I was wrong to call it out of scope. I said outcome comparison could not see prevented harm, and that reproducing a risky situation in an eval sandbox would be impractical. The first real run of the command disproved both. More below.

Point 5 stands as a gap. Environment coverage needs a matrix we cannot stand up for evals today, and I would rather say so than oversell it after getting 4 wrong.

What changed

  • Verdicts collapse from seven to three plus an error state: earns_it, harmful, no_signal, ungraded. The other four were never different answers to the question this command asks; each was one of these with a caveat attached. They survive as counts and advice next to the evidence for them.
  • --from-logs, --run-id and --unlabelled-logs are gone, to return on their own. STALENESS_RUN_KEY went with them, since nothing read it afterwards.
  • Each half's wall time, model calls, tokens and cost are reported, and each case carries its own time for both sides.
  • Roman's judge finding was the real bug. A sandbox that would not list, or a grader that answered in prose, came back as a check that failed. In behavioral that is a red case; here it manufactured earns_it out of a broken baseline and harmful out of a broken graded run. It now raises and the case comes out ungraded. That one would have shipped, thank you.

On the other PRs: #26 turns out to be 90 commits behind main and edits functions that no longer exist, so I rebuilt its mechanism against current code rather than rebasing onto it. Volen, #39 adds a third instance of the same fault in its complete branch, worth catching there. I also do not think #39's room check and this one overlap: yours is a prospective registry listing on the routing leg, this is transcript forensics on the behavioral leg, and neither can do the other's job. The real collision is cmd_behavioral in cli.py, where my _run_behavioral extraction meets your in-place expansion. Happy to land second and re-apply.

The run that changed my mind on point 4

Two cases of rocm-doctor, default engine, opus.

Half Wall time Model calls Tokens Cost
with the skill 1m40s 12 12,576 $0.72
nothing installed 9m58s 85 35,995 $3.13

Both cases earns_it. Six times the wall clock and four times the money, to fail both cases. That is point 3, and no verdict table can express it.

Then the part I had argued was out of reach. On rocm-permission-denied-kfd the unaided agent attempted a sudo write to a file under /opt/rocm/share; the agent holding the skill stayed on read-only diagnostics. The judge recorded both. Both sides failed the case, so every verdict was identical, and the only difference between the two runs was the one the report did not mention.

Guards were counted in one direction only. A reader saw "none was broken with the skill installed" and learned that the skill did no harm, never that it stopped any. The data was already on the comparison, so crediting it cost about thirty lines. The report now names what the skill prevented.

So on your point that harm is rare and hard to elicit: that is true, and it still showed up unprompted on the first skill we pointed this at. I would not claim that generalises from one run.

On size

Roman, you asked for a smaller core. You got a different one rather than a much smaller one, and I would rather say that plainly than let the diff imply otherwise. Removing the offline path took about 900 lines; the judge fix, the spend reporting and the per-case time added back about 550.

I then went looking for padding in the report on the assumption it was bloated, and did not find it. Of its conditional notes, all but one either change how a number should be read or exist because a bug was found there. The single genuine duplicate was eleven lines. If the size is still wrong, the honest lever is scope rather than shaving, and the candidate is the contamination machinery at roughly 600 lines including tests. That buys a cheaper command that cannot vouch for its own baseline. I would rather not, but I would rather you decided than I did.

Happy to take the call either way.

@johnl-amd

Copy link
Copy Markdown
Collaborator Author

Review — needs work

Full review of the whole change (17 files, +4389/-86), base 7a8c00b9ea153c5efc5679f60d7a5e9df0deb056 → head a018fd1ab976073d172ca5817a0b1ab568d67a77.
States it closes #6 by adding a staleness command that compares a skill's behavioral cases with and without the skill installed. At this head the comparison uses a three-verdict taxonomy (earns_it/harmful/no_signal) plus an ungraded error state — a substantial rewrite from the seven-verdict design reviewed in earlier rounds on this PR, which also dropped the --from-logs/--run-id/--unlabelled-logs log-rebuild path entirely. The implementation matches that reduced scope; see Blocking for where the description does not.

Blocking

  • The PR description documents a design this head no longer has. It still prints the old seven-row verdict table (earns_it/harmful/unattributed/no_signal/fails_either_way/guards_only/ungraded) and a whole "A finished run can be read again for free" section promising skillscope staleness --from-logs .skillscope/logs and --run-id. None of that exists at this head: a full-tree search finds zero source references to from_logs/run_id/unlabelled outside two explanatory comments in the test file describing why they were removed, and the verdict set is now four values, not seven (confirmed by direct reading of classify() and by the test suite, 652 tests green, with no test referencing the retired verdicts). A reader who follows the description's own "How to try it" section and runs the --from-logs command gets an argparse error for an unrecognized flag. This is exactly the first-listed description hazard reviewed-code conventions call out: the description reads as an account of the current change but describes an earlier one.
    Confidence 90 · documentation · Fix: rewrite the description against the current four-outcome design and drop the --from-logs/--run-id/--unlabelled-logs section, or say explicitly that this round scoped those out as a follow-up.

Non-blocking

  • A stray extra blank line sits at skillscope/cli.py:1050-1053, right after the can_isolate_config() refusal block — plausibly left behind by the removed --from-logs branch. Harmless. Confidence 70 · mechanical.
  • Inspect's own raw Score.value (CORRECT/INCORRECT) has no way to represent "ungraded," so a case skillscope reports as ungraded still reads as a plain incorrect sample to anything consuming the raw .eval log directly (e.g. inspect view) rather than through skillscope's own report. skillscope's own pipeline never reads this value for pass/fail, only token usage, so this is cosmetic to an alternate viewer, not a defect in this PR's own reporting. Confidence 70 · mechanical.
  • behavior.render_markdown (unchanged by this diff) still labels any case with .error set as (run failed), which reads slightly wrong for the new "ungraded" case ("could not be graded" is the real reason, and it is in the adjoining detail column) — a wording nit, not a logic error, and the function wasn't touched here. Confidence 70 · mechanical.

Decisions for the author

  • usage.leg()'s cost/time accounting swaps a single module-global accumulator rather than snapshotting it — tradeoff. Correct today because cmd_staleness always runs the graded half to completion before starting the baseline half, so the two legs never overlap. If a future change ever ran both halves concurrently (plausible, since the baseline's long wall-clock is this command's stated biggest cost), this would misattribute spend across legs with nothing to flag it. Worth a guard or comment noting legs must not nest/overlap, if concurrency is ever considered.
  • The graded half always runs before the baseline half, under one shared --timeout — tradeoff. An overrunning graded pass can eat the baseline's entire budget, leaving a run with no comparison at all; the alternative (baseline first, or splitting the budget) has its own downside, since a graded-alone result is independently useful as a behavioral report. Already a deliberate, if undocumented-as-a-tradeoff, ordering choice.
  • Contamination is detected on the baseline room only, never the graded room — tradeoff, already justified in the code's own reasoning (contamination is harmless noise in an ordinary behavioral run but fatal to a staleness baseline). One-directional by design, not an oversight, but worth the author confirming that's still the intended scope now that the room check also runs on every ordinary behavioral/verify run per this PR's description.
  • --attempts > 1 changes what a verdict means, not just how confident it is (earns_it at 3 attempts requires the baseline to fail all three) — already documented in the report and the description, but a sharp edge for anyone comparing two staleness runs taken at different --attempts values without reading that caveat.
  • Staleness has no per-case spend cap, unlike routing's --max-budget-usd-style controls — consistent with behavioral's own flag surface (neither has one either), but given staleness is explicitly costly (twice behavioral, with an effectively unbounded baseline short of the overall timeout), a per-case cap could be a reasonable future addition. No clear winner here, flagging for awareness.

Positive signals

  • _gradable_cases is explicitly shared between behavioral and staleness so the two halves of one comparison can never silently grade a different case set from each other — a well-reasoned, tested safeguard.
  • The contamination detector's boundary-matched path marks (_mentions/_skill_marks) fix a real false-positive class (a skill literally named run previously matching /run/user/1000 as a substring) and are backed by dedicated adversarial tests, including the sibling-skill-name-extension case.
  • The judge-outage bug fix (raising Ungradable instead of returning a plain False) is traced correctly end to end into staleness.answered(), and the regression tests rebuild the real pipeline rather than hand-constructing the failure state, so they'd actually catch a reversion.
  • One shared Deadline governs the whole staleness invocation (not one per half), and --timeout's default and its --help text are both derived from the same value, avoiding a help-text-vs-behavior drift bug an earlier round's docstring explicitly calls out.
  • Careful None-vs-zero-seconds handling throughout the timing/report code specifically so a baseline that never finished is never misreported as having run instantaneously.

Deployment notes

Opt-in only: staleness is absent from selection.plan() and from every workflow file, confirmed by direct search, so no existing CI run changes behavior by this merging. The default engine refuses to start without ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN in the environment (needed to keep the baseline's config dir isolated), and the baseline half can run far longer than the graded half, which is why --timeout defaults to twice the usual bound here.

What this covered

The full diff across all 17 files, read completely, split by coupling across three independent fan-out passes: (1) skillscope/cli.py, usage.py, the datasets.py diff, docs/usage.md, action.yml; (2) skillscope/staleness.py in full plus the complete staleness test block (~1850 lines) and the remaining doc files; (3) skillscope/engine/behavioral.py, judge.py, verify.py, scorers.py, no_sandbox.py, behavior.py in full at both base and head. Two of the three passes independently ran the actual test suite against the PR head (652 tests passing). CI on the reviewed commit is fully green (8/8 checks). The description (Step 9) was read only after all findings above were otherwise settled, and is what produced the one blocking finding.

One process note, for transparency: the three fan-out passes above were genuinely independent and blind (none read this PR's discussion; each says so). Consolidating their results hit a snag — an intermediate orchestrating step received a relayed summary of the three passes' conclusions and, unable to independently confirm that relay's authenticity, correctly declined to act on it rather than rubber-stamp unverified findings. This final report was produced by reading the three passes' own original output directly and consolidating it from there, rather than through that intermediate relay, so the findings above rest on the fan-out passes' firsthand reports, not on anything secondhand.

Did not run: agent-instruction adherence (no CLAUDE.md/AGENTS.md in this repository). Not reconciled against this PR's own review discussion beyond what Step 9 requires (the description) — the earlier rounds' findings and the design conversation in the comments were not used to shape or filter anything above.


🤖 Automated review by an agent.
Model claude-sonnet-5[1m] · Not a human review. Reply here if a finding is wrong.

@johnl-amd johnl-amd left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated review · review-code · head e4c71b62

Review — needs work

Full review of the whole change (18 files, +4664/-89, merge-base 7a8c00b9…head). The review was carried out against commit 1fb1024f; while consolidating, the author pushed one further commit, e4c71b62 ("Build the room-check test's paths the way Windows spells them"), which turned out to fix exactly one of this review's own findings (see Blocking #1) and touches nothing else. All other findings were re-confirmed against the actual current head, e4c71b6, which is what this report and its line numbers refer to.
This implements issue #6 ("Add skill staleness test") in full — there is no partial-implementation gap against that ticket.

Blocking

  • The PR description's verdict table no longer matches the shipped code. The description lists seven verdicts (earns_it, harmful, unattributed, no_signal, fails_either_way, guards_only, ungraded), three of which (unattributed, fails_either_way, guards_only) do not exist anywhere in the code. skillscope/staleness.py:87-91 defines exactly four: earns_it, harmful, no_signal, ungraded. The repo's own commit history on this branch shows why: a later commit on this same PR, titled "Collapse seven verdicts into three and an error state," intentionally merged the seven down to four — the description was never updated to match. A reader deciding what to test or review from the description alone would be checking for distinctions the code doesn't make.
    Confidence 85 · architectural · Fix: rewrite the verdict table (and the "unattributed"/"fails_either_way" mentions elsewhere in the description) to match the current four-verdict design, or restore the richer taxonomy if it was dropped unintentionally.

  • _ancestors()'s path-depth threshold produces zero ancestors for the staged skill mark, silently disabling sweep-based contamination detection on the default (and only registered) harness. skillscope/engine/behavioral.py:203-225 (_ancestors), consumed by _swept (184-200) and _skill_marks (237-281) — all new in this PR. The staged mark built in _skill_marks is f"{harnesses.get().skills_dir}/{skill}", i.e. .claude/skills/<name> for the one registered harness — exactly 3 segments after .split("/"). _ancestors's loop (while len(parts) > 3: ...) therefore never executes for this mark and contributes no ancestors at all. Reproduced directly against the production code and the project's own test fixtures: find .claude -name SKILL.md -exec cat {} +, grep -r rocm .claude/skills/, and ls .claude/skills/ all come back as a clean room (None) instead of the documented UNKNOWN_ROOM, even though the surrounding docstrings explicitly name find/grep -r/a bare sweep as the realistic way an unguided agent looks for something. This is the exact mechanism skillscope staleness's baseline-integrity claim rests on: a baseline workspace with leftover/contaminated .claude/skills content (e.g. from incomplete cleanup between cases) would be silently credited as empty, which can turn a would-be ungraded/no_signal/harmful case into a false earns_it/no_signal. The existing regression tests (TestAnAgentThatWentLookingIsNotACleanRoom) only exercise sweeps against the absolute "home" mark, which is many segments deep and never hits this edge, so they didn't catch it.
    Confidence 85 · logic · Fix: size the stop threshold relative to each mark's own depth (e.g., stop one segment short of the mark's own directory rather than a fixed length-3 floor calibrated for absolute POSIX paths), and add a test that sweeps the staged/relative mark specifically.

  • A test documented as token/network-free makes a real, unmocked call into model/API reachability checks. tests/test_skillscope.py:7156-7164, test_the_default_harness_is_not_refused in TestAFlagThatCannotDoAnythingIsRefused. It calls cli.cmd_staleness(parser.parse_args(["staleness", "--harness", "claude-code"])) with no --skip-preflight and nothing mocked. Tracing the call: cli.py:1022 reaches _prepare_graded_run (cli.py:464-539), and since DEFAULT_ENGINE = "claude-code-no-sandbox" (cli.py:127) and args.skip_preflight is false by default, execution unconditionally reaches engine_models.check_reachable(...) (cli.py:524) and then check_api_reachable(...) (cli.py:531, which spawns the real claude CLI) whenever Claude Code credentials are present in the environment the suite runs in — a realistic state on a developer machine or a credentialed CI runner, and the actual state of the sandbox this review ran in. Every sibling test in the same class is refused by SystemExit before ever reaching _prepare_graded_run; only this one proceeds into it. The test file's own stated contract for this class of test is "no agent, no tokens, no network."
    Confidence 80 · mechanical · Fix: mock _prepare_graded_run/engine_models.check_reachable/check_api_reachable as other tests in the file do, or pass --skip-preflight.

  • The new attempt field's only production code path (deriving it from sample.epoch) has no test coverage; the fixture built for exactly that purpose is unused. skillscope/engine/behavioral.py:525 (attempt=int(getattr(sample, "epoch", 1) or 1), new this PR); tests/test_skillscope.py:6439-6448 (_StubSample, whose own docstring states its purpose is to test this). _StubSample is defined once and never instantiated or passed to _outcomes() anywhere in the file; every other "multi-attempt" test constructs BehaviorOutcome objects directly with attempt=i, bypassing _outcomes() entirely, and no test anywhere asserts on .attempt. The line that turns an inspect sample's epoch into the field that lets staleness tell repeat attempts of the same case apart could be deleted, hard-coded to 1, or read the wrong attribute, and the whole suite would stay green.
    Confidence 75 · test-coverage · Fix: instantiate _StubSample with varying epoch through _outcomes() and assert the resulting attempt values, or remove the unused fixture.

Non-blocking

  • staleness's JSON report omits the harness/harness_version meta keys that routing and behavioral reports always carry — skillscope/cli.py:1088-1111 builds the staleness meta dict by hand instead of splicing **_leg_meta(args) (cli.py:190-202) the way cmd_routing (cli.py:636) and cmd_behavioral (cli.py:946) do. Harmless today since staleness is restricted to one harness, so the value can't currently be wrong, but it's a schema gap across three report types this PR's own description treats as siblings, and would become a real correctness gap if staleness ever gains a second harness leg.
    Confidence 70 · mechanical · Fix: splice **_leg_meta(args) into the staleness report meta like the other two commands.

Decisions for the author

  • --attempts N's "any attempt passed" rule has a real, derivable asymptotic bias — tradeoff. As N grows, any case where the baseline has even a small nonzero chance of an accidental pass converges toward no_signal and away from earns_it, for purely statistical reasons unrelated to whether the skill still matters (the same holds for harmful requiring the graded side to never pass). The module's docstring and the rendered report already disclose the mechanism correctly and concretely (the worked "1/3 baseline pass takes the case to no_signal" example), but frame raising --attempts as "a higher bar" / "a stronger claim" — language that reads as increased rigor rather than a one-directional statistical pull toward no_signal. Worth naming the direction explicitly so a user raising --attempts to gain confidence in a positive finding doesn't read the result backwards.
  • The baseline's "empty room" guarantee is log-based heuristic detection, not real isolation, on the default engine — tradeoff. On claude-code-no-sandbox (the default for staleness), the agent runs on the host with the skill's real directory fully present and reachable the whole time; the code's own docstrings already acknowledge this ("nothing stops it naming an absolute path... the sandboxed claude-code leg does close it"). Blind spots beyond what's already documented include indirect access (MCP resource reads, env-var-sourced paths, runtime string concatenation) and, most structurally, a model's own memorized/trained knowledge of a public skill's contents, which leaves no tool-call evidence at all — not even the fallback UNKNOWN_ROOM. This reads as a reasonable, unusually well-documented best-effort design rather than a fundamental flaw, but worth the author explicitly confirming (perhaps in the command's own help text or report output, not just in source comments) that the no-sandbox baseline's "clean room" claim is log-based only, and considering whether the sandboxed claude-code engine should be the recommended choice specifically for staleness, given that command's integrity depends on room emptiness more than any other.

Positive signals

  • The contract between staleness.py's Comparison fields and its verdict/report logic is unusually well specified — nearly every field and helper's docstring states the exact edge case it exists to handle, and in every instance checked, the implementation matched the stated contract.
  • usage.leg()'s design (swapping the global usage accumulator rather than snapshot-and-subtract) correctly sidesteps a real bug class its own docstring names — snapshot-and-subtract would have reported every leg as having spent nothing, a plausible-looking zero rather than a loud error.
  • The CI failure this review independently found (the Windows-only test-path bug in TestAnAgentThatWentLookingIsNotACleanRoom) was fixed within about 20 minutes of this review starting, correctly diagnosed down to the exact root cause (rsplit("/") on a native OS path) and fixed with a portable Path-based rewrite rather than a platform-specific patch.

Deployment notes

  • staleness is opt-in: it is absent from selection.plan() and from this repo's own selftest.yml, and is only exposed as an additional action.yml command input for downstream consumers to invoke explicitly. Nothing in this repo's own CI cost or runtime changes as a result of this merge.
  • behavioral's JSON output gains three additive per-case keys (activated, contaminated, attempt); existing consumers reading by name are unaffected, but a consumer diffing whole JSON blobs will see new keys appear on ordinary behavioral runs, not just staleness ones.

What this covered

Every file in the change read completely: skillscope/staleness.py (new, 1232 lines), skillscope/behavior.py, skillscope/cli.py, skillscope/usage.py, skillscope/engine/behavioral.py, skillscope/engine/verify.py, skillscope/engine/no_sandbox.py, skillscope/engine/judge.py, skillscope/engine/scorers.py, skillscope/datasets.py, and the full diff of tests/test_skillscope.py (~2200 new/changed lines), plus action.yml, .github/workflows/selftest.yml, README.md, docs/usage.md, docs/harnesses.md, docs/authoring-evals.md, and pyproject.toml, against 7a8c00b9…e4c71b62. Four independent code-review passes ran in parallel, grouped by coupling (core module; CLI/usage/docs wiring; engine internals; test suite), followed by two independent design-review passes on specific strategic questions, then consolidation and adversarial re-verification of every surviving finding against the actual source (including direct reproduction for the two highest-confidence findings).

Did not run: a "prior changes" pass against earlier PRs' review comments on these files (not performed, for effort reasons — git history was read instead, which surfaced the relevant in-branch context directly). No repo-root CLAUDE.md/AGENTS.md exists, so agent-instruction adherence had nothing to check. CI's check-run results were read directly from GitHub and taken at face value; all 8 checks are green on the current head.
Not reconciled against any existing discussion, review comments, or threads on this PR — none were read, per this review's independence requirement.


Not a human review. This is an automated code review; please verify findings independently before acting on them.

@johnl-amd

Copy link
Copy Markdown
Collaborator Author

Review — needs work

Full review of the whole change.
Closes #6 ("check whether a skill is stale") by implementing exactly that: skillscope staleness runs a skill's existing behavioral cases twice, with and without the skill, and reports per-case verdicts. The PR description's claims (effect on existing commands, the judge.py Ungradable-exception behavior change, additive JSON fields, scoped-out --from-logs/--run-id/--unlabelled-logs, staleness absent from CI selection) were all independently verified against the code and are accurate.

Blocking

  • A no_signal report can print a false causal story about why contamination didn't void the verdict — skillscope/staleness.py:1101 (the elsewhere branch), depending on VOIDED_BY_CONTAMINATION (:398), dirty_on_pass (:401-411), _voided_if_dirty (:414-417).
    The fixed sentence reads: "the baseline passed on an attempt that was clean and was dirty on one it failed." That's only true for the harmful case and the "both-sides-passed" flavor of no_signal — both of which require the baseline to have passed somewhere. But no_signal also covers the "both sides failed" case, which can land in this same elsewhere bucket: if a baseline attempt is contaminated (e.g. flagged by _swept) but that attempt failed the case (never passed, on any attempt), dirty_on_pass is False (it only looks at attempts that both passed and were contaminated), so the verdict is never voided and the case appears in elsewhere — yet the baseline never passed on any attempt, clean or otherwise. The printed sentence then asserts something that didn't happen. Traced directly through the three functions above to confirm this path is reachable with a single attempt per side (--attempts not required).
    Confidence 85 · logic · Fix: branch the sentence on whether the case actually has a passing baseline attempt (e.g. use stood/elsewhere's own verdict plus a check for any passed attempt) rather than asserting the narrative unconditionally.

  • Dead tuple entry in the contamination-sweep detector — skillscope/engine/behavioral.py:181.
    _SWEEPS = ("*", "?", "-r ", "-R ", "--recursive", "find ", "**", "rglob", "ls ") is only ever tested against _normalise(v) (:356, :385), which lowercases every string before the check. "-R " can therefore never match — only "-r " ever will, since the input is always already lowercase by the time _swept sees it. This is unreachable code masquerading as case-sensitive coverage, in the one detector this whole command's correctness rests on (it already needed a 4-defect patch in this same branch for missed sweep shapes — see commit 1fb1024, "Close the four defects a review pass found").
    Confidence 90 · mechanical · Fix: drop the redundant "-R " entry (harmless today only because "-r " already covers it).

  • cmd_staleness's report meta omits harness/harness_version, contradicting its own adjacent comment — skillscope/cli.py:1088-1111 (the staleness.summarize(comparisons, {...}) call), vs. _leg_meta (:190-201, which supplies engine/harness/harness_version and is spliced into every other graded report: cmd_behavioral at :946, _finish_routing at :636).
    cmd_staleness hand-writes "engine": args.engine (:1092, skipping the _engine_label() normalization _leg_meta applies) and never calls _leg_meta(args), so a staleness report's JSON has no harness/harness_version key at all — unlike every other graded command. This directly contradicts the comment two lines above the "legs" key (:1106-1107): "every other report's meta keeps the shape it had." A downstream reader/tool that expects every graded report to carry harness/harness_version (as routing and behavioral do) gets a schema that silently differs for staleness.
    Confidence 85 · logic · Fix: splice **_leg_meta(args) into the meta dict the same way cmd_behavioral does.

Non-blocking

  • UNKNOWN_ROOM's message hardcodes "the baseline" even though it's used on every behavioral run, not just staleness — skillscope/engine/behavioral.py:231-234, surfaced via _inspect_room (:509 in _outcomes), which runs unconditionally for every behavioral evaluation. Confirmed .contaminated is only ever read by skillscope/staleness.py today, so this wording is currently inert outside staleness (not user-visible in a plain behavioral report) — but the field is stored on every BehaviorOutcome, and the generic-sounding constant name doesn't match its baseline-specific text.
    Confidence 70 · mechanical · Fix: word it without assuming a staleness baseline, e.g. "could not tell whether this run read the skill."

  • A typo'd or unknown --harness value on staleness produces a misleading error — skillscope/cli.py:991-1007. asked != harnesses.DEFAULT is checked and raises "has no staleness leg yet" before harnesses.get() ever validates whether the name is real, so e.g. --harness cluade-code (typo) reads as "a real, known harness just not supported yet" rather than "unrecognized harness." The refusal direction itself is still correct.
    Confidence 70 · mechanical · Fix: validate the harness name first (via harnesses.get()/its known-names list) and only emit the staleness-specific message for a real, non-default harness.

Decisions for the author

  • The contamination/"empty room" detector only inspects tool-call arguments — not tool results, cross-call shell state, or subprocess execution — tradeoff. It catches an agent whose tool-call arguments textually reference the skill's path (directly, via ~, via ..-collapsing, or via a sweep/wildcard over an ancestor directory), with a conservative "unknown room" fallback for shapes it can't interpret. It does not catch: content the agent learned from a tool result rather than a path it typed; a path built across two calls in a persistent shell session (cd in one call, a bare filename in the next); or a script the agent runs that itself opens the skill file, where the outer call's argument is just the script's name. The module's docstrings are candid about being "best effort" for the textual/path dimension specifically, but don't mention these three channels. Given this exact mechanism already needed a 4-defect patch within this same PR branch for missed argument shapes, it's worth deciding explicitly whether these remaining gaps are acceptable residual risk (consistent with the feature's own framing as "a lead to investigate," never a merge gate) or worth hardening/documenting now.

Positive signals

  • _by_case groups a side's attempts by case id into a list rather than the {id: outcome} pattern an earlier version used, specifically because keying on id alone silently dropped all but the last attempt — a real fix to a real class of silent data loss, and the comment names exactly why.
  • _within_deadline() is polled between the two halves of a staleness run (not just at the very end), so a baseline the timeout stopped partway is correctly attributed as a floor rather than misreported as a finished, clean measurement.
  • The fix-round commit on this branch (1fb1024) added a guard test (TestASidePassesOnlyByMeetingEveryClaim) specifically pinning _met's all(...) rather than any(...) — a fragile invariant that every asymmetric test had previously left unpinned, closing exactly the kind of silent-flip risk this review would otherwise have had to flag.

Deployment notes

skillscope staleness is intentionally opt-in: it is absent from selection.plan() and from every CI workflow (confirmed — reusable.yml is untouched by this PR), so adopting it requires a repo to add it manually/on a schedule; nothing changes for existing CI pipelines by merging this. It needs ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN (or --engine claude-code) and is Claude-Code-only — --harness is refused for every other value regardless of credentials present.

What this covered

Every file changed by the PR, read completely at both 7a8c00b9ea153c5679f60d7a5e9df0deb056 (confirmed merge-base) and e4c71b62d541452bbe81b5dac2aa90e9a65a7328 (current head): skillscope/staleness.py (new, standalone pass), skillscope/engine/{behavioral,verify,no_sandbox,judge,scorers}.py (coupled engine-package pass, plus behavior.py/cli.py contract spot-checks), skillscope/cli.py + usage.py + datasets.py + behavior.py (CLI/wiring pass), tests/test_skillscope.py (full test-suite pass, which additionally ran the real suite plus targeted mutation testing against several of the subtlest claims — all passed, flake8 F401,F841 clean), and README.md/docs/usage.md/docs/harnesses.md/docs/authoring-evals.md/action.yml/pyproject.toml/.github/workflows/selftest.yml (docs/config pass, including an empirical ruff check --select F401,F841 run against the actual HEAD tree). One additional focused pass investigated the completeness of the contamination detector as a design question (see Decisions for the author). All CI checks on the current head are green (confirmed via live query, 8/8 success).

Did not run: agent-instruction adherence — no CLAUDE.md/AGENTS.md or equivalent exists in this repo. Prior-changes — no accessible record of review comments on these files outside this PR's own discussion thread, which was deliberately not read (see below). History was covered via direct git log/git show on the modified files and this PR's own commit history, which already surfaced and fixed (within this branch, before this review ran) a prior round's contamination-detector false negatives, an unpinned all/any conjunction, an untested judge-glue path, and a headline-drawn-from-an-ungraded-case bug.

Not reconciled against any discussion on this PR — this review was conducted blind to the PR's comments and did not read them at any point.


🤖 Automated review by an agent.
Model claude-sonnet-5[1m] · Not a human review. Reply here if a finding is wrong.

…need it

`skillscope behavioral` tells you a skill's cases pass. It cannot tell you
they pass *because of* the skill. A case any competent agent satisfies stays
green forever whatever happens to the skill, and the report reads as a
measurement of the skill when it is a measurement of the model.

`skillscope staleness` runs a skill's existing behavioral cases twice, once
with the skill installed and once with nothing installed, and puts the two
runs side by side. It reads the same `evals.json`, so there is no new file to
write and no new assertion to learn. Each case comes back as `earns_it`,
`harmful`, `unattributed`, `no_signal`, `fails_either_way`, `guards_only` or
`ungraded`.

Installing a skill does not make an agent use it, so the graded side also
records whether the agent reached for it: a case that moved without the skill
firing is `unattributed`, not `earns_it`. `--attempts N` runs each case N
times a side, and a side counts as passed if any attempt did, which makes a
higher N a harder question rather than a steadier reading of the same one.

The baseline is only a measurement if the room was empty, so both halves run
against a throwaway `CLAUDE_CONFIG_DIR`, one per half. That works only when
auth comes from the environment, so the host engine refuses to start without
`ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`, as routing does. An agent
reading the skill's source off disk is detected too, and a dirty baseline
voids only what it refutes: `no_signal` and `harmful` claim the baseline
managed something unaided, so contamination kills them, while `earns_it` and
`fails_either_way` rest on the baseline having failed and survive it.

A finished run can be read again for free: both halves record which side they
were and which invocation wrote them, so `--from-logs` rebuilds the
comparison without reaching a model.

No verdict fails a run. Two runs of a nondeterministic agent disagree with
themselves often enough that a verdict is somewhere to look, not a build
failure. An expired deadline or a refused run still exits non-zero.

`staleness` is absent from `selection.plan()` and from every workflow, so
nothing new runs anywhere and no downstream repo starts running it as a
result of this merge. `behavioral`'s JSON gains three additive per-case keys
(`activated`, `contaminated`, `attempt`), and behavioral and verify tasks now
record which skill and side they were about.

Closes #6.
`judge.grade` returned `(False, reason)` when the sandbox would not list and
when the grader answered in prose. Both are statements about the run, and
both arrived as a check that simply failed.

In `behavioral` that is a red case. In `staleness` it is the headline.
`answered()` keeps any attempt without an `error`, so a baseline whose judge
broke counted as a baseline that genuinely failed, which is `earns_it`, and a
graded run whose judge broke counted as `harmful`. The two verdicts the
command exists to produce were the two a grader hiccup manufactured.

The polarity is where it bites hardest: on `must_happen=False` a `False`
asserts the forbidden thing happened, so a grader that rambled was reported
as the agent having done it.

`judge.Ungradable` carries the distinction now. An exception rather than a
changed return type, because the caller has to do something different with it
and a `bool | None` invites being read as falsy by every caller that does not
know yet. Checks gain a `graded` flag, read through `scorers.graded()` so a
dict from a log written before this still reads correctly, and one ungraded
check errors the attempt: a side passes only by meeting every positive claim,
so a claim with no verdict leaves the side undecidable rather than negative.

`staleness.answered()` needs no change. It already drops attempts with an
error, which is why that was the channel to use.
`--from-logs`, `--run-id` and `--unlabelled-logs` rebuilt a comparison from
logs already on disk. The feature is sound and will return, but it arrived
welded to the command it reads for, and reviewers asked for the smaller thing
first: one PR that runs two halves and reports what differed.

Most of the complexity it carried was its own. A log directory is shared with
`behavioral` and `routing`, is never pruned, and holds every run anybody has
done in it, so the rebuild needed a run id to pair on, a rule for picking one
run per skill, a flag for logs written before the id existed, and a way to
report a side it could not answer for. None of that is needed to run two
halves and compare them.

`STALENESS_RUN_KEY` goes with it. Nothing read it once `outcomes_from_log`
was gone, and a key written for a reader that does not exist is the kind of
dead weight this round of review was about. Removing it also narrows the
`build_task`/`run` keyword surface that #39 touches.

`test_the_default_harness_is_not_refused` is fixed rather than deleted. It
passed `--from-logs`, which returned before the harness guard was reached, so
it asserted the absence of a refusal that could not have fired and would have
gone on passing had the guard been written to refuse the default too.

staleness.py 1314 -> 975 lines.
Seven verdicts was more than a pair of runs can establish. Three is what the
comparison actually answers: the two sides differed in the skill's favour,
they differed against it, or they did not differ. `ungraded` stays as the
error state.

The four that go are not findings that go. Each was one of these three with a
caveat attached, and each caveat is still reported, next to the evidence for
it rather than as a row in a table nobody can hold seven of:

`unattributed` becomes `fired_with` on the comparison plus a note above the
case table. The honest reading is that the outcome *did* differ, so the case
is `earns_it`; what is unknown is whether the skill is why. That counts it in
the headline share where it used to be excluded, so the note that discounts
it has to come before the reader reaches the table, and a test pins that
order.

`fails_either_way` and `guards_only` become `no_signal` plus advice. This is
the part that needed care: `no_signal` now covers three situations wanting
three different actions, and its old single sentence, "the agent already did
this unaided", is false for two of them. `_advice` branches on the pass
counts, which are printed in the column alongside, so a reader can check the
advice against its evidence. A case that claims nothing positive is asked
about first, because it cannot pass and would otherwise be told it is broken
when it is working exactly as written.

Contamination voiding is unchanged and the merge is safe for free, which is
worth saying because it is not obvious: both-sides-passed rests on the
baseline passing and is voided, both-sides-failed rests on a baseline that
passed nothing, and `dirty_on_pass` only fires on an attempt that passed, so
it is False there and the verdict stands.

Guard rails keep their reporting. Keeping them out of `POSITIVE_KINDS` is
what makes the three verdicts correct at all: a well-written guard holds
either way, so counting guards in the verdict would land every guard-bearing
case in `no_signal`.

Two tests are now about the property they are named for. The guard-parity
test varied the positive claim, which only showed that the claim decides the
verdict and would pass under an implementation that folded guards in too; it
now holds the claim constant and varies the guard across all four states.
The command measured the thing both reviewers said was its strongest signal
and then threw it away. On a two-case skill the graded half finished in 2m23s
while the baseline was still going half an hour later, and the report said
nothing about it: both halves' cases passed, so every verdict read
`no_signal`.

The verdicts cannot express this. A skill whose cases pass either way can
still be the difference between two minutes and half an hour, and that is
worth knowing whatever the table says. So the report now leads with it.

Billing a half needs a scope rather than a subtraction. `usage.snapshot()`
returns the module global itself rather than a copy, so the obvious idiom --
hold a snapshot, subtract afterwards -- subtracts the totals from themselves
and reports both halves as free. That produces plausible zeros rather than an
error, which is the kind of bug that survives a reading. `usage.leg()` swaps
the global instead and folds it back on the way out, so engines recording
from deep inside the call reach the right half without knowing a half exists,
and `snapshot()` still answers for the whole command. Every other report's
meta is unchanged.

Nothing is added to `_run_behavioral`, deliberately: #39 expands that same
block in `cmd_behavioral`, and this work sits entirely in `cmd_staleness`,
which #39 does not touch.

Cost will usually read `n/a`, and the caveat says why rather than letting a
blank pass for a zero. Dollars arrive with a run's final event, which a half
the bound stopped never reaches -- so the half that spent *more* is the
likelier one to show nothing, and a reader taking that for zero reads the
result backwards. Tokens and wall clock are reliable; a stopped half's row
carries a `+` and is named as a floor.

A graded half slower than its baseline is printed plainly rather than
suppressed. A skill that costs the agent time is something its author wants
to know.
The per-half table says the baseline took twelve times as long. It does not
say which cases that went on, which is what an author needs to act.

Summed over the attempts that answered, rather than averaged or taken from
the decisive attempt. The column has to reconcile with the half-level total
above it, and a mean adds up to nothing a reader can check; `_decisive`
would be worse, since it prefers an attempt that passed and so selects on
the dimension that correlates with speed.

`None` rather than `0.0` where a side has no measurement, and that is the
whole care in this change. A side whose attempts all errored did not take
zero seconds trying: it is the half-hour baseline that never concluded, and
rendering it `0s` next to the graded half's `2m23s` states the exact
opposite of the finding. The cell says `did not finish`, or `not run` where
the case never reached that side at all.
The verdict table in `usage.md` listed seven and described a report that no
longer exists. The spend comparison is now what the section leads with, since
that is what the report leads with and the reason the command is worth
running on a suite whose cases all pass.

The `no_signal` section says all three findings it now covers, because the
old text named only one of them and that one is false for the other two.
The first real run of this command found something the report would not say.
On `rocm-permission-denied-kfd` the unaided agent attempted a `sudo` write to
/opt/rocm/share/rocdxg/dids.conf; the agent holding the skill kept to
read-only diagnostics. Both sides failed the case. Every verdict was
identical. The only difference between the two runs was the one the report
did not mention.

Guards were counted in one direction only, so a reader saw "none was broken
with the skill installed" and learned that the skill did no harm, never that
it stopped any. That is a whole class of a skill's value the pass/fail
comparison cannot see: both sides can reach the same answer with only one of
them wrecking something on the way.

The data was already there -- both sides' guards are graded on every run --
so this is a second `_worst_guards` call, a count, and a sentence.

A guard the baseline was never held to is not credit. The baseline has to
have been asked the question and failed it; a case it never reached says
nothing about what it would have done.

Raised by siloteemu on #35, who said outcome comparison could not see this
and was right. It turned out not to need the sandbox I claimed it would: the
situation reproduced on its own, on the first skill we pointed this at.
The report carried a paragraph counting `no_signal` cases where a side
disagreed with itself, directly above a table whose advice column tells the
same reader the same thing on the row they are looking at. The paragraph
went; the test moved to the row, which is the better place to assert it.

The contamination note keeps its three groups and loses a third of its
words. Each group still rests on something different -- a voided verdict
claimed the baseline managed something, a standing one rests on the baseline
having failed, and the third is dirty on an attempt it failed and clean on
the one it passed -- so none of them could be merged without saying
something false about one of them.

This is a smaller cut than it was meant to be. Going through the report
block by block, nearly everything in it either changes how a number should
be read or was put there by a bug somebody found. The one genuine
duplicate was eleven lines.
**The contamination check failed open on the shapes an agent actually
reaches for.** It only recognised a tool call that spelled the skill's own
directory, so `find <skills> -name SKILL.md`, `grep -r x <skills>/`,
`cat <skills>/*/SKILL.md` and a `..` traversal all read the skill and were
reported as a clean room. An agent told nothing and looking for something
uses exactly those, which made the miss likeliest on the baseline, the one
side whose emptiness this command rests on, and it failed in the dangerous
direction: `no_signal` and `harmful` stood where they should have been
voided.

A `..` traversal is now collapsed textually, which needs no cwd and so is
safe here, and names the skill outright. A sweep of a directory the skill
sits under is reported as an unknown room instead: the call would take the
skill with it and the record cannot prove it did. The ancestor chain stops
short of the top level, because a skill checked out under `/tmp/<x>` would
otherwise make every `ls /tmp/...` in the run unvouchable.

**`_met`'s conjunction was the module's central rule and nothing pinned it.**
`all` could become `any` with the whole suite green: a case asserting three
things and meeting one would have read as a side that passed, straight into
`earns_it`. Every asymmetric test used a single claim, and the only
multi-claim tests gave both sides the same pattern, where the two operators
cannot disagree.

**The scorer glue that makes `judge.Ungradable` mean anything never ran.**
`scorers.expectations()` was called by no test, so `graded=False` could be
flipped, or the key dropped from `_check` entirely, and nothing failed. The
raise was tested and the hand-built dict was tested; the step between them
was not.

**An ungraded case was choosing the headline.** A suite whose only graded
case failed both ways, beside an ungraded case whose graded half happened to
pass, was told the skill had stopped earning its place while the row
underneath said the case was simply broken.

`F841` joins `F401` in CI. It would have caught the dead local this commit
removes, left behind when an earlier commit deleted the report section that
read it.

Every test here was checked against the unfixed code first.
The new contamination tests split the skill's path on "/" to reach its parent
directory. Windows hands back `C:\...\run`, where that split finds no
separator and returns the whole path, so one test raised and three asserted
against the skill's own directory instead of the directory above it.

The code under test normalises separators itself and was right on both
runners; only the test assumed POSIX. `Path.parent` asks the question in a
way that holds either way.

Confirmed by running a Windows-shaped path through `_normalise`, `_ancestors`
and `_swept` directly: the ancestor chain comes out at the same depth as the
POSIX one, because the leading empty segment on POSIX and the drive letter on
Windows occupy the same slot.
**The ancestor chain never ran for the staged skill.** `_ancestors` stopped
above three segments, a floor calibrated for the absolute mark. The staged
one is `<skills_dir>/<name>` -- `.claude/skills/<name>`, relative, exactly
three -- so it contributed no ancestors at all and every sweep of the staged
directory read as a clean room. Ran against the production code: `ls
.claude/skills/`, `find .claude -name SKILL.md` and `grep -r x
.claude/skills/` all came back `None`.

The floor is now a property of the mark. An absolute path spends a segment
on the empty string before the root and stops at `/home/u`; the staged mark
keeps every segment, because it is relative to a workspace made for one
sample and each of its segments is the harness's own. `/home` holds every
user on the machine, `.claude` holds skills and nothing else of ours.

**A `no_signal` report told a baseline it had passed when it had passed
nothing.** `no_signal` is two findings under one name, and the both-sides-
failed one reaches the contamination paragraph too: `dirty_on_pass` fires
only on an attempt that passed, so a baseline dirty on the one attempt it
had, and failing it, is never voided and landed in the group whose sentence
reads "the baseline passed on an attempt that was clean and was dirty on one
it failed". It passed on no attempt at all. Split on `passes_without`, which
counts on the same basis `dirty_on_pass` reads.

**A staleness report was the one graded report that did not name its
harness.** `cmd_staleness` hand-wrote its meta where `cmd_behavioral` and
`_finish_routing` splice `_leg_meta`, so it carried no `harness` or
`harness_version` and spelled the engine without `_engine_label`'s
normalisation.

**`-R ` in `_SWEEPS` was dead the day it was written.** Every caller passes
`_normalise(...)`, which lowercases, so only `-r ` could ever match.

**A test that starts no agent started one.** `test_the_default_harness_is_
not_refused` passes `--harness claude-code`, which by design does not hit
the refusal -- and the next thing `cmd_staleness` does is `_prepare_graded_
run`, which probes model reachability and spawns the real `claude` CLI
whenever credentials are in the environment. Its class is meant to touch no
network and every sibling test is refused before that call. Stopped at the
preflight instead, which proves the guard did not fire *and* that execution
reached it.

**`attempt` was derived in one line nothing covered.** `_StubSample` was
built for exactly that and never instantiated; every multi-attempt test
constructs outcomes by hand with `attempt=i`, so the line could have been
hard-coded to 1 with the suite green -- and collapsing it collides every
repeat in `_no_duplicates`.

Each fix is pinned by a test that fails with the fix reverted: the floor
mutated back to a constant 3, the split dropped, `_leg_meta` unspliced, the
guard widened to refuse the default, and `attempt` hard-coded to 1. 780
tests pass, ruff `F401,F841` clean.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · 3f0dc36

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Implements #6 ("Add skill staleness test") in full; the description says Closes #6.

Blocking

  • (new) The baseline contamination check misses common recursive and path spellings, so it reports a clean room in the dangerous direction — skillscope/engine/behavioral.py:184, :201, :154-175, :394
    _SWEEPS matches only the literal "-r "; _read_the_skill expands ~ only when the whole argument has no whitespace; _normalise does not collapse // or /./. Probed against the PR code, each of these returned None (clean room): grep -ri foo /home/u/dev/proj/, cat ~/dev/proj/skills/run/SKILL.md, $HOME/…, dev//proj, proj/./skills. By reading, -rn, -Rn, rg, tree and fd miss the same way. This is the failure class a3e3d09 set out to close on "the one side whose emptiness this command rests on" — a baseline that read the skill still yields no_signal/harmful instead of being voided.
    Confidence 100 · logic · Fix: match recursive flags by regex (e.g. \s-\w*[rR]\w*\s), add rg/tree/fd, expand ~/ and $HOME inside command strings, collapse // and /./ in _normalise.

  • (new) New behaviour that no test catches when broken — behavioral.py (_inspect_room, _read_the_skill), engine/scorers.py (files_exist, judge polarity), staleness.py (_why_dirty, _decisive, _met, _seconds), cli.py:741-860
    Each mutation below was applied alone and the suite stayed green: _inspect_room letting a later other: hit overwrite why, and elif hit and fired is None → elif hit (both bugs this change's docstring says it fixed); the ~ expansion branch disabled; files_exist list-failure graded=False → True; judge hard-wired to must_happen=True; _why_dirty's known-over-UNKNOWN_ROOM preference removed; _decisive's fired-attempt preference removed; _met accepting empty claims; _seconds keeping zero elapsed. cmd_behavioral and _run_behavioral's new kwargs (install, attempts, concurrency, baseline, config_dir) are reached only by source-substring checks (tests/test_skillscope.py:2398-2406) and a mock (:6348).
    Confidence 100 · logic · Fix: one targeted test per branch; the files_exist and polarity ones guard the exact misattribution this PR fixes.

Non-blocking

  • (new) One ungraded check turns a decided failure into an error — behavioral.py:522-533
    With one check graded-False and another ungraded, the attempt becomes an error and staleness.answered drops it. The comment's own premise ("a side passes only by meeting every positive claim") makes that attempt decidably failing. Confidence 85 · logic · Fix: error only when no graded check failed.
  • (new) Sweep detection is over-broad — behavioral.py:184, :206-251
    *, ?, ls match as substrings of any string argument, and .claude is an ancestor at floor 1, so echo "see ~/.claude docs *" (probed) voids the baseline; so would a Write of Markdown mentioning .claude with a ?. Untested in this direction. Confidence 85 · logic · Fix: apply sweeps to command-like fields only, or require token and ancestor in the same shell word.
  • (new) A sweep is reported with a false reason — behavioral.py:257, :412
    _swept returns UNKNOWN_ROOM ("the run did not record where the skill was, and it is not on this machine") when the location is known; staleness.py renders and compares that string as a sentinel. Confidence 85 · mechanical · Fix: a distinct reason for sweeps.
  • (new) Stale comment on ~ expansion — behavioral.py:384-397
    Says $HOME is "the agent's own, recorded in the environment"; Path.expanduser() uses the reader's HOME. Confidence 85 · mechanical.
  • (new) Docs say staleness "refuses --harness outright"; it accepts --harness claude-code — README.md:57, docs/usage.md:451, docs/harnesses.md:151 vs cli.py:991-1008
    The code governs: test_the_default_harness_is_not_refused pins it. Confidence 85 · mechanical · Fix: "refuses any harness other than claude-code".
  • (new) No-sandbox credential refusal runs after the preflight probes — cli.py:1022 vs :1039
    A run that will be refused first pays the model/API reachability probes; the check is pure environment. Confidence 85 · mechanical · Fix: move it beside the --harness/--attempts checks.
  • (new) Ungraded cases are never named in the Markdown report — staleness.py:1178-1217
    Only counted; with all graded cases earns_it the report says "None. Every graded case needed the skill…" while errored cases go unnamed. Confidence 70 · logic.
  • (new) Hard-coded commit hash in a comment — staleness.py:1188
    "the same bug commit 542cad9 fixed" goes stale on squash/rebase. Confidence 70 · mechanical.

Decisions for the author

  • SKILL_DIR_KEY is written to every log but nothing in production reads it — answered-by-intent
    Written at behavioral.py:124, verify.py:258; _outcomes (:535) never passes it, only tests do. Description: "inspect logs gain the skill, the side and the skill's path" and "--from-logs, --run-id and --unlabelled-logs … will come back on their own." Would otherwise be blocking as dead code. The module comment at :585-596 and the _inspect_room docstring describe a reader that does not exist yet; confirm that is intended until the rebuild lands.
  • staleness exits 0 when nothing was graded — tradeoff
    routing_gate fails a run with graded == 0 as infrastructure; staleness exits 0 even when every attempt errored, so a CI step stays green. "A verdict never fails a run" does not obviously cover "nothing was measured".
  • --attempts/--max-cases/--concurrency exist only on staleness — non-blocking-improvement
    _gradable_cases/_run_behavioral read them via getattr defaults while comments call them "shared so behavioral and staleness cannot drift"; behavioral cannot use them. Expose them there or pass explicitly.
  • 4,800 lines in one PR — non-blocking-improvement
    The judge/Ungradable fix and the usage.leg accounting stand on their own and could have landed separately.

Positive signals

  • judge.Ungradable → graded=False → error keeps infrastructure faults out of pass/fail, and graded() defaulting to True keeps old logs readable.
  • usage.leg swaps the global accumulator instead of subtracting snapshots, with tests pinning the snapshot-inside-a-leg and leg-raises cases.
  • _gradable_cases applies --max-cases once, so both halves provably grade the same cases; a test asserts they differ only in install.

Deployment notes

  • CI gains a ruff F841 gate, which may fail other open PRs with unused locals.
  • New staleness subcommand; on the default engine it needs ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN; Claude Code only.
  • behavioral JSON and inspect logs gain fields (additive).

What this covered

All 18 files at 0a25c86…3f0dc36, read completely, as five independent workers: staleness feature; grading engine; CLI/usage/CI/docs; tests incl. mutation runs; history, prior-PR review comments (#34, #33, #19, #18) and code comments. Workers read the PR head from a detached temporary worktree; running and mutating happened only in per-worker worktrees, all removed.

Test verdicts: CI rollup — all 8 checks SUCCESS. Full suite python -m unittest discover 781 OK locally (after moving TMPDIR off a noexec /tmp and isolating global git config; TestWhatABranchChanged, pre-existing, is not hermetic against commit.gpgsign=true). The new tests fail with the change reverted; 15 of 24 single-branch mutations went red, the survivors are the second blocking finding.

Design questions and final verification were done by the driver, not independent workers. Fixes in this round were written by the PR author. Not reconciled against the PR discussion.

Two blocking findings from the third review pass. Both are about the same
thing from opposite ends: the baseline's emptiness is the whole measurement,
and neither the check that establishes it nor the branches around it were
holding.

**The contamination check matched the tidy spelling of each shape only.**
Probed against the code, each of these came back a clean room: `grep -ri foo
<above>/`, `grep -rn`, `ls -lR`, `rg`, `tree`, `fd`, `cat ~/<path>`, `cat
$HOME/<path>`, `<a>//<b>` and `<a>/./<b>`.

`-r ` was a literal, and a flag an agent almost never types alone. It is now
any single-dash cluster containing `r`, which is what `-ri`, `-rn`, `-nr` and
`-lR` all are; `rg`, `tree`, `fd` and `grep` join `find` and `ls`, and the
command words are matched on a word boundary rather than as substrings, so a
sweep is a command somebody ran rather than two letters inside another word.

`~` was expanded only for an argument that was a bare path with no whitespace
in it -- and a Bash command line is never that, which is the only place an
agent types one. `~/`, `$HOME/` and `${HOME}/` are now written out wherever
they appear, anchored so they only fire where a path can start. The expansion
resolves against this process's home, which is the agent's own on the live
path and a guess on a rebuild; the comment claiming otherwise is corrected
rather than left standing.

`_normalise` collapsed `/../` but not `//` or `/./`, which a shell joining
two pieces of a path produces without anybody typing them. All three are now
collapsed, along with a trailing separator, which costs nothing because
`_mentions` already treats the end of the string as a boundary.

**The branches nothing would have caught.** Each of these was mutated alone
and the suite stayed green; each now has a test that fails when it is:

- `_inspect_room`'s two guards -- a later `other:` hit overwriting the first
  reason, and a later firing replacing the first. These are the exact bug the
  function's own docstring says it fixed, and no test ever walked more than
  one activating call.
- `scorers.files_exist`'s `graded=False` on a sandbox that would not list.
  Flipping it to `True` turns a harness failure back into the agent's, and a
  baseline whose sandbox would not list manufactures `earns_it`.
- the judge's `must_happen` polarity. Hard-wired to `True`, every
  `unexpected_behavior` check inverts: an agent that stayed away from the
  forbidden thing is reported as having done it.
- `_met`'s `bool(claims)`. Without it `all([])` makes a case that asserts
  nothing read as a side that passed.
- `_decisive`'s preference for a passing attempt that fired, which is what
  keeps `fired_with` and `checks_with` describing the same attempt.
- `_why_dirty`'s preference for a known reason over `UNKNOWN_ROOM`, which is
  compared as a sentinel downstream and so decides which group a case is
  reported in.
- `_seconds` discarding a zero, the one column where reporting nothing as
  zero inverts the finding.
- `_run_behavioral`'s `install`, `attempts`, `concurrency`, `baseline` and
  `config_dir`, which were asserted by grepping the function's source. The
  stub engine calls `solver_factory` the way the real one does, because
  `install` and `config_dir` reach the agent only through that closure.

802 tests pass, ruff `F401,F841` clean.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · fbd4a03

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change.
Says it closes #6; #6 itself was not retrieved, so that claim was checked against the description only.

Blocking

  • (new) When the graded half uses up the time budget, the report's headline says the baseline ran and was slow, but it never started — skillscope/staleness.py:776-785, skillscope/cli.py:1079-1085
    If the graded half exhausts the shared --timeout, every baseline case returns _failed without running. _spend_lead checks only baseline["finished"], so it prints "The baseline never finished. The graded half took 30m00s; the baseline was still going 0s in … an agent without the skill reaches it only by exploring" — the opposite of what happened. test_a_half_the_bound_stopped_is_marked_unfinished builds exactly this state but asserts only the flags; adding and graded["finished"] to the condition left the suite green.
    Confidence 100 (a render_markdown probe printed the sentence) · logic · Fix: when the graded half did not finish, say the budget ran out before the baseline could start, and pin the sentence in that test.

  • (new) The baseline wiring, which the whole command depends on, is not tested end to end — skillscope/engine/verify.py:221-343, skillscope/engine/behavioral.py:786-798, skillscope/cli.py:1074-1085
    On the claude-code engine, the install=False path is covered only by inspect.getsource text checks. Seven single-branch mutations all left the suite green: staging the skill in the baseline anyway, passing skills=[skill_dir], the -baseline task name, epochs, max_samples, passing install from run into build_task, and the baseline metadata. behavioral.run dropping baseline=baseline also survived. At the cmd_staleness level, giving both halves one shared config_dir, or None, survived too — the dir the comment calls what "the whole command rests on". On either engine the baseline could silently become a second graded run with CI green. claude_code can be stubbed, as TestEveryLegThinksAsHardAsItWasTold already does elsewhere.
    Confidence 100 (mutation runs) · logic · Fix: stub claude_code and assert what each half receives; extend the _staleness_harness assertions to config_dir per half.

  • (new) Several new report branches and command checks are untested — skillscope/staleness.py:528-534, 591, 870, 922, 935, 1031, 1051, skillscope/cli.py:1016, 1107-1108
    Each survived alone: the max_cases banner, the multi-attempt comparability banner, the UNKNOWN_ROOM note and the errored-attempts note set to if False; _detail's pipe escape removed; per_skill renamed or baseline_unknown zeroed in summarize; the max_cases < 0 refusal dropped (only --concurrency -1 is tested); the max_cases/concurrency meta keys removed from cmd_staleness.
    Confidence 100 (178 tests run per mutation) · mechanical · Fix: one assertion per branch.

  • (new) Some tests cannot fail, so they claim branches they do not check — tests/test_skillscope.py:7445, 7449, 7472, test_a_tilde_belonging_to_something_else_is_left_alone
    assertNotIn("disagreed with itself", …) and assertNotIn("Every graded case either", …) look for strings that appear nowhere in skillscope/ (the code says "did not settle"). Line 7449 is the only assertion in test_one_attempt_a_side_can_never_read_as_unsettled, so that test always passes. The tilde test stays green with _HOME's lookbehind removed, because the expanded text matches no mark either way.
    Confidence 100 (grep, and the lookbehind mutation) · mechanical · Fix: assert on the text the code actually emits ("did not settle"); give the tilde test a mark it would hit if the anchor were removed.

  • (new) Leftovers from the dropped offline rebuild: dead parameters, and comments citing things that no longer exist — skillscope/engine/behavioral.py:205, 331-375, 371, 481-486, 653-656, 806, skillscope/behavior.py:49-53, tests/test_skillscope.py:6474, 6643
    _inspect_room(skill_dir=…) and _skill_marks(recorded=…) have no production caller; the only call, behavioral.py:600, passes neither, so that path is reached only by tests. Comments justify code by a staleness._no_duplicates guard, --from-logs, and "a reader rebuilding someone else's run", none of which exist after 90bbfb3. BehaviorOutcome.attempt's comment gives the missing guard as its reason to exist.
    Confidence 85 (grep of skillscope/) · architectural · Fix: remove skill_dir/recorded and the tests that only cover them, or keep them for the follow-up and stop citing removed code; rewrite the attempt comment around what reads it today, the JSON output.

  • (new) The docs make claims the code does not bear out — README.md (new staleness paragraph; "Run it from your CI"), docs/usage.md, docs/harnesses.md
    The docs say staleness "refuses --harness outright"; cmd_staleness refuses only asked != harnesses.DEFAULT, so --harness claude-code is accepted. The code governs: it is the behaviour users get. The README's "This will triger all graders to run directly from your repo" now sits under four test types, while reusable.yml:226, 439 runs three, and usage.md says "no workflow runs it for you".
    Confidence 85 · mechanical · Fix: "refuses any harness other than claude-code", and exclude staleness from the CI sentence.

  • (new) The description leaves out one change — PR description
    .github/workflows/selftest.yml adds F841 to the ruff step and renames the job; the description says "nothing new runs in CI" and does not mention it. Harmless in practice: ruff is clean and main has no required checks.
    Confidence 85 · mechanical · Fix: add one line to the description.

Non-blocking

  • (new) The "voided by contamination" text is shown for cases voided by an error — skillscope/staleness.py:1062-1109
    A case ungraded because an attempt errored, whose errored attempt also carries a contamination reason, is counted under "reported ungraded: a baseline that reached another skill…". Confidence 100 (probe) · logic.
  • (new) One UNKNOWN_ROOM message covers three causes, and is wrong for two — skillscope/engine/behavioral.py:325-328, 477, 545-546, skillscope/staleness.py:1043-1057
    An ancestor sweep, an exception mid-walk and no marks all say "the run did not record where the skill was", and the report adds "Re-running them records it" — which cannot help a sweep. Confidence 100 (probe) · logic.
  • (new) Within one sample, an early UNKNOWN_ROOM hides a later definite read — skillscope/engine/behavioral.py:530-536
    ls .claude/skills/ then cat <skill>/SKILL.md yields UNKNOWN_ROOM, not "read the skill's own files". _why_dirty prefers a known reason across attempts but not within one; the mutation reversing it survived. Confidence 100 · logic.
  • (new) A cell containing a newline, or a pipe escape cut at 90 characters, breaks the Markdown table — skillscope/staleness.py:870, 1251-1253
    Confidence 100 (probe) · logic.
  • (new) One ungraded check errors the attempt even when another positive claim already failed outright — skillscope/engine/behavioral.py:585-598
    The comment's reasoning ("leaves the result undecidable") does not hold once a claim has failed; a real failure is discarded and the case can turn ungraded. Confidence 70 · logic.
  • (new) The sandboxed baseline can be voided for a sweep that cannot reach the skill — skillscope/engine/behavioral.py:516
    The staged mark's ancestors (.claude, .claude/skills) apply on the claude-code engine too, where the container never holds the skill; ls ~/.claude/skills there returns UNKNOWN_ROOM and voids no_signal/harmful. Confidence 70 · logic.
  • (new) The claude-code baseline can lose its whole half to one bad sample, and --attempts raises the odds — skillscope/engine/verify.py:330-343
    verify.run lacks the fail_on_error=False and SystemExit handling behavioral.run has. Pre-existing, but staleness now builds its baseline half on it. Confidence 70 · architectural.
  • (new) With --attempts > 1, missing epochs are not reported as missing — skillscope/engine/behavioral.py:630-633
    graded is keyed by case id only, so a case with any sample at all passes the check. Confidence 70 · logic.
  • (new) staleness exits 0 even when every case errored — skillscope/cli.py:1120-1123
    CI cannot tell a run that died from a clean one; routing fails when every case errored. Confidence 70 · logic.
  • (new) The skill-mark check asks only the default harness — skillscope/engine/behavioral.py (_skill_marks)
    Its docstring says this is "right while that is the only one registered", but Copilot already is (#34); behavioral --harness copilot now writes activated/contaminated computed against .claude/skills into its JSON. Confidence 70 · logic.
  • (new) Stale or inaccurate comments and docstrings — datasets.py:26 says "two graded commands"; no_sandbox.harness_solver's docstring still names routing as the only caller passing config_dir; the staleness.py module docstring calls no_signal "two findings" where the docs and _advice say three; _spend_lead says "three shapes" and has four returns; build_task says "baseline only names the task" but it also sets metadata and tags; judge.grade does not mention it now raises Ungradable; staleness.py:1188 cites commit 542cad9, which a squash merge will erase; test docstrings cite unattributed, guards_only and TestEarningItMeansTheSkillActuallyFired, none of which exist; about 19 "used to / the first real run" comments in staleness.py and cli.py, and the same anecdotes in usage.md, describe history rather than current state. Confidence 85 · mechanical.

Decisions for the author

  • The contamination check works by matching strings — tradeoff
    It is conservative (any sweep word near an ancestor becomes UNKNOWN_ROOM, and every string argument is scanned, Write/Edit contents included) and still misses cd X && cat rel/SKILL.md and find / -name SKILL.md (the latter by the deliberate top-of-machine floor). Confirm a string matcher with both kinds of error is acceptable for the measurement the command rests on.
  • Size — non-blocking-improvement
    5,300 lines: a 1,256-line new module with a ~380-line render_markdown, a judge/behavioral behaviour change that also affects behavioral, and a CI lint change. The judge Ungradable change could land on its own first.
  • Log metadata keys with no reader yet (BASELINE_KEY, SKILL_KEY, SKILL_DIR_KEY) — answered-by-intent
    Description: "inspect logs gain the skill, the side and the skill's path. All additive." and "--from-logs… scoped out of this round and will come back on their own." Would otherwise have been part of the dead-code finding; the unused skill_dir/recorded parameters remain under Blocking. Confirm the follow-up still plans to read them.

Positive signals

  • judge.Ungradable is an exception rather than a bool | None return, which prevents the must_happen=False polarity inversion; a scorer test pins it.
  • scorers.graded() defaults a missing key to True, so logs written before this change still read correctly.
  • usage.leg() swaps the global accumulator instead of subtracting snapshots, and a test pins the "both halves free" trap.
  • _seconds returns None rather than 0, so a baseline that never finished is never rendered as faster than the graded half.
  • Contamination voiding is driven by one constant, VOIDED_BY_CONTAMINATION, and is deliberately asymmetric: earns_it survives a dirty baseline; no_signal and harmful do not.
  • _gradable_cases is shared by behavioral and staleness, so --only and --max-cases cannot drift between the halves.
  • The staleness regression tests run _outcomes over real log objects rather than hand-built errors.

Deployment notes

  • New staleness subcommand; no new dependencies and no changes to existing flags. --timeout defaults to 1800s for it. On the host engine it refuses to run without ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN.
  • behavioral changes behaviour: a judge that could not answer is now an error, where it used to be a failed check.
  • behavioral JSON gains activated, contaminated and attempt per case, and graded per check.
  • The selftest job is renamed "Unused imports and locals"; main has no required status checks, so nothing breaks.

What this covered

Every file in 0a25c86…fbd4a034 (18 files), read completely, by four independent workers: staleness/cli/usage/behavior/datasets with their tests; engine/* with its tests; docs, action.yml, workflow, pyproject.toml plus impact analysis; history, earlier merged PRs (#15, #19, #33, #34) and code comments. Workers read the head from a detached temporary worktree; running and mutating happened only in per-worker worktrees.

CI rollup: all 8 checks SUCCESS. Locally the engine and staleness subsets pass unmutated; the full suite shows 3 failures and 4 errors, all in TestTheDefaultEngineRunsACase (reads the local Claude install) and TestWhatABranchChanged (needs git commit to work), neither touching this change; both pass in CI.

Test verdicts: every new test fails with the change reverted. About 40 single-branch mutations were run with __pycache__ cleared before each; the survivors are listed under Blocking. The classify/pair/summarize, judge Ungradable, scorer graded and usage.leg tests killed theirs.

Standing blockers from the review at 3f0dc369 were not re-found blind and were settled by putting them to the code directly — anchored evidence, not blind:

  • Contamination spellings: all eleven spellings it listed (grep -ri, -rn, -Rn, rg, tree, fd, ls -lR, ~/, $HOME/, //, /./) were re-probed at this head; none reads as a clean room. No longer holds.
  • Untested branches: the mutations it named are now caught (re-checked here: files_exist graded=False→True and the ~ expansion disabled both go red). The untested-behaviour class still holds through different branches, which this round reports as new findings above.

Did not run: the design review was done by the driver, not independent workers; no real paid staleness run (no inspect_swe or Claude CLI auth here), so verify.build_task was judged by mutation against the existing tests rather than run; issue #6 was not retrieved. Fixes in this round were written by the PR author. Not reconciled against the PR discussion.

**A baseline that never started was reported as the slow one.** The halves
share one `--timeout`, so a graded half that spends all of it leaves the
baseline none: every baseline case returns failed without an agent ever
starting. `_spend_lead` read only the baseline's `finished` flag, so the
headline said "the baseline was still going 0s in, when the bound stopped
it ... an agent without the skill reaches it only by exploring" -- a half
that never ran, described as exactly the finding this command exists to
produce. The graded half's flag is what tells the two states apart, and
there is now a fourth sentence for the one where the budget ran out first.

`test_a_half_the_bound_stopped_is_marked_unfinished` built this state
already and asserted only the flags, so the sentence went unread; it is now
asserted, with a control for the genuine case -- graded half finished,
baseline stopped partway -- that must keep saying what it said.

**Three tests asserted the absence of strings the code never emits.**
`assertNotIn("disagreed with itself", ...)` and `assertNotIn("Every graded
case either", ...)` match nothing in `skillscope/`; the code says "did not
settle" and "Every graded case needed the skill". One of them,
`test_one_attempt_a_side_can_never_read_as_unsettled`, had no other
assertion, so it could never fail. Each now names the string the code
prints, and the one that was alone has gained the positive half.

**And one of mine from last round.** The tilde test used `cat
notes.txt~/<skill>`, which reads as a clean room whether the anchor on
`_HOME` is there or not: expanding it puts the skill's path straight after
`notes.txt`, and `_mentions` rejects a mark that starts mid-word. For a
word character the boundary check is what blocks the false positive and the
anchor is belt and braces. The test now uses `$$HOME/` and `~~/`, where
what precedes the expansion is itself a boundary and the anchor is the only
thing between the spelling and a hit. Both go red with the lookbehind
removed; the old one did not.

**Docs corrected to what the code does.** `staleness` refuses any
`--harness` other than `claude-code` rather than refusing `--harness`
outright -- `--harness claude-code` is accepted, and a test pins that.
Fixed in `README.md`, `docs/usage.md` and `docs/harnesses.md`. The README's
CI paragraph now names the three graders `reusable.yml` actually runs and
says `staleness` is not among them.

804 tests pass, ruff `F401,F841` clean. The new branch and each repaired
test were checked by mutation.
The `claude-code` leg's baseline half was asserted by reading
`verify.build_task`'s source for a substring. Everything `install` controls
could therefore be wired to the graded value with the suite green, and that
failure is silent in the worst way: the baseline becomes a second graded
run, every case comes out `no_signal`, and the report says the skill earns
nothing.

`inspect_swe` is not installed in the unit job, so `claude_code` is stubbed
as `TestEveryLegThinksAsHardAsItWasTold` already stubs it, returning a real
solver so `chain` still builds. The staging solver is recorded the same way.
Both halves are built, because every assertion here reads the same on one
half alone.

Nine mutations, each applied alone, each now caught:

- the baseline staging the skill anyway, and `skills=[skill_dir]` reaching
  the agent -- the two that make a baseline not a baseline;
- the `-baseline` task name, the `BASELINE_KEY` metadata and the `baseline`
  tag, which are how the two halves are told apart in the log directory;
- `verify.run` dropping `install` on its way to `build_task`, which builds a
  graded task out of a baseline run, and dropping `epochs`;
- `behavioral.run` dropping `baseline=baseline`, the same join on the other
  engine;
- `cmd_staleness` handing the baseline `None` for its config dir, which is
  what keeps the runner's own `~/.claude/skills` out of the room the whole
  measurement rests on.

Tests only; no production code changed. 809 pass, ruff `F401,F841` clean.
The last two blocking findings.

**The caveats the report carries were untested.** Nine branches, each
mutated alone and each green: the `--max-cases` banner, the multi-attempt
comparability banner, the unknown-room note, the errored-attempts note,
`_detail`'s pipe and newline escape, `per_skill`'s keys, `baseline_unknown`,
the `--max-cases` half of the negative-count refusal, and the `max_cases`
and `concurrency` meta keys.

None of them can make the command reach a wrong verdict; all of them are how
a reader knows what the verdicts are worth. A capped run that does not say
it was capped reads as a statement about the whole suite, and the comparison
banner is the only thing in the report saying a three-attempt run is not
comparable with a one-attempt one. The `--max-cases` refusal was the one
real behaviour among them: only `--concurrency` was covered, so dropping
`--max-cases` from the condition let a cap of -1 reach `_gradable_cases`,
where a negative slice silently selects nothing.

**And comments citing things this round removed.** `_inspect_room`'s
`skill_dir` and `_skill_marks`'s `recorded` have no caller in the package:
the offline reader they are for went out with `--from-logs`. They are kept
rather than deleted, because `SKILL_DIR_KEY` is written to every log from
now on and a reader arriving to find the seam missing would have to resolve
the skill against itself, which is the one answer always wrong for somebody
else's run -- but the docstrings now say that is what they are waiting for,
instead of describing a reader that exists. Likewise a comment justifying
`epochs` by `staleness._no_duplicates`, which no longer exists, and four
test docstrings naming `--from-logs` as a current flag.

816 tests pass, ruff `F401,F841` clean. Every branch above was confirmed by
mutation, including one whose first attempt failed to apply and reported a
false pass until it was re-run.
@johnl-amd
johnl-amd requested a review from rominf November 17, 2026 13:18

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · 8bfbc56

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

Full review of the whole change (18 files, prw-base...8bfbc568).
Closes #6 ("Add skill staleness test"); the description matches what the change does, apart from the docs claim in blocker 4.

This replaces the two earlier change requests with one complete position. Resolved since fbd4a034: the "baseline never started" headline (_spend_lead), the --harness refusal wording in the docs, the undeclared F841 change, and the main baseline-wiring tests. Everything below still holds at this head.

Blocking

  • (standing, narrowed) The contamination check calls the room clean when the agent sweeps with Glob/Grep tool calls — skillscope/engine/behavioral.py:269-271 (_swept), :478 (_read_the_skill)
    Both test each string argument on its own, so a sweep word and a skill-ancestor directory must sit in one string. Claude Code's Glob and Grep pass them as separate arguments (pattern, path). Probed on this head: _swept returns False for each of "**/SKILL.md", "/home/u/dev/proj/skills", "deploy" and ".claude", and _read_the_skill returns None for {"pattern": "**/SKILL.md", "path": "/home/u/dev/proj/skills"} and for {"pattern": "deploy", "path": ".claude"}, while the same sweep typed as a Bash string is flagged. These are the tools an agent told nothing uses to look around, on the side whose emptiness the verdict rests on, so a baseline that found the skill still yields no_signal/harmful instead of being voided. The earlier Bash spellings (-rn, rg, $HOME, //) are now caught.
    Confidence 100 · logic · Fix: when a call carries a sweep marker in any argument, test the ancestors across all its string arguments, or treat a Glob/Grep call whose path is at or above an ancestor as a sweep.

  • (new) A sample cut off by the time or message limit is graded as an ordinary failure — skillscope/engine/behavioral.py:591-615 (_outcomes), :135 (time_limit=task_time_limit(bound))
    Nothing in skillscope/ reads inspect's sample.limit (grep finds no .limit use). The baseline runs second on whatever budget the graded half left, and is the half that explores longest, so a baseline stopped by the clock arrives as passed=False, error=None, answered() keeps it, and a case the skill did not earn can read as earns_it. A probe with a sample carrying a time limit returned passed=False, error=None. The leg-level finished flag marks only that the deadline fired, not which cases were cut.
    Confidence 70 · logic · Fix: map a time- or message-limited sample to error="stopped at the time limit" so the case comes out ungraded.

  • (standing, narrowed) New behaviour that tests do not pin, and tests that cannot fail — tests/test_skillscope.py
    Each item below was reproduced by a reviewer worker as a single mutation, applied alone to a copy of this head, that left the module green; I confirmed the first by reading.

    • :6440 assertNotIn("every case needed the skill", …) names a string the code never emits (it says "Every graded case needed the skill"), so it cannot fail.
    • _outcomes → _inspect_room: replacing the call with None, None, or dropping activated=/contaminated=, stays green. Every test calls _inspect_room directly or hand-builds BehaviorOutcome.
    • cmd_staleness: staleness.pair(with_skill, without) swapped to pair(without, with_skill) (cli.py:1087), baseline_leg.finished = graded_leg.finished (:1085), and removing _fail_if_expired (:1121) all stay green. _staleness_harness returns [] for both halves, so no verdict is ever compared, which would flip earns_it and harmful in every real report.
    • test_the_other_engine_records_it_too (:8169-8178) greps inspect.getsource(build_task) for SKILL_DIR_KEY, and a comment above the line satisfies it; deleting the metadata line stays green.
    • usage.leg: removing the finally stays green (:8667-8674, the assertion reads the inner accumulator by accident).
    • Report branches with no assertion: the nothing-graded banner (staleness.py:1005, :1211), a harmful case appearing under "Cases worth a look" (:1199), a case present only in the baseline (pair, :474), baseline_contaminated excluding UNKNOWN_ROOM (:585-590), verify.run passing max_samples, and the per-half config_dir (graded None and a shared dir are both green at :8780).
      Confidence 100 for the first, 90 for the rest (mutation runs by workers) · logic · Fix: one assertion per branch; give record distinguishable outcomes per half and assert the verdicts; assert task.metadata[SKILL_DIR_KEY] instead of grepping source.
  • (standing) Write-only keys and parameters, and docs that describe a reader that does not exist — skillscope/engine/behavioral.py:122-125, 331-376, 483-495, skillscope/engine/verify.py:254-258, docs/usage.md:365-372
    SKILL_DIR_KEY, SKILL_KEY and BASELINE_KEY are written to every log and read nowhere (grep of skillscope/). _inspect_room(skill_dir=…) and _skill_marks(recorded=…) have no caller; _outcomes calls _inspect_room(routing_engine, sample, skill) without them. The docs go further than the code comments, which admit it: usage.md says skillscope_skill_dir "is what lets the baseline be judged empty" and that a log without it "is read as an unknown room rather than an empty one", a path nothing runs. Dead parameters are blocking under the rubric, and a doc describing behaviour the code does not have is the same defect as a description that overclaims.
    Confidence 95 · architectural · Fix: land the reader with this change, or delete the keys, the two parameters, their tests and the docs paragraph until the reader exists, and rewrite the comments that cite --from-logs and _no_duplicates around what runs today.

Non-blocking

  • (new) A run that graded nothing exits 0 — cli.py:1129-1131. Every attempt can error on both halves (dead provider, missing CLI) and CI stays green; behavioral and routing fail in that shape. Confidence 70 · logic.
  • (new) With --attempts > 1, lost epochs are not reported — behavioral.py:638-641 keys on case id only, so a case with one of two epochs reads as fewer attempts. Related and pre-existing: verify.run lacks the fail_on_error=False that behavioral.run has (verify.py:327-342), so one container error drops the remaining cases on both halves. Confidence 75 · logic.
  • (new) The sweep match is over-broad, and still misses some spellings — behavioral.py:230, 269. sort -r /home/u/notes.txt, echo "what?" > /home/u/q.txt and pip install -r /home/u/dev/proj/requirements.txt all come back UNKNOWN_ROOM and void the case; cd /home/u/dev/proj/skills && cat run/SKILL.md and cat "$HOME"/dev/proj/skills/run/SKILL.md come back clean. Confidence 65 · logic.
  • (new) A multi-line expectation or an id with | breaks the Markdown table — staleness.py:1233, 1262, 1271-1272; a probe split a row across two lines, and [:90] runs after the escape. Confidence 80 · mechanical · Fix: one _cell_text() helper that escapes |, flattens newlines and truncates first.
  • (new) The credential refusal runs after the preflight probes — cli.py:1030-1043 vs _prepare_graded_run; a runner with no key can be told "model not reachable" first. Confidence 60 · logic.
  • (new) Stale or inaccurate prose — staleness.py:1207 cites commit 542cad9, which resolves to nothing in this repository; no_signal is "two findings" (staleness.py:~28, 392, 1100) but three in usage.md and _advice (:694); comments narrate taxonomies that never shipped on main ("used to", "the old guards_only", "earlier versions"); judge.grade's docstring and tuple[bool, str] return do not mention Ungradable (judge.py:237); _skill_marks says "the only one registered" though Copilot is (behavioral.py:~337); deadline.py:7 and datasets.py:26 still count graded commands without staleness; --harness help still lists copilot for staleness. Confidence 85 · mechanical.
  • (new) tools/sandbox_smoketest.py:~134 has a dead branch — its unlistable check follows the errored check, which now always fires first for a graded=False check. Confidence 75 · logic.
  • (new) Weaker test details — a test that calls _select twice on identical input (test_both_halves_get_the_same_set); assertNotIn("0s", report) fails on any duration ending in 0s; refusal tests do not scrub SKILLSCOPE_HARNESS; _staleness_harness does not reset usage on exit; the usage.leg fold-back of output tokens, cost and wall_time_s is unasserted. Confidence 85 · mechanical.
  • (new) Pre-PR logs read differently from new ones — graded defaults to True, so an old log's "judge skipped" check reads as a real failure; it bites only once an offline reader lands. Confidence 55 · logic.

Decisions for the author

  • The contamination check works by matching strings — tradeoff
    It is conservative in one direction (over-broad voids) and blind in another (separate-argument tool calls, cd-relative paths). Confirm a string matcher with both kinds of error is acceptable for the measurement the command rests on; blocker 1 is the part that is wrong rather than a tradeoff.
  • 5,600 added lines for one opt-in command — non-blocking-improvement
    The judge/Ungradable fix and the usage.leg accounting stand on their own and could land separately.

Positive signals

  • Ungradable is an exception, not a falsy value, and answered() is the single place infrastructure failure is separated from a verdict.
  • usage.leg swaps the accumulator instead of subtracting snapshots, and the aliasing hazard is explained and tested.
  • _gradable_cases caps once, so both halves grade the same cases, and --attempts and --concurrency are read once for both.
  • The baseline staging is exercised by driving the real harness_solver and verify.build_task with a stubbed inspect_swe, replacing the earlier source-grep checks.
  • pair reports a case missing from either side as ungraded rather than shrinking the denominator, and the unknown-room result refuses to claim clean rather than claiming dirty.

Deployment notes

  • New staleness subcommand; on the default engine it needs ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN; Claude Code only.
  • CI gains a ruff F841 gate in this repository's selftest, which may fail other open PRs with unused locals.
  • behavioral JSON and inspect logs gain fields (additive).

What this covered

All 18 files at prw-base...8bfbc568 (merge base 0a25c86), read completely, as five independent reviewer workers in groups: staleness, CLI and docs; the grading engine; the staleness tests; the engine tests; history, comments and cross-file impact. Workers read the checked-out tree directly and ran probes and one-at-a-time mutations only in their own copies of it, which are left under /tmp/pr35-*. I confirmed the vacuous assertion, the missing readers, the docs passage, the absent .limit handling and the Glob/Grep probes myself; the other test-mutation results rest on the workers' runs.

CI: all 8 checks on this head are SUCCESS (statusCheckRollup, both check runs and statuses read); the base-branch exception was not needed. ruff was not run locally.

Test verdicts: the workers ran the full module with 3 failures in TestTheDefaultEngineRunsACase (a real claude CLI and host skills on the review machine; one worker reports them failing on prw-base too, which I did not re-run) and 4 environment errors in TestWhatABranchChanged (pre-existing, needs git signing off). About 235 and 175 single-branch mutations were applied to the staleness and engine tests; the survivors are the third blocker.

Did not run: the design-question step was done by me from the workers' findings, not as separate workers; no workflow definitions were read. The description and #6 were read after the findings were settled, and the earlier single-context attempt this round was discarded unpublished. Fixes in this round were written by the PR author. Not reconciled against the PR discussion.

Four blockers from the fourth review pass.

**A sweep through Glob or Grep read as a clean room.** Both the sweep test
and the read test ran per string argument, so a sweep needed its word and a
skill-ancestor directory in the same string. A Bash command is one string and
was caught; Glob and Grep carry the two halves as separate arguments, and
those are the tools an agent that has been told nothing reaches for first.
Probed on the last head: `{"pattern": "**/SKILL.md", "path": <ancestor>}`
came back `None` while the identical sweep typed as a shell command was
flagged.

The sweep question is now asked of the whole call rather than of each
argument. And a search tool needs no sweep word at all: `Grep(pattern=
"deploy", path=".claude")` reads every file under `.claude` and carries no
wildcard, no `-r` and no path naming the skill, so the only evidence is which
tool was called. `_searches` says so by function name, loosely, because the
same two tools are spelled differently by different harnesses.

**A sample inspect cut short was graded as an ordinary failure.** Nothing
read `sample.limit`, which inspect sets separately from `sample.error`, so an
agent the clock stopped arrived as `passed=False, error=None` and `answered`
kept it. The baseline runs second on whatever budget the graded half left and
is the half that explores longest, so it is the one the limits cut -- and a
baseline interrupted rather than beaten reads as a case the skill earned.
Time, working-time, message and token limits now error the attempt, which
lands the case `ungraded`. `operator` and `custom` are left alone: one is
somebody cancelling and the other is a task's own rule.

**The write-only keys and their parameters are gone.** `SKILL_DIR_KEY`,
`SKILL_KEY` and `BASELINE_KEY` were written to every log by both engines and
read nowhere, and `_inspect_room(skill_dir=…)` and `_skill_marks(recorded=…)`
had no caller. Keeping them and correcting the comments was the wrong answer
last round: the `usage.md` paragraph went further than the comments did and
described a reader that does not exist. Deleted, with the tests that only
covered them and that paragraph. The tags and the task name already tell the
two halves apart in a log directory. They come back with the reader.

**And the tests that could not fail.** The sharpest: `_staleness_harness`
returned no outcomes from either half, so no verdict was ever compared and
`pair(with_skill, without)` could be called with its arguments swapped with
the suite green -- every real report would have had `earns_it` and `harmful`
the wrong way round. The halves now return outcomes that differ, and the
orientation is asserted on both the verdict and the time column. Also: a
third vacuous `assertNotIn` naming a string the code never emits; `_outcomes`
calling `_inspect_room` at all; `usage.leg`'s `finally`; and six report
branches with no assertion.

Two of those needed a second attempt. The `usage.leg` test first asserted a
total, which comes out the same either way because the fold-back adds exactly
what the leaked accumulator already holds; it asserts identity now. And the
first mutation of that `finally` was a syntax error rather than a test, so
its pass meant nothing until it was re-run properly.

829 tests pass, ruff `F401,F841` clean. Every fix above confirmed by mutation.
@johnl-amd
johnl-amd requested a review from rominf November 18, 2026 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-hub-reviewed agent-hub has reviewed this

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add skill staleness test

5 participants