Repository navigation
Conversation
|
Thanks for this. The with/without-skill comparison answers a question that our A heads-up, not a blocker. #33 (the
None of this changes your design. It's only about where the Claude-specific knowledge lives. I'm happy to help with the rebase or review it once it's out of draft. If you'd rather land this first and have #34 adapt to it, tell me. I just want to avoid us both rewriting |
93547d5 to
5fa09b0
Compare
|
Rebased on main. The only conflict was I also did the contamination one. The other three I cannot do from here. On who adapts to whom: I would rather #34 land first and then rebase this on it. The credential check and the engine comparison in |
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 5fa09b0e
Review — needs work
Full review of the whole change (base 39eff043…head 5fa09b0e). Implements the staleness check requested in #6 ("How do we check whether a skill is stale"); #6 is still open, which is expected since it closes on merge rather than on PR open.
Blocking
-
A tool-call shape
_inspect_roomcannot walk is silently reported as a clean room instead ofUNKNOWN_ROOM—skillscope/engine/behavioral.py:359-375.
The function's own docstring states the intended contract: "a shape this cannot walk is an unknown room, not a failed case." But theexcept Exceptionbranch at line 373-374 returns the exact same expression as a walk that completed cleanly and found nothing (why or (UNKNOWN_ROOM if unknown else None)), andunknownis set once before the walk starts (marks is None, line 358) — it is never updated when the walk is cut short._activation_event(skillscope/engine/routing.py:240,return _activation_event_from(call.function, call.arguments or {})) accessescall.functionwith nogetattrguard, unlike every other accessor in this same call chain (_read_the_skill,_tool_calls), so a call object of an unexpected shape (plausible from--from-logsreading an older/foreign log, which this exact code path exists to tolerate) raises partway through the loop. Any contamination evidence on a later tool call is never inspected, and the room is reported as un-contaminated — silently restoring ano_signal/harmfulverdict the module is otherwise careful to void whenever the baseline can't be vouched for.
Confidence 70 · logic · Fix: in theexceptbranch, treat an interrupted walk asUNKNOWN_ROOMunconditionally (e.g.return fired, why or UNKNOWN_ROOM), rather than reusing the "walk completed, nothing found" fallback. -
No test exercises the
harmfulverdict under baseline contamination —tests/test_skillscope.py:6283-6337(TestADirtyBaselineCannotShowWhatAnAgentDoesUnaided).
VOIDED_BY_CONTAMINATION(skillscope/staleness.py:165) names bothno_signalandharmfulas voided when the baseline is dirty, and this class tests theno_signalcase and theearns_it-survives case, but never pairs a contaminated baseline with aharmfulresult. I confirmed by readingclassify()that the current code handles it correctly, so this is a coverage gap rather than a live bug — but it means a future regression narrowing the voided set to just("no_signal",)would pass the whole suite while silently reporting an actively harmful skill as standing evidence from a baseline that wasn't clean.
Confidence 75 · logic · Fix: add a case pairing a contaminated baseline with aharmfulresult and assert it comes outungraded. -
cmd_staleness's documented non-zero exit on an expired deadline is untested at the command level —tests/test_skillscope.py:7179-7233(TestTheSecondPassGetsABudgetOfItsOwn),skillscope/cli.py:946-947.
The command's own help text says a completed run always exits 0 but "an expired deadline or a refused run still exits non-zero." Refusal is tested viaassertRaises(SystemExit)and the happy path is tested to return 0, and_fail_if_expired()has a generic unit test elsewhere, but nothing drivescmd_stalenessthrough to its final_fail_if_expired()gate with an expired/interrupted deadline to confirm the command-level exit code the help text promises.
Confidence 70 · logic · Fix: add a test that mocks the deadline as expired/interrupted and assertscmd_stalenessreturns non-zero.
Non-blocking
None
Decisions for the author
- Contamination on the graded ("with-skill") side is recorded but never consulted by
classify()/the voiding logic, which only inspectswithout.contaminated— non-blocking-improvement. In principle anearns_itverdict could be credited to the wrong skill if an unrelatedother:skill also fired on the graded side, but the harness's isolation design (tempconfig_dir, the sandboxedclaude-codeguest having no user-level skills) makes this close to unreachable given the engines this PR wires up, and the docstrings scope the contamination check specifically to the baseline's unique claim of being empty. Worth a one-line docstring note if this is intentional, otherwise worth symmetric handling.
Positive signals
_mentions's boundary-matching (skillscope/engine/behavioral.py:223-240) and its docstring calling out the exact prior bugs it fixes (/run/user/1000matching a skill mark,foo-extramatchingfoo) is a good example of a fix that documents the regression it closes, not just the new behavior.- The contamination-voiding asymmetry (
earns_it/fails_either_waysurvive,no_signal/harmfulare voided) is implemented exactly as the PR describes and is directly covered by tests (tests/test_skillscope.py:6296-6307).
Deployment notes
None — staleness is opt-in, confirmed absent from selection.plan() and .github/workflows/reusable.yml/selftest.yml/skill-evals.yml, so no downstream repo starts running it as a side effect of this merge.
What this covered
Every changed file read in full against the PR head (skillscope/staleness.py, skillscope/engine/behavioral.py, skillscope/behavior.py, skillscope/cli.py, skillscope/engine/no_sandbox.py, skillscope/engine/verify.py, skillscope/datasets.py, tests/test_skillscope.py in full — 7238 lines, README.md, action.yml, docs/usage.md, docs/authoring-evals.md, pyproject.toml), split across four independent fan-out workers grouped by coupling (staleness engine core; CLI+engine wiring; test suite; docs+config), plus my own consolidation, a light history pass on the flagged region, and a direct source check confirming each worker's candidate findings against skillscope/engine/routing.py and the test file. All 8 CI checks on this commit are green (selftest workflow, incl. unit tests on Linux/Windows × Python 3.10/3.12, sandbox machinery, action-against-a-throwaway-repo).
Did not run: agent-instruction-adherence pass (no CLAUDE.md/AGENTS.md in this repository); prior-changes pass (no related earlier PRs on these specific files were checked in depth, since staleness.py is new and this is its first review round).
Not reconciled against any review discussion — none exists on this PR as of this head commit.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Review — needs workFull review of the whole change ( Blocking
Non-blockingNone Decisions for the author
Positive signals
Deployment notesOpt-in only: absent from What this coveredEvery file in the diff, read completely, split across three independent workers grouped by coupling: (1) Did not run: agent-instruction adherence — no 🤖 Automated review by an agent. |
johnl-amd
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 9f4527f8
Review — needs work
Full review of the whole change (base 39eff043…head 9f4527f849e2ce463980f6ea13eb23165e94e5f7, +4526/-66 across 13 files, 25 commits), split across four independent fan-out workers grouped by coupling (the staleness engine core; CLI/engine wiring, selection and CI gating config; the test suite; docs and config), each blind to the others and to this PR's existing comment thread. Closes #6 ("Add skill staleness test" — how do we check whether a skill is stale). The description accurately describes what the diff implements, including its own disclosed gaps (the sandboxed claude-code leg wasn't exercised against a live agent), and the change does close what #6 asks for.
Blocking
- A guard rail broken with the skill installed can go completely unreported when
--attempts> 1 —skillscope/staleness.py:724-729(summarize'sbroken/guard_checkscomputation),skillscope/staleness.py:195-224(_decisive),skillscope/staleness.py:410(checks_with=list(here.checks) if here else [])
The module's own stated purpose for this is unambiguous: "Guard rails, counted separately. A broken one is the serious finding here: a skill that made the agent do the forbidden thing" (also stated indocs/usage.md: "a guard broken with the skill installed is the serious finding"). Butsummarize()only inspectschecks_with, whichpair()populates from_decisive(these)— "one that passed, else the first" — selected purely on positive claims (expected_behavior/logs_contain/files_exist) and firing, with no awareness of guard-rail (unexpected_behavior) results. Verified directly by constructing two synthetic attempts for one case (attempt 1: positive check passes, guard passes; attempt 2: positive check fails, guard fails — "ran sudo!") and runningpair()+summarize(): the result reportsverdict: earns_it,guards_broken_with_skill: 0, even though the skill broke a guard rail on one of its attempts with the skill installed. The contamination-voiding logic in the same file correctly scans across the relevant attempts for its own purpose, which makes this an inconsistency within the new code rather than a deliberate simplification.
Confidence 90 (confirmed by direct execution against the realpair/summarizefunctions) · logic · Fix: havesummarize()check guard rails across every with-skill attempt for a case, not only the decisive one — e.g. by havingComparison/pair()retain each attempt's full check list rather than collapsing to the decisive attempt before the guard scan runs.
Non-blocking
docs/usage.mdoverstates "It reports and never gates" —docs/usage.md:395-399
The doc's blanket claim, read immediately after a paragraph about--timeouttruncation, omits that an expired deadline or a refused run still exits non-zero — which the CLI's own--helptext (skillscope/cli.py'sstaleness_parserdescription: "a verdict here never fails a run, though an expired deadline or a refused run still exits non-zero") andcmd_staleness's own docstring both state explicitly, and which_fail_if_expired()'s return value confirms in code. A reader of only this doc (not--help) would reasonably conclude a staleness run can never produce a non-zero exit, which is false for a timed-out or refused run.
Confidence 80 · logic · Fix: add the expired-deadline/refused-run exception to the "It reports and never gates" sentence, matching the CLI's own--helptext.
Decisions for the author
None
Positive signals
- The verdict classification (
earns_it/harmful/unattributed/no_signal/fails_either_way/guards_only/ungraded) was directly mutation-tested against five of its most load-bearing branches (the firing guard, the contamination-voiding set, the discriminating-share exclusion ofguards_only, the--from-logsnewest-run selection, and the attempt-collision key) — each mutation was caught immediately by the specific test the code's own docstrings/history point to, with no false-green behavior found. _mentions's path-boundary matching was exercised directly against 16 cases (boundary at string start/end, sibling-name false positives likefoo-extra/foo.bak/myfoo, overlapping occurrences, empty text/mark) — all correct, no false positives or infinite loops.- The contamination-voiding asymmetry (
no_signal/harmfulvoided by a dirty baseline,earns_it/fails_either_waynot) is implemented exactly as documented and is directly tested. - Baseline cleanliness for the default engine is real, not assumed: both halves run against a throwaway
CLAUDE_CONFIG_DIR, gated behindrouting.can_isolate_config()requiringANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKENbefore aclaude-code-no-sandboxrun is allowed to start at all.
Deployment notes
- GitHub reports this PR's
mergeable_stateasdirty(conflicts with currentmain) at the time of this review — a rebase will be needed before merging regardless of this review's findings. stalenessis confirmed opt-in and not wired into CI: absent fromselection.plan()and from.github/workflows/reusable.yml, so no downstream repo starts running it as a side effect of this merge.- No CI checks are recorded against this exact head commit (
gh/the GitHub API report 0 check runs for9f4527f8) — unlike the other PRs reviewed in this batch, there is no CI-level confirmation that the 25-commit, 4500+-line change passes anywhere other than the author's own local run described in the PR body.
What this covered
Every changed file read in full against the PR head by one of four independent fan-out workers grouped by coupling: skillscope/staleness.py (the new ~1282-line module, core verdict/contamination/log-pairing logic), skillscope/cli.py/skillscope/selection.py/.github/workflows/skill-evals.yml (CLI wiring, gate reporting, CI annotation), tests/test_skillscope.py (the ~2100-line test diff, verified via direct mutation testing in a disposable scratch clone), and README.md/docs/usage.md/docs/authoring-evals.md/pyproject.toml/tools/benchmark_engines.py (docs and config, cross-checked against the actual CLI help text and flag defaults, including running --help against an extracted copy of the PR tree). Findings were consolidated and re-verified against source directly before inclusion in this report.
Did not run as a separate pass: agent-instruction adherence (no CLAUDE.md/AGENTS.md exists anywhere in this repository). History and prior-changes were not run as fully separate dedicated passes this round beyond what each coupled-file worker incidentally checked (e.g. git log context on the touched modules); no contradiction with history was found where it was checked. Not reconciled against this PR's existing comment thread (3 comments) or any earlier review round on a prior head of this same branch — this pass was kept blind to those by design, per the review method's independence rule.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Review — needs workFull review of the whole change (29 commits, Blocking
Non-blocking
Decisions for the author
Positive signals
Deployment notes
What this coveredEvery changed file read completely across six parallel, independent fan-out workers plus direct verification of the findings above: Did not run: a "prior changes" pass in the strict sense (review comments on other PRs that touched these files) — no such retrievable record beyond in-repo pattern comparison, which was done. No Not reconciled against any prior review or PR discussion — this is a from-scratch blind review of the current, full diff. 🤖 Automated review by an agent. |
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (base 7a8c00b9…head 4b99e511, +4643/-65 across 13 files, 29 commits).
Implements the full ask of issue #6 ("Add skill staleness test" — how do we check whether a skill is stale), which remains open (expected: it closes on merge, not on PR open). The change goes beyond the ticket's minimal framing with a seven-verdict taxonomy, contamination detection, and a --from-logs offline-rebuild path.
Blocking
-
A tool-call shape
_inspect_roomcannot walk is silently reported as a clean room —skillscope/engine/behavioral.py(_inspect_room, theexcept Exceptionbranch),skillscope/engine/routing.py(_activation_event/_activation_event_from).
The function's own docstring states the contract: "a shape this cannot walk is an unknown room, not a failed case." Butunknownis computed once before the walk starts, from whether_skill_marksresolved a path — never updated if the walk itself is cut short. Theexcept Exceptionbranch then returnsfired, why or (UNKNOWN_ROOM if unknown else None), the identical expression a clean, completed walk that found nothing would return._activation_eventdoescall.function/call.argumentswith nogetattrguard, unlike every other accessor in this chain, so a call object of a shape it doesn't expect (plausible on--from-logsreading an older or foreign log — exactly the input this code exists to tolerate) raises mid-loop, and any contamination evidence on a later tool call is never inspected; the room is reported un-contaminated instead ofUNKNOWN_ROOM. No test constructs this case (marks resolved, walk raises partway) — the one existingUNKNOWN_ROOMtest only covers the pre-walk "marks never resolved" path. Because_inspect_roomnow runs on everybehavioral/verifysample, not onlystalenessones, this silently corrupts the newcontaminated/activatedfields on any run, and directly defeatsstaleness'sno_signal/harmfulvoiding logic on exactly the runs most in need of it.
Confidence 90 · logic · Fix:return fired, why or UNKNOWN_ROOMunconditionally in theexceptbranch. -
Several new tests are structurally unable to fail under the regression each claims to guard against —
tests/test_skillscope.py:TestTheRoomIsJudgedWhereTheRunWasNotWhereTheReaderIs::test_the_verdict_is_the_same_from_any_directorypasses the same resolvedskill_dir=ownto both compared calls, so_skill_marks's cwd-dependent fallback is never reached by either call; the chosen argument ({"file_path": "run/SKILL.md"}) also cannot match either mark under any implementation, so both calls returnNoneregardless.assertEqual(None, None)passes whether or not the cwd-independence property holds.TestADirtyBaselineCannotShowWhatAnAgentDoesUnaided::test_an_earns_it_survives_its_own_contaminationsets up a baseline that fails, sodirty_on_pass(which requires a contaminated attempt that passed) is false regardless of whetherearns_itis correctly excluded fromVOIDED_BY_CONTAMINATION— a tuple that wrongly included"earns_it"would pass this test unchanged.TestGuardRailsAreNotJudgedAsIfTheyDiscriminated::test_a_guard_holding_on_both_sides_does_not_make_a_case_no_signalvaries only the positive claim (which alone decides the verdict); guard-state parity is inert in the fixture, so it can't distinguish correct guard-exclusion from a bug that folds guards into the verdict.TestTheCostControlsBindBothHalvesEqually::test_both_halves_get_the_same_setassertsf(x) == f(x)on a pure function without driving a real two-halvescmd_stalenesscomparison.TestAFlagThatCannotDoAnythingIsRefused::test_the_default_harness_is_not_refusedalways supplies--from-logs, which the harness-refusal check is unconditionally gated behind; the test silently exercises an unrelated "spent flags" refusal instead of the one it names.TestTheSecondPassGetsABudgetOfItsOwn(whole class) is named for per-pass budgeting, but its own docstring says "two passes under one bound," and every one of its five tests only checks argparse default/help-text wiring — none exercises or could distinguish a sharedDeadlinefrom an independent one per pass.
Each is a distinct authoring mistake, but the recurrence (at least six instances across the PR's ~27 new test classes) across otherwise well-targeted, well-documented tests suggests new tests in this PR were not routinely run against a deliberately-broken implementation before being accepted as coverage.
Confidence 85-90 (each traced directly against the production code it claims to cover) · test-coverage · Fix: for each, either change the fixture/argument to one that actually reaches the differentiating branch, or rename/re-scope the test to what it actually checks.
-
staleness's own report never discloses whether the run was sandboxed/isolated —skillscope/staleness.py(render_markdown), contrastskillscope/behavior.py's_isolation_note.
cmd_staleness's summary meta spreads**_sandbox_meta(args)in, identically tocmd_behavioral's — but unlikebehavior.py'srender_markdown, which calls_isolation_note(meta)to print "Cases ran isolated" or warn "Cases ran unsandboxed",staleness.py'srender_markdownnever reads or prints that data anywhere. For a command whose entire premise is "the baseline was a genuinely empty, trustworthy room," omitting whether the room was even sandboxed understates exactly the risk the feature exists to surface, and reads as a stronger guarantee than the run actually gives.
Confidence 80 (confirmed by source cross-check; not exercised by an actual run with missing isolation) · architectural · Fix: call an isolation-note helper fromstaleness.render_markdowntoo.
Non-blocking
docs/usage.mdoverstates staleness's harness refusal as a "stronger version of the same reason" routing refuses — contrastcmd_staleness's actual check. Routing's refusal is credential-scaled per harness; staleness's is an unconditional block on any non-default--harness, regardless of what credentials are present. A reader could wrongly conclude a correctly-credentialed non-default harness would be let through.
Confidence 80 · mechanical · Fix: state plainly that staleness has no leg for any non-default harness yet, rather than framing it as the same check scaled up.cmd_staleness's live-run meta dict is built by hand instead of reusing_leg_meta(args)—skillscope/cli.py, contrastcmd_behavioral's**_leg_meta(args). This omits theharness/harness_versionkeys from staleness's JSON report entirely and skips_engine_label()'s translation. Inert today, since staleness force-restricts to the default harness — but the schema gap becomes a live inconsistency the moment that restriction is loosened for another harness.
Confidence 80 · mechanical · Fix: use**_leg_meta(args)incmd_stalenessthe same way the other graded commands do.
Decisions for the author
- Both halves of one
stalenessinvocation share a single temporaryCLAUDE_CONFIG_DIR— tradeoff. The contamination detector only inspects what the agent's tool calls did, not whether the underlying CLI process itself persists any non-project-keyed state into that shared directory across the two halves. Worth confirming this is safe, or giving each half its own fresh directory, which costs nothing extra. cmd_stalenesshand-rolls a Claude-Code-specific isolation check rather than the genericHarness.can_isolate_room()abstractionengine/routing.pyalready uses for the identical concern — tradeoff. Functionally equivalent today, since staleness is restricted to the default harness regardless. But a later harness generalization can't just drop the blanket non-default-harness refusal and reuse what already exists; it will need to swap this check out too. Worth confirming this was a deliberate staged rollout.- The judge-availability guard in
_prepare_graded_run(scoped tocommand == "behavioral") was not extended tostaleness, unlike the sibling_require_hook_supportguard in this same diff (which was) — non-blocking-improvement. Currently masked by staleness's own, unrelated non-default-harness refusal, which happens to produce the same practical effect today for a different reason. Worth confirming this omission is intentional.
Positive signals
_mentions's boundary-aware path matching explicitly documents and defends against two real historical false-positive bugs (a bare mark matching inside an unrelated longer path; a sibling skill name matching this one's) — a model of a fix that names the regression it closes.- The contamination-voiding asymmetry (
no_signal/harmfulvoided by a dirty baseline,earns_it/fails_either_waynot) is implemented correctly, and the fired/why conflation it depends on is defended by a genuinely well-targeted test class. _worst_guardsscans every answered attempt for a side rather than only the one the headline verdict rests on, so a guard broken on a non-decisive attempt is still surfaced in the report.- A
guards_onlycase (no positive claim to credit or fail) is carved out of the report's "stood despite a dirty baseline" narrative into its own, accurately-worded bucket, rather than being told it "rests on the baseline failing" when it asserts nothing that could. - The
--from-logsrebuild path correctly withholdsmodel/engine/effortfrom its report meta, since a rebuild reading old logs did not itself run anything on them.
Deployment notes
stalenessis opt-in only: absent fromselection.plan()and from all three workflow files (reusable.yml,selftest.yml,skill-evals.yml) — confirmed directly, no downstream repo starts running it as a side effect of this merge.- The default engine (
claude-code-no-sandbox) refuses to start withoutANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKENin the environment, since its isolation mechanism only works when auth comes from there rather than the runner's own config. - Budget roughly twice a
behavioralrun's wall-clock and cost per the command's own documented defaults and example. - All 8 CI checks are green on the current head commit (
4b99e511): unit tests on Linux/Windows × Python 3.10/3.12, sandbox machinery, unused-imports, and the action run against a throwaway repo.
What this covered
Every file in the diff read completely at the PR head, split across five independent fan-out workers grouped by coupling: skillscope/staleness.py + skillscope/behavior.py (core verdict/contamination logic); skillscope/cli.py (CLI wiring); skillscope/engine/behavioral.py + engine/verify.py + engine/no_sandbox.py + datasets.py (engine integration); tests/test_skillscope.py (itself further split by that worker into four independent sub-workers across its ~27 new test classes); README.md + docs/usage.md + docs/authoring-evals.md + action.yml + pyproject.toml (docs/config). All workers were blind to the PR's description, comments and each other's findings. The highest-impact candidates were then independently re-traced directly against the source at 7a8c00b9…4b99e511 before scoring.
Did not run: agent-instruction adherence (no CLAUDE.md/AGENTS.md in this repository); a dedicated "prior changes" pass beyond git log/git blame on the flagged regions, which surfaced nothing that changed this round's conclusions. Not reconciled against this PR's existing review/comment thread — this was a from-scratch blind pass over the code at its current head.
Automated review by an agent.
|
Thanks John, and thanks for the thorough write-up. The concern is real: we shouldn't ship skills that do nothing. I'd like to raise a higher-level question before we invest further, because "does the skill beat no skill, per case" seems to me a narrow slice of what skill value is. This is about intent and scope rather than the implementation. 1. Value isn't a boolean. A skill that is 90% unnecessary is a problem too, and 2. Our evals weren't designed to cover a skill's value. Skill behavior is too open-ended for an eval set to cover fully, and many cases exist for routing or guard rails rather than for what the skill adds. Running every case twice spends most of the compute on cases never meant to show value, and 3. What is the bar? The PR defines it as pass/fail, and with 4. Part of a skill's value is doing things safely. A skill may reach the same outcome as no skill, but make sure the agent doesn't cause harm on the way: no destructive commands, no leaked credentials, no changes to the wrong environment. Outcome comparison can't see that. The PR flags a guard broken with the skill, but doesn't credit a skill for guards the baseline would break, and 5. Skills that exist for environment coverage. Some skills (rocm-doctor is the example I have in mind) make a straightforward task work across many environments, GPUs and failure states. A single baseline in a clean sandbox can't exercise that, so the verdict says little about their value. I'd want to know whether we can even set up those environments for evals. 6. Continuous cost for a mostly-true answer. I'd expect this check to almost always come back "yes, the skill matters", and it is least informative for exactly the safety and environment-coverage cases above. Running it continuously seems a poor use of effort compared with other ways of evaluating skill quality and value. So my concern isn't the implementation. I'd rather first decide what we want to learn about skill value and whether this is the way to learn it. If we keep it, I'd suggest a much thinner version (time/cost to completion, opt-in, run in the skill's own repo) rather than the full verdict taxonomy. Happy to discuss on a call. |
rominf
left a comment
There was a problem hiding this comment.
Right idea, but it does too much at once. One real gap and the scope question need settling before merge.
Direction: grading cases against a no-skill baseline is a useful addition, and keeping it opt-in and out of CI is right. But this is about 1.3k lines of new module plus 2.1k lines of tests for one opt-in command, and the question raised in the thread above (what do we actually want to learn about a skill's value?) is still unanswered. I'd answer that first and land a smaller core: the paired run, the verdicts, --attempts and contamination voiding. --from-logs, --run-id and --unlabelled-logs can follow. Reporting graded-vs-baseline time and cost costs almost nothing and covers that comment's point 3.
Before merge:
- A judge that could not grade counts as a real failure, and staleness turns that into a verdict.
engine/judge.py:235and:280returnFalsewith no error when the sandbox listing fails or the judge returns no JSON.answered()keeps those attempts, so a judge hiccup on the baseline givesearns_itand one on the graded side givesharmful. Inbehavioralthis only shows up as a red case. Here it produces the headline verdict. Treat those checks as unanswered (so the case comes outungraded), or add agradedflag to checks, the way #26 does for the legacy path. - #34 has merged. On Nov 3 you planned to convert the credential and engine checks to
can_isolate_room,credentialsandresolve_legonce it landed. What landed instead is a blanket refusal of any non-default harness (cli.py:990). That's fine as a staged step, but say so in the PR and track it in an issue. Right nowdocs/usage.md:467andREADME.md:53call it "the same terms" as routing, and it isn't. - #39 rewrites the room check and the judge path in
cli.py,engine/behavioral.py,engine/no_sandbox.pyanddocs/usage.md, so agree on a merge order with volen-silo. #29 touches the same engine files.
AgentHub check (latest review):
- Blocking #1 (
_inspect_room's except branch): the mechanism is real (behavioral.py:373), but calling it blocking overstates it. The calls come from inspect'smessage.tool_calls(routing.py:354), which always have.function, so there's no realistic trigger and no evidence it corrupts every run. Take the one-line fix anyway. It was first raised on Nov 4. - Blocking #2 (tests that can't fail): overstated as blocking.
- Two really can't fail, so fix them: the cwd test (
test_skillscope.py:7261) and the default-harness test (:6776), because--from-logsskips the refusal atcli.py:991. - The earns_it-survives, guard-parity and same-set tests check properties that hold by construction, since
earns_itneeds a failing baseline anddirty_on_passonly fires on a passing one. They're harmless. - The point on
TestTheSecondPassGetsABudgetOfItsOwnis wrong. Its tests do check the doubled timeout and the help text. Only the class name is off.
- Two really can't fail, so fix them: the cwd test (
- Blocking #3 (no isolation note): a parity nit, not blocking. Whether the baseline is empty depends on the isolated config dir and the contamination check, not on the filesystem sandbox. Printing
_isolation_noteis still cheap. - Judge guard not extended to staleness: moot. Staleness refuses non-default harnesses earlier, and #39 deletes that guard.
- Shared
CLAUDE_CONFIG_DIRacross halves: I can't confirm or rule out a leak, because workspaces are per sample. Two temp dirs cost nothing, so I'd just use them. - Hand-built meta instead of
_leg_meta: confirmed, but has no effect today. Optional.
volen-silo
left a comment
There was a problem hiding this comment.
🟡 Automated review · 4b99e51
🟡 Blocking: yes · Non-blocking: yes · Decisions: yes
Review — needs work
Full review of the whole change: PR #35 head 4b99e511 against the main tip 7a8c00b9, three-dot diff, 13 files, 4643 added lines.
Closes #6, but the issue is a single vague sentence, so the change cannot be said to under- or over-implement it. The description matches the code except for the Copilot credential claim in docs/usage.md, which is under Blocking.
Blocking
-
The sandbox (
--engine claude-code) baseline has no test. —skillscope/engine/verify.py:252-293,skillscope/engine/verify.py:337-353
Each of these mutations left the full suite green:skills=[skill_dir] if install else []changed to[skill_dir]; always staging the skill;BASELINE_KEY: not install; the task name and tags;install=installnot forwarded fromrun();epochs=attempts;max_samples. The first two make the baseline identical to the graded run, which defeats the whole feature. The PR text admits the leg was never run live, andcli.pysteers users to this engine. The suite already fakesinspect_sweviasys.modules(tests/test_skillscope.py:3848), so a test that records theclaude_code(skills=...)kwargs is practicable.
Confidence 90 · logic · Fix: testinstall=Falseandinstall=True, and thatrun()forwardsinstall,run_key,epochsandmax_samples. -
The pairing keys never reach Task metadata in any test, and the test that claims to guard this cannot fail. —
skillscope/engine/behavioral.py:120-128,skillscope/engine/behavioral.py:626-656,skillscope/engine/verify.py:254-266,tests/test_skillscope.py:7245-7253
Droppingrun_keyfrom the metadata, droppingSKILL_KEY, or dropping the tags all stay green.test_the_other_engine_records_it_tooasserts"SKILL_DIR_KEY" in inspect.getsource(engine_verify.build_task). Deleting the real linebehavioral.SKILL_DIR_KEY: ...still passes, because the comment above it contains the string. It certifies a gap as covered, which is worse than having no test.
Confidence 95 · mechanical · Fix: build the task with a fakedinspect_sweand assert ontask.metadataand tags. -
The rebuild (
--from-logs) dies on one unreadable log. —skillscope/staleness.py:600-601
from_logsloopsfor info in list_eval_logs(log_dir): log = read_eval_log(info.name)with notry. The docstring promises logs that are not skillscope's own are passed over and that failing on a neighbour's file "would make this unusable". Pointing it at a directory with one truncated.evalraisedValueError: EOCD not foundand aborted.cmd_stalenessdoes not catch it. The loop also reads every log in full before checkingBASELINE_KEY, whileengine/behavioral.py:545already usesheader_only=Truefor this check.
Confidence 100 (ran it) · logic · Fix: read the header first, skip on any read error with a one-line note, and fully read only logs that carryBASELINE_KEY. -
A failed room walk is reported as a clean baseline, contradicting its own docstring. —
skillscope/engine/behavioral.py:373-375, docstring at:347-348
The code isexcept Exception: return fired, why or (UNKNOWN_ROOM if unknown else None). The docstring says a shape this cannot walk is "an unknown room, not a failed case". With known marks, an exception mid-walk discards the remaining calls and returnswhy=None, whichdirty_on_passreads as clean. A baseline that was not inspected is credited as clean, which is the dangerous direction for this feature. Dropping theUNKNOWN_ROOMfallback from theexceptbranch left the suite green.
Confidence 70 (the exception was not triggered with any shape tried) · logic · Fix: returnfired, why or UNKNOWN_ROOMon exception, and test it with a call that raises. -
Git-Bash-style host paths slip past the contamination check on Windows. —
skillscope/engine/behavioral.py:156-158,skillscope/engine/behavioral.py:170-214
With the recorded skill dirC:\Users\me\repo\skills\foo, all of these came out(None, None), meaning clean:cat /c/Users/me/repo/skills/foo/SKILL.md,ls /c/Users/me/repo/skills/foo,find /c/Users/me/repo -name SKILL.md.C:\...andC:/...are caught. On Windows, Claude Code's Bash tool is Git Bash, so/c/...is the natural spelling. A missed contamination turnsungradedinto a confidentno_signal.
Confidence 85 (ran it) · logic · Fix: have_normalisealso map/<drive>/to<drive>:/on Windows. -
Two report sentences state things the code did not establish. —
skillscope/staleness.py:932-951,skillscope/staleness.py:1111,skillscope/staleness.py:1150-1155
(a) "Some cases ran on one side only" fires for a case whose every graded attempt errored, becauseper_sideis built from graded attempts only. Ran:attempts_with=0, errored_with=1printed "ran on one side only" for a case that ran on both. (b)voided = [c for c in dirty if c["verdict"] == "ungraded"]prints "a baseline that reached another skill ... cannot show". Ran: a case ungraded because the graded side hitAPI Error 504, with a contaminated baseline, was blamed on contamination.
Confidence 100 (ran both) · logic · Fix: (a) countattempts + errored, or say "graded on one side only"; (b) record whether the contamination did the voiding, for examplevoided_by_contaminationonComparison. -
stalenessis documented as refusing Copilot "on the same terms", but it refuses it outright. —docs/usage.md:463-467againstskillscope/cli.py:975-991
The docs nameCOPILOT_GITHUB_TOKEN,GH_TOKENandGITHUB_TOKENand say the host leg "refuses astalenessrun on the same terms". The code refuses any--harnessother than the default ("has no staleness leg yet"), and I ran it and got that error. The code governs. The same limit is missing from the "still assume Claude Code" list indocs/harnesses.md:134-137.
Confidence 90 · mechanical · Fix: saystalenessis Claude Code only and add it to theharnesses.mdlist. -
Dead field
degradedin the rebuild. —skillscope/staleness.py:503-507
degraded=routing_core.is_provider_error(error)is never read instaleness.py; consumers exist only inbehavior.pyandrouting.py. The comment says leaving itFalsewould report a provider outage as a skill failure, but any outcome witherrorset is already excluded byanswered(). Mutating it toFalseleft every test green.
Confidence 85 · mechanical · Fix: remove the field and its comment from the rebuild. -
Untested behaviour the change introduces. — one finding for several gaps
Each of these was mutated toif Falseor the equivalent and stayed green:- The
outcomes_from_logwiring ofSKILL_DIR_KEY(staleness.py:474,520-522, mutated toNone; 192 selected tests stayed green).TestTheRoomIsJudgedWhereTheRunWasNotWhereTheReaderIscalls_inspect_roomdirectly and_StubLognever sets the key, although its docstring says it guards a rebuilt verdict. _mentionsbefore-boundary and the~expansion branch (behavioral.py:301-307), plus the first-reason-winsif why is Noneguard (:367).- The live-path
attempt=int(getattr(sample, "epoch", 1) or 1)(behavioral.py:433). Without it every repeated attempt would collide in_no_duplicates. - Render and summary branches: the
--max-casescaveat (staleness.py:971), the errored-attempts note (:1074), the unknown-room note andbaseline_unknown(:1094,:793), theguards_onlyplus dirty-baseline note (:1174), the "and N more" suffix (:1024), the\|escape (:904), and_why_dirty's known-beats-unknown rule (:374-375). - On the CLI side: the harness refusal (
test_the_default_harness_is_not_refuseduses--from-logs, which skips the guard),max_cases < 0,--from-logs --concurrency, theconfig_dirhanded to each half,baseline=not install, and theattemptspass-through toverify.run.
Confidence 100 (mutated) · mechanical · Fix: one small test per branch.
- The
Non-blocking
--from-logssilently accepts--model,--engine,--effortand--skip-preflight—skillscope/cli.py:1052-1075
Thespentlist refuses--attempts,--max-cases,--concurrency,--no-extendedand--harnesson the stated rule "refuse rather than ignore". Ran with those four flags: accepted with no complaint. The same flags are ignored at the live path's rebuild. Confidence 90 · mechanical · Fix: add them tospent.- Wrong error for a bad
--from-logsdirectory —skillscope/cli.py:1125-1146
An empty directory plus--skill fooreports "nothing in matches --skill foo". A nonexistent path reports the "no staleness logs, check STALENESS_RUN_KEY" explanation.--attempts 0 --from-logsgets a live-run message. All three were run. Confidence 90 · mechanical · Fix: checkis_dir()and "no logs" before narrowing, and validate--attemptsafter thefrom_logsbranch. - The free isolation refusal runs after the paid preflight —
skillscope/cli.py:1197,skillscope/cli.py:1214
_require_hook_supportis deliberately placed before the model probe because it costs nothing, butrouting.can_isolate_config()runs after_prepare_graded_run. Confidence 85 · logic · Fix: move it ahead of_prepare_graded_run. - README and
usage.mddisagree about CI —README.md
It says "four main types of tests" and thatskillscope-evals"will trigger all graders", whileusage.md:551andusage.md:405-407say no workflow runsstaleness. Confidence 70 · mechanical · Fix: say in the README thatstalenessis not in the workflow. - Doc and error text drift —
docs/usage.md:329-335,skillscope/cli.py:1062-1064
The--from-logsrefusal list omits--harness. The doc says plainbehaviorallogs "rebuild as nothing", but the code exits 1 with "no skillscope staleness logs". Thecli.pyerror callsbehavioral"harness-neutral" whiledocs/harnesses.md:136says it is refused on other harnesses. The key requirement is stated as unconditional inusage.md:262-278andREADME.md:45-54, butcli.py:1214checks only the host engine and runs after the "nothing gradable" return at:1203. Confidence 85 · mechanical · Fix: reconcile each against the code. - Table-cell injection —
skillscope/staleness.py:1311-1312
text[:70].replace('|', ...)does not strip newlines, andcase['id']is unescaped, although_detailstrips them. Ran: an expectation of"line1\nline2"split the row. Confidence 100 · mechanical · Fix: one escape helper for|,\nand\r. - Docstring and constant drift —
skillscope/staleness.py:18,skillscope/engine/behavioral.py:102-105
"Five things a case can turn out to be" whileVERDICTShas seven; thebuild_taskdocstring saysbaseline"only names the task" but it also stampsBASELINE_KEYand a tag. Confidence 85-100 · mechanical · Fix: correct both. - Comments narrate the history of code that never existed on
main—skillscope/staleness.py:419-423,566-574,917-921,1005-1009,1233-1235,1279-1284;skillscope/cli.py:1191-1194
"used to say" and a commit hash in a comment go stale after a squash merge. Confidence 85 · mechanical · Fix: state what the code does now and why. - Advice wrongly fires for an all-
fails_either_waysuite —skillscope/staleness.py:1050
"No case in this suite shows the skill doing anything ... stopped earning its place" also fires when every graded case failed on both sides, where the right advice is to fix the cases. Confidence 65 · logic · Fix: split the branch. - Pairing-evidence loss is not reported per skill —
skillscope/staleness.py:1014-1020
read = set(used.values())is global across skills, so one skill's skipped run is not listed if another skill read it. Confidence 60 · logic · Fix: list per(skill, run). - Partial epochs vanish —
skillscope/engine/behavioral.py:447-450
gradedis keyed on case id, so a case with--attempts 3and one cancelled epoch is reported as present. Confidence 55 · logic. - Smaller items —
_bundled_skills(behavioral.py:309-314) is a magic staging-directory name with no cited source and is global rather than per skill (confidence 70); the "no logs" message says both halves carrySTALENESS_RUN_KEYeven under--unlabelled-logs(60);createdtimestamps are compared as strings (60);UNKNOWN_ROOMis imported lazily in three places; thegetattr(args, "attempts", 1)fallbacks in_run_behavioraland_gradable_casesare a hidden contract with the other parsers' defaults (60); verdicts are bare strings, where aLiteralorEnumwould make a typo fail at definition rather than as aKeyError(70).
Decisions for the author
- The sample-to-outcome mapping is copied between the live path and the rebuild — non-blocking-improvement
staleness.py:456-532againstengine/behavioral.py:378-440. The comments keep saying "exactly what the live path sets", and a test (test_the_live_path_and_the_rebuild_word_it_the_same_way) pins the copy by grepping source text. Drift already exists: the live path turns a log with no samples into one failed outcome per case carryinglog.error.message, while the rebuild useslog.samples or []and yields nothing. A shared helper inengine/behavioral.pywould remove the copy and the grep test. stalenessexits 0 when every case errored — tradeoff
The description states this is deliberate ("No verdict fails a run"), and a nondeterministic agent justifies not failing on a verdict. The other side: a run where both halves hit infrastructure errors still exits 0 with an all-ungradedreport, whichrouting_gate-style callers would treat as failure. Confirm that is wanted, or return non-zero when zero case pairs were graded.--timeoutdefaults to 1800s, "twice the usual", while the docstring calls twice "the floor, not the figure" — tradeoff
On the measured case the baseline was still running past 30 minutes, so the default still truncates it. The parser description reads both "more than twice" and "defaults to twice". Truncating a baseline is cheaper than an open-ended run; the choice is yours, but the wording contradicts itself.contaminatedis also set on the graded leg's JSON — tradeoff
The description acknowledges this. The graded agent legitimately reads its own skill, so the field is routinely set in ordinarybehavioraloutput, and the name reads like a problem. Confirm the shape for external consumers.- The change bundles a feature, a reader (
--from-logs), contamination detection and two engine changes (27 commits, 4643 added lines) — non-blocking-improvement
It reviews as one coupled unit, but the live and rebuild paths and the engine metadata could have landed separately.
Positive signals
- Verdicts are asked over the whole side, not off one attempt. The attempt order cannot change the activation verdict.
_no_duplicateserrors both colliding logs instead of choosing one, and keys on(id, attempt)so--attemptsrepeats are not mistaken for collisions.- Contamination voiding is driven from
VOIDED_BY_CONTAMINATION, so the report text follows the rule. UNKNOWN_ROOMrefuses to claim a clean baseline where the room cannot be judged.- The default is threaded into
_add_timeout_argument, so--helpstates the value in force, and a test checks the rendered text. harness_solver(install=False)is tested by driving the solver and observing whetherinstall_skillis reached, not by reading a signature._gradable_casesis shared by both commands, and the cap applies once and per skill so both halves grade the same cases.- The
--harnessrefusal fails loudly rather than reporting a baseline it never checked.
Deployment notes
- No packaging, version or
action.ymlchange is needed. The version is0.1.3everywhere, the new module sits inside the package, and thepyproject.tomlchange is a comment only. behavioralJSON gains three per-case keys (activated,contaminated,attempt), and every behavioral and verify task now records skill, side and skill dir in log metadata and passesepochs(1 on an ordinary run). Additive, as the description says.behavioral.pynow walks the tool calls of every sample on all behavioral and verify runs, not only staleness runs. It is local work, and per the first Blocking item on the room walk it can return a wrong answer rather than failing.stalenessneedsANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKENon the host engine.
What this covered
Every changed file was read completely by four workers split by coupling: staleness.py with its tests; cli.py and its CLI tests; engine/behavioral.py, verify.py, no_sandbox.py, behavior.py and datasets.py with their tests; and the non-code files checked claim by claim against the code. The range is 7a8c00b9…4b99e511 (head 4b99e511ec38c4a9176c2731671bcc68201ca4af).
Scope was resolved from refs/remotes/pr/35, fetched with the recipe's flags into the shared clone's remote-tracking refs only, with the base read as origin/main after a fetch. The PR was open, and git log --left-right showed only this branch's commits (27 by one author, one merge from main). Workers read a temporary detached worktree outside the clone and made their own worktrees for runs and mutations. The review ran as a fan-out of five sonnet workers, one pass each, not as a single context.
Test verdicts:
- Revert verdict: with the production change reverted, the staleness, contamination, cost-control and
build_tasktests fail on import or attribute errors. Exceptions: the CLI-argument tests exercisecli.pyrather thanstaleness.py, andtest_the_other_engine_records_it_toopasses with its key line deleted (first two Blocking items). - Mutation verdict: roughly 50 single-branch mutations in total. The survivors are listed under Blocking. About 20 branches were killed by existing tests, among them verdict preference, the duplicate guard, the contamination rule per verdict,
--attemptsinspent, and theno_sandboxif install:gate. - Verdict basis: both were established by mutation in worker-owned worktrees, with
__pycache__cleared andPYTHONDONTWRITEBYTECODE=1set, not by literal revert.
Test runs:
- Windows ran 775 tests with one failure:
TestTheHostLegActuallyKillsTheProcess.test_the_work_after_the_decision_never_happens(1.41s against a 1.4s bound), and again alone (1.66s). This is the known flaky bound on that machine, unrelated to the change, and the PR does not touch the code it covers. It was not investigated further. - WSL Ubuntu ran the same 775 tests from a copy with all passing, including that test.
- Each run printed
skillscope.__file__and the path pointed into its own tree.
CI: on the head commit all 8 checks succeeded (Unused imports, unit tests on ubuntu and windows with Python 3.10 and 3.12, sandbox machinery on both, and the action check). The base-branch check-runs were all success, so no failing check needed the base exception. The /status surface was not read, and the rollup showed only CheckRun rows. The CI read did not run tests beyond that. statusCheckRollup rows were taken at face value.
Did not run: the history pass (staleness.py is a new file; the other workers' history reads found nothing relevant), the prior-changes pass (no earlier review was read, by design), and the agent-instruction adherence pass (the repository has no CLAUDE.md or AGENTS.md). A live staleness run was not made, so the API-key refusal, timeout and cost figures are traced rather than run. inspect_swe is not installed in the review environment, so what claude_code(skills=[]) does versus skills=None and whether _bundled_skills exists could not be checked. config.py, deadline.py and usage.py were read only where cli.py calls them. ai-instruction-quality was not activated because no AI instruction file is in the diff. .github/workflows and examples/ were grepped for staleness, not read.
Not reconciled against any discussion here. The description, and issue #6, were read only after the findings were settled.
This watch does not file change requests, so the blocking findings above are published as a comment-type review and gate nothing on their own.
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
4b99e51 to
af1b17f
Compare
The first real run of this command found something the report would not say. On `rocm-permission-denied-kfd` the unaided agent attempted a `sudo` write to /opt/rocm/share/rocdxg/dids.conf; the agent holding the skill kept to read-only diagnostics. Both sides failed the case. Every verdict was identical. The only difference between the two runs was the one the report did not mention. Guards were counted in one direction only, so a reader saw "none was broken with the skill installed" and learned that the skill did no harm, never that it stopped any. That is a whole class of a skill's value the pass/fail comparison cannot see: both sides can reach the same answer with only one of them wrecking something on the way. The data was already there -- both sides' guards are graded on every run -- so this is a second `_worst_guards` call, a count, and a sentence. A guard the baseline was never held to is not credit. The baseline has to have been asked the question and failed it; a case it never reached says nothing about what it would have done. Raised by siloteemu on #35, who said outcome comparison could not see this and was right. It turned out not to need the sandbox I claimed it would: the situation reproduced on its own, on the first skill we pointed this at.
|
Thanks both. Taking the scope question first, because it decided what I did next. Teemu, points 1, 2 and 6: conceded. One correction on 6: Point 3 was right, and it is now the headline. The 2m23s-against-30-minutes gap was sitting in the run and the report discarded it. The report now leads with what each half spent. Point 4 was right too, and I was wrong to call it out of scope. I said outcome comparison could not see prevented harm, and that reproducing a risky situation in an eval sandbox would be impractical. The first real run of the command disproved both. More below. Point 5 stands as a gap. Environment coverage needs a matrix we cannot stand up for evals today, and I would rather say so than oversell it after getting 4 wrong. What changed
On the other PRs: #26 turns out to be 90 commits behind main and edits functions that no longer exist, so I rebuilt its mechanism against current code rather than rebasing onto it. Volen, #39 adds a third instance of the same fault in its The run that changed my mind on point 4Two cases of
Both cases Then the part I had argued was out of reach. On Guards were counted in one direction only. A reader saw "none was broken with the skill installed" and learned that the skill did no harm, never that it stopped any. The data was already on the comparison, so crediting it cost about thirty lines. The report now names what the skill prevented. So on your point that harm is rare and hard to elicit: that is true, and it still showed up unprompted on the first skill we pointed this at. I would not claim that generalises from one run. On sizeRoman, you asked for a smaller core. You got a different one rather than a much smaller one, and I would rather say that plainly than let the diff imply otherwise. Removing the offline path took about 900 lines; the judge fix, the spend reporting and the per-case time added back about 550. I then went looking for padding in the report on the assumption it was bloated, and did not find it. Of its conditional notes, all but one either change how a number should be read or exist because a bug was found there. The single genuine duplicate was eleven lines. If the size is still wrong, the honest lever is scope rather than shaving, and the candidate is the contamination machinery at roughly 600 lines including tests. That buys a cheaper command that cannot vouch for its own baseline. I would rather not, but I would rather you decided than I did. Happy to take the call either way. |
Review — needs workFull review of the whole change (17 files, +4389/-86), base Blocking
Non-blocking
Decisions for the author
Positive signals
Deployment notesOpt-in only: What this coveredThe full diff across all 17 files, read completely, split by coupling across three independent fan-out passes: (1) One process note, for transparency: the three fan-out passes above were genuinely independent and blind (none read this PR's discussion; each says so). Consolidating their results hit a snag — an intermediate orchestrating step received a relayed summary of the three passes' conclusions and, unable to independently confirm that relay's authenticity, correctly declined to act on it rather than rubber-stamp unverified findings. This final report was produced by reading the three passes' own original output directly and consolidating it from there, rather than through that intermediate relay, so the findings above rest on the fan-out passes' firsthand reports, not on anything secondhand. Did not run: agent-instruction adherence (no 🤖 Automated review by an agent. |
johnl-amd
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head e4c71b62
Review — needs work
Full review of the whole change (18 files, +4664/-89, merge-base 7a8c00b9…head). The review was carried out against commit 1fb1024f; while consolidating, the author pushed one further commit, e4c71b62 ("Build the room-check test's paths the way Windows spells them"), which turned out to fix exactly one of this review's own findings (see Blocking #1) and touches nothing else. All other findings were re-confirmed against the actual current head, e4c71b6, which is what this report and its line numbers refer to.
This implements issue #6 ("Add skill staleness test") in full — there is no partial-implementation gap against that ticket.
Blocking
-
The PR description's verdict table no longer matches the shipped code. The description lists seven verdicts (
earns_it,harmful,unattributed,no_signal,fails_either_way,guards_only,ungraded), three of which (unattributed,fails_either_way,guards_only) do not exist anywhere in the code.skillscope/staleness.py:87-91defines exactly four:earns_it,harmful,no_signal,ungraded. The repo's own commit history on this branch shows why: a later commit on this same PR, titled "Collapse seven verdicts into three and an error state," intentionally merged the seven down to four — the description was never updated to match. A reader deciding what to test or review from the description alone would be checking for distinctions the code doesn't make.
Confidence 85 · architectural · Fix: rewrite the verdict table (and the "unattributed"/"fails_either_way" mentions elsewhere in the description) to match the current four-verdict design, or restore the richer taxonomy if it was dropped unintentionally. -
_ancestors()'s path-depth threshold produces zero ancestors for the staged skill mark, silently disabling sweep-based contamination detection on the default (and only registered) harness.skillscope/engine/behavioral.py:203-225(_ancestors), consumed by_swept(184-200) and_skill_marks(237-281) — all new in this PR. The staged mark built in_skill_marksisf"{harnesses.get().skills_dir}/{skill}", i.e..claude/skills/<name>for the one registered harness — exactly 3 segments after.split("/")._ancestors's loop (while len(parts) > 3: ...) therefore never executes for this mark and contributes no ancestors at all. Reproduced directly against the production code and the project's own test fixtures:find .claude -name SKILL.md -exec cat {} +,grep -r rocm .claude/skills/, andls .claude/skills/all come back as a clean room (None) instead of the documentedUNKNOWN_ROOM, even though the surrounding docstrings explicitly namefind/grep -r/a bare sweep as the realistic way an unguided agent looks for something. This is the exact mechanismskillscope staleness's baseline-integrity claim rests on: a baseline workspace with leftover/contaminated.claude/skillscontent (e.g. from incomplete cleanup between cases) would be silently credited as empty, which can turn a would-beungraded/no_signal/harmfulcase into a falseearns_it/no_signal. The existing regression tests (TestAnAgentThatWentLookingIsNotACleanRoom) only exercise sweeps against the absolute "home" mark, which is many segments deep and never hits this edge, so they didn't catch it.
Confidence 85 · logic · Fix: size the stop threshold relative to each mark's own depth (e.g., stop one segment short of the mark's own directory rather than a fixed length-3 floor calibrated for absolute POSIX paths), and add a test that sweeps the staged/relative mark specifically. -
A test documented as token/network-free makes a real, unmocked call into model/API reachability checks.
tests/test_skillscope.py:7156-7164,test_the_default_harness_is_not_refusedinTestAFlagThatCannotDoAnythingIsRefused. It callscli.cmd_staleness(parser.parse_args(["staleness", "--harness", "claude-code"]))with no--skip-preflightand nothing mocked. Tracing the call:cli.py:1022reaches_prepare_graded_run(cli.py:464-539), and sinceDEFAULT_ENGINE = "claude-code-no-sandbox"(cli.py:127) andargs.skip_preflightis false by default, execution unconditionally reachesengine_models.check_reachable(...)(cli.py:524) and thencheck_api_reachable(...)(cli.py:531, which spawns the realclaudeCLI) whenever Claude Code credentials are present in the environment the suite runs in — a realistic state on a developer machine or a credentialed CI runner, and the actual state of the sandbox this review ran in. Every sibling test in the same class is refused bySystemExitbefore ever reaching_prepare_graded_run; only this one proceeds into it. The test file's own stated contract for this class of test is "no agent, no tokens, no network."
Confidence 80 · mechanical · Fix: mock_prepare_graded_run/engine_models.check_reachable/check_api_reachableas other tests in the file do, or pass--skip-preflight. -
The new
attemptfield's only production code path (deriving it fromsample.epoch) has no test coverage; the fixture built for exactly that purpose is unused.skillscope/engine/behavioral.py:525(attempt=int(getattr(sample, "epoch", 1) or 1), new this PR);tests/test_skillscope.py:6439-6448(_StubSample, whose own docstring states its purpose is to test this)._StubSampleis defined once and never instantiated or passed to_outcomes()anywhere in the file; every other "multi-attempt" test constructsBehaviorOutcomeobjects directly withattempt=i, bypassing_outcomes()entirely, and no test anywhere asserts on.attempt. The line that turns an inspect sample's epoch into the field that letsstalenesstell repeat attempts of the same case apart could be deleted, hard-coded to1, or read the wrong attribute, and the whole suite would stay green.
Confidence 75 · test-coverage · Fix: instantiate_StubSamplewith varyingepochthrough_outcomes()and assert the resultingattemptvalues, or remove the unused fixture.
Non-blocking
staleness's JSON report omits theharness/harness_versionmeta keys thatroutingandbehavioralreports always carry —skillscope/cli.py:1088-1111builds the staleness meta dict by hand instead of splicing**_leg_meta(args)(cli.py:190-202) the waycmd_routing(cli.py:636) andcmd_behavioral(cli.py:946) do. Harmless today sincestalenessis restricted to one harness, so the value can't currently be wrong, but it's a schema gap across three report types this PR's own description treats as siblings, and would become a real correctness gap ifstalenessever gains a second harness leg.
Confidence 70 · mechanical · Fix: splice**_leg_meta(args)into the staleness report meta like the other two commands.
Decisions for the author
--attempts N's "any attempt passed" rule has a real, derivable asymptotic bias — tradeoff. As N grows, any case where the baseline has even a small nonzero chance of an accidental pass converges towardno_signaland away fromearns_it, for purely statistical reasons unrelated to whether the skill still matters (the same holds forharmfulrequiring the graded side to never pass). The module's docstring and the rendered report already disclose the mechanism correctly and concretely (the worked "1/3 baseline pass takes the case tono_signal" example), but frame raising--attemptsas "a higher bar" / "a stronger claim" — language that reads as increased rigor rather than a one-directional statistical pull towardno_signal. Worth naming the direction explicitly so a user raising--attemptsto gain confidence in a positive finding doesn't read the result backwards.- The baseline's "empty room" guarantee is log-based heuristic detection, not real isolation, on the default engine — tradeoff. On
claude-code-no-sandbox(the default forstaleness), the agent runs on the host with the skill's real directory fully present and reachable the whole time; the code's own docstrings already acknowledge this ("nothing stops it naming an absolute path... the sandboxedclaude-codeleg does close it"). Blind spots beyond what's already documented include indirect access (MCP resource reads, env-var-sourced paths, runtime string concatenation) and, most structurally, a model's own memorized/trained knowledge of a public skill's contents, which leaves no tool-call evidence at all — not even the fallbackUNKNOWN_ROOM. This reads as a reasonable, unusually well-documented best-effort design rather than a fundamental flaw, but worth the author explicitly confirming (perhaps in the command's own help text or report output, not just in source comments) that the no-sandbox baseline's "clean room" claim is log-based only, and considering whether the sandboxedclaude-codeengine should be the recommended choice specifically forstaleness, given that command's integrity depends on room emptiness more than any other.
Positive signals
- The contract between
staleness.py'sComparisonfields and its verdict/report logic is unusually well specified — nearly every field and helper's docstring states the exact edge case it exists to handle, and in every instance checked, the implementation matched the stated contract. usage.leg()'s design (swapping the global usage accumulator rather than snapshot-and-subtract) correctly sidesteps a real bug class its own docstring names — snapshot-and-subtract would have reported every leg as having spent nothing, a plausible-looking zero rather than a loud error.- The CI failure this review independently found (the Windows-only test-path bug in
TestAnAgentThatWentLookingIsNotACleanRoom) was fixed within about 20 minutes of this review starting, correctly diagnosed down to the exact root cause (rsplit("/")on a native OS path) and fixed with a portablePath-based rewrite rather than a platform-specific patch.
Deployment notes
stalenessis opt-in: it is absent fromselection.plan()and from this repo's ownselftest.yml, and is only exposed as an additionalaction.ymlcommand input for downstream consumers to invoke explicitly. Nothing in this repo's own CI cost or runtime changes as a result of this merge.behavioral's JSON output gains three additive per-case keys (activated,contaminated,attempt); existing consumers reading by name are unaffected, but a consumer diffing whole JSON blobs will see new keys appear on ordinarybehavioralruns, not juststalenessones.
What this covered
Every file in the change read completely: skillscope/staleness.py (new, 1232 lines), skillscope/behavior.py, skillscope/cli.py, skillscope/usage.py, skillscope/engine/behavioral.py, skillscope/engine/verify.py, skillscope/engine/no_sandbox.py, skillscope/engine/judge.py, skillscope/engine/scorers.py, skillscope/datasets.py, and the full diff of tests/test_skillscope.py (~2200 new/changed lines), plus action.yml, .github/workflows/selftest.yml, README.md, docs/usage.md, docs/harnesses.md, docs/authoring-evals.md, and pyproject.toml, against 7a8c00b9…e4c71b62. Four independent code-review passes ran in parallel, grouped by coupling (core module; CLI/usage/docs wiring; engine internals; test suite), followed by two independent design-review passes on specific strategic questions, then consolidation and adversarial re-verification of every surviving finding against the actual source (including direct reproduction for the two highest-confidence findings).
Did not run: a "prior changes" pass against earlier PRs' review comments on these files (not performed, for effort reasons — git history was read instead, which surfaced the relevant in-branch context directly). No repo-root CLAUDE.md/AGENTS.md exists, so agent-instruction adherence had nothing to check. CI's check-run results were read directly from GitHub and taken at face value; all 8 checks are green on the current head.
Not reconciled against any existing discussion, review comments, or threads on this PR — none were read, per this review's independence requirement.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Review — needs workFull review of the whole change. Blocking
Non-blocking
Decisions for the author
Positive signals
Deployment notes
What this coveredEvery file changed by the PR, read completely at both Did not run: agent-instruction adherence — no Not reconciled against any discussion on this PR — this review was conducted blind to the PR's comments and did not read them at any point. 🤖 Automated review by an agent. |
…need it `skillscope behavioral` tells you a skill's cases pass. It cannot tell you they pass *because of* the skill. A case any competent agent satisfies stays green forever whatever happens to the skill, and the report reads as a measurement of the skill when it is a measurement of the model. `skillscope staleness` runs a skill's existing behavioral cases twice, once with the skill installed and once with nothing installed, and puts the two runs side by side. It reads the same `evals.json`, so there is no new file to write and no new assertion to learn. Each case comes back as `earns_it`, `harmful`, `unattributed`, `no_signal`, `fails_either_way`, `guards_only` or `ungraded`. Installing a skill does not make an agent use it, so the graded side also records whether the agent reached for it: a case that moved without the skill firing is `unattributed`, not `earns_it`. `--attempts N` runs each case N times a side, and a side counts as passed if any attempt did, which makes a higher N a harder question rather than a steadier reading of the same one. The baseline is only a measurement if the room was empty, so both halves run against a throwaway `CLAUDE_CONFIG_DIR`, one per half. That works only when auth comes from the environment, so the host engine refuses to start without `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`, as routing does. An agent reading the skill's source off disk is detected too, and a dirty baseline voids only what it refutes: `no_signal` and `harmful` claim the baseline managed something unaided, so contamination kills them, while `earns_it` and `fails_either_way` rest on the baseline having failed and survive it. A finished run can be read again for free: both halves record which side they were and which invocation wrote them, so `--from-logs` rebuilds the comparison without reaching a model. No verdict fails a run. Two runs of a nondeterministic agent disagree with themselves often enough that a verdict is somewhere to look, not a build failure. An expired deadline or a refused run still exits non-zero. `staleness` is absent from `selection.plan()` and from every workflow, so nothing new runs anywhere and no downstream repo starts running it as a result of this merge. `behavioral`'s JSON gains three additive per-case keys (`activated`, `contaminated`, `attempt`), and behavioral and verify tasks now record which skill and side they were about. Closes #6.
`judge.grade` returned `(False, reason)` when the sandbox would not list and when the grader answered in prose. Both are statements about the run, and both arrived as a check that simply failed. In `behavioral` that is a red case. In `staleness` it is the headline. `answered()` keeps any attempt without an `error`, so a baseline whose judge broke counted as a baseline that genuinely failed, which is `earns_it`, and a graded run whose judge broke counted as `harmful`. The two verdicts the command exists to produce were the two a grader hiccup manufactured. The polarity is where it bites hardest: on `must_happen=False` a `False` asserts the forbidden thing happened, so a grader that rambled was reported as the agent having done it. `judge.Ungradable` carries the distinction now. An exception rather than a changed return type, because the caller has to do something different with it and a `bool | None` invites being read as falsy by every caller that does not know yet. Checks gain a `graded` flag, read through `scorers.graded()` so a dict from a log written before this still reads correctly, and one ungraded check errors the attempt: a side passes only by meeting every positive claim, so a claim with no verdict leaves the side undecidable rather than negative. `staleness.answered()` needs no change. It already drops attempts with an error, which is why that was the channel to use.
`--from-logs`, `--run-id` and `--unlabelled-logs` rebuilt a comparison from logs already on disk. The feature is sound and will return, but it arrived welded to the command it reads for, and reviewers asked for the smaller thing first: one PR that runs two halves and reports what differed. Most of the complexity it carried was its own. A log directory is shared with `behavioral` and `routing`, is never pruned, and holds every run anybody has done in it, so the rebuild needed a run id to pair on, a rule for picking one run per skill, a flag for logs written before the id existed, and a way to report a side it could not answer for. None of that is needed to run two halves and compare them. `STALENESS_RUN_KEY` goes with it. Nothing read it once `outcomes_from_log` was gone, and a key written for a reader that does not exist is the kind of dead weight this round of review was about. Removing it also narrows the `build_task`/`run` keyword surface that #39 touches. `test_the_default_harness_is_not_refused` is fixed rather than deleted. It passed `--from-logs`, which returned before the harness guard was reached, so it asserted the absence of a refusal that could not have fired and would have gone on passing had the guard been written to refuse the default too. staleness.py 1314 -> 975 lines.
Seven verdicts was more than a pair of runs can establish. Three is what the comparison actually answers: the two sides differed in the skill's favour, they differed against it, or they did not differ. `ungraded` stays as the error state. The four that go are not findings that go. Each was one of these three with a caveat attached, and each caveat is still reported, next to the evidence for it rather than as a row in a table nobody can hold seven of: `unattributed` becomes `fired_with` on the comparison plus a note above the case table. The honest reading is that the outcome *did* differ, so the case is `earns_it`; what is unknown is whether the skill is why. That counts it in the headline share where it used to be excluded, so the note that discounts it has to come before the reader reaches the table, and a test pins that order. `fails_either_way` and `guards_only` become `no_signal` plus advice. This is the part that needed care: `no_signal` now covers three situations wanting three different actions, and its old single sentence, "the agent already did this unaided", is false for two of them. `_advice` branches on the pass counts, which are printed in the column alongside, so a reader can check the advice against its evidence. A case that claims nothing positive is asked about first, because it cannot pass and would otherwise be told it is broken when it is working exactly as written. Contamination voiding is unchanged and the merge is safe for free, which is worth saying because it is not obvious: both-sides-passed rests on the baseline passing and is voided, both-sides-failed rests on a baseline that passed nothing, and `dirty_on_pass` only fires on an attempt that passed, so it is False there and the verdict stands. Guard rails keep their reporting. Keeping them out of `POSITIVE_KINDS` is what makes the three verdicts correct at all: a well-written guard holds either way, so counting guards in the verdict would land every guard-bearing case in `no_signal`. Two tests are now about the property they are named for. The guard-parity test varied the positive claim, which only showed that the claim decides the verdict and would pass under an implementation that folded guards in too; it now holds the claim constant and varies the guard across all four states.
The command measured the thing both reviewers said was its strongest signal and then threw it away. On a two-case skill the graded half finished in 2m23s while the baseline was still going half an hour later, and the report said nothing about it: both halves' cases passed, so every verdict read `no_signal`. The verdicts cannot express this. A skill whose cases pass either way can still be the difference between two minutes and half an hour, and that is worth knowing whatever the table says. So the report now leads with it. Billing a half needs a scope rather than a subtraction. `usage.snapshot()` returns the module global itself rather than a copy, so the obvious idiom -- hold a snapshot, subtract afterwards -- subtracts the totals from themselves and reports both halves as free. That produces plausible zeros rather than an error, which is the kind of bug that survives a reading. `usage.leg()` swaps the global instead and folds it back on the way out, so engines recording from deep inside the call reach the right half without knowing a half exists, and `snapshot()` still answers for the whole command. Every other report's meta is unchanged. Nothing is added to `_run_behavioral`, deliberately: #39 expands that same block in `cmd_behavioral`, and this work sits entirely in `cmd_staleness`, which #39 does not touch. Cost will usually read `n/a`, and the caveat says why rather than letting a blank pass for a zero. Dollars arrive with a run's final event, which a half the bound stopped never reaches -- so the half that spent *more* is the likelier one to show nothing, and a reader taking that for zero reads the result backwards. Tokens and wall clock are reliable; a stopped half's row carries a `+` and is named as a floor. A graded half slower than its baseline is printed plainly rather than suppressed. A skill that costs the agent time is something its author wants to know.
The per-half table says the baseline took twelve times as long. It does not say which cases that went on, which is what an author needs to act. Summed over the attempts that answered, rather than averaged or taken from the decisive attempt. The column has to reconcile with the half-level total above it, and a mean adds up to nothing a reader can check; `_decisive` would be worse, since it prefers an attempt that passed and so selects on the dimension that correlates with speed. `None` rather than `0.0` where a side has no measurement, and that is the whole care in this change. A side whose attempts all errored did not take zero seconds trying: it is the half-hour baseline that never concluded, and rendering it `0s` next to the graded half's `2m23s` states the exact opposite of the finding. The cell says `did not finish`, or `not run` where the case never reached that side at all.
The verdict table in `usage.md` listed seven and described a report that no longer exists. The spend comparison is now what the section leads with, since that is what the report leads with and the reason the command is worth running on a suite whose cases all pass. The `no_signal` section says all three findings it now covers, because the old text named only one of them and that one is false for the other two.
The first real run of this command found something the report would not say. On `rocm-permission-denied-kfd` the unaided agent attempted a `sudo` write to /opt/rocm/share/rocdxg/dids.conf; the agent holding the skill kept to read-only diagnostics. Both sides failed the case. Every verdict was identical. The only difference between the two runs was the one the report did not mention. Guards were counted in one direction only, so a reader saw "none was broken with the skill installed" and learned that the skill did no harm, never that it stopped any. That is a whole class of a skill's value the pass/fail comparison cannot see: both sides can reach the same answer with only one of them wrecking something on the way. The data was already there -- both sides' guards are graded on every run -- so this is a second `_worst_guards` call, a count, and a sentence. A guard the baseline was never held to is not credit. The baseline has to have been asked the question and failed it; a case it never reached says nothing about what it would have done. Raised by siloteemu on #35, who said outcome comparison could not see this and was right. It turned out not to need the sandbox I claimed it would: the situation reproduced on its own, on the first skill we pointed this at.
The report carried a paragraph counting `no_signal` cases where a side disagreed with itself, directly above a table whose advice column tells the same reader the same thing on the row they are looking at. The paragraph went; the test moved to the row, which is the better place to assert it. The contamination note keeps its three groups and loses a third of its words. Each group still rests on something different -- a voided verdict claimed the baseline managed something, a standing one rests on the baseline having failed, and the third is dirty on an attempt it failed and clean on the one it passed -- so none of them could be merged without saying something false about one of them. This is a smaller cut than it was meant to be. Going through the report block by block, nearly everything in it either changes how a number should be read or was put there by a bug somebody found. The one genuine duplicate was eleven lines.
**The contamination check failed open on the shapes an agent actually reaches for.** It only recognised a tool call that spelled the skill's own directory, so `find <skills> -name SKILL.md`, `grep -r x <skills>/`, `cat <skills>/*/SKILL.md` and a `..` traversal all read the skill and were reported as a clean room. An agent told nothing and looking for something uses exactly those, which made the miss likeliest on the baseline, the one side whose emptiness this command rests on, and it failed in the dangerous direction: `no_signal` and `harmful` stood where they should have been voided. A `..` traversal is now collapsed textually, which needs no cwd and so is safe here, and names the skill outright. A sweep of a directory the skill sits under is reported as an unknown room instead: the call would take the skill with it and the record cannot prove it did. The ancestor chain stops short of the top level, because a skill checked out under `/tmp/<x>` would otherwise make every `ls /tmp/...` in the run unvouchable. **`_met`'s conjunction was the module's central rule and nothing pinned it.** `all` could become `any` with the whole suite green: a case asserting three things and meeting one would have read as a side that passed, straight into `earns_it`. Every asymmetric test used a single claim, and the only multi-claim tests gave both sides the same pattern, where the two operators cannot disagree. **The scorer glue that makes `judge.Ungradable` mean anything never ran.** `scorers.expectations()` was called by no test, so `graded=False` could be flipped, or the key dropped from `_check` entirely, and nothing failed. The raise was tested and the hand-built dict was tested; the step between them was not. **An ungraded case was choosing the headline.** A suite whose only graded case failed both ways, beside an ungraded case whose graded half happened to pass, was told the skill had stopped earning its place while the row underneath said the case was simply broken. `F841` joins `F401` in CI. It would have caught the dead local this commit removes, left behind when an earlier commit deleted the report section that read it. Every test here was checked against the unfixed code first.
The new contamination tests split the skill's path on "/" to reach its parent directory. Windows hands back `C:\...\run`, where that split finds no separator and returns the whole path, so one test raised and three asserted against the skill's own directory instead of the directory above it. The code under test normalises separators itself and was right on both runners; only the test assumed POSIX. `Path.parent` asks the question in a way that holds either way. Confirmed by running a Windows-shaped path through `_normalise`, `_ancestors` and `_swept` directly: the ancestor chain comes out at the same depth as the POSIX one, because the leading empty segment on POSIX and the drive letter on Windows occupy the same slot.
**The ancestor chain never ran for the staged skill.** `_ancestors` stopped above three segments, a floor calibrated for the absolute mark. The staged one is `<skills_dir>/<name>` -- `.claude/skills/<name>`, relative, exactly three -- so it contributed no ancestors at all and every sweep of the staged directory read as a clean room. Ran against the production code: `ls .claude/skills/`, `find .claude -name SKILL.md` and `grep -r x .claude/skills/` all came back `None`. The floor is now a property of the mark. An absolute path spends a segment on the empty string before the root and stops at `/home/u`; the staged mark keeps every segment, because it is relative to a workspace made for one sample and each of its segments is the harness's own. `/home` holds every user on the machine, `.claude` holds skills and nothing else of ours. **A `no_signal` report told a baseline it had passed when it had passed nothing.** `no_signal` is two findings under one name, and the both-sides- failed one reaches the contamination paragraph too: `dirty_on_pass` fires only on an attempt that passed, so a baseline dirty on the one attempt it had, and failing it, is never voided and landed in the group whose sentence reads "the baseline passed on an attempt that was clean and was dirty on one it failed". It passed on no attempt at all. Split on `passes_without`, which counts on the same basis `dirty_on_pass` reads. **A staleness report was the one graded report that did not name its harness.** `cmd_staleness` hand-wrote its meta where `cmd_behavioral` and `_finish_routing` splice `_leg_meta`, so it carried no `harness` or `harness_version` and spelled the engine without `_engine_label`'s normalisation. **`-R ` in `_SWEEPS` was dead the day it was written.** Every caller passes `_normalise(...)`, which lowercases, so only `-r ` could ever match. **A test that starts no agent started one.** `test_the_default_harness_is_ not_refused` passes `--harness claude-code`, which by design does not hit the refusal -- and the next thing `cmd_staleness` does is `_prepare_graded_ run`, which probes model reachability and spawns the real `claude` CLI whenever credentials are in the environment. Its class is meant to touch no network and every sibling test is refused before that call. Stopped at the preflight instead, which proves the guard did not fire *and* that execution reached it. **`attempt` was derived in one line nothing covered.** `_StubSample` was built for exactly that and never instantiated; every multi-attempt test constructs outcomes by hand with `attempt=i`, so the line could have been hard-coded to 1 with the suite green -- and collapsing it collides every repeat in `_no_duplicates`. Each fix is pinned by a test that fails with the fix reverted: the floor mutated back to a constant 3, the split dropped, `_leg_meta` unspliced, the guard widened to refuse the default, and `attempt` hard-coded to 1. 780 tests pass, ruff `F401,F841` clean.
4dfcc8b to
3f0dc36
Compare
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · 3f0dc36
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
Implements #6 ("Add skill staleness test") in full; the description says Closes #6.
Blocking
-
(new) The baseline contamination check misses common recursive and path spellings, so it reports a clean room in the dangerous direction —
skillscope/engine/behavioral.py:184,:201,:154-175,:394
_SWEEPSmatches only the literal"-r ";_read_the_skillexpands~only when the whole argument has no whitespace;_normalisedoes not collapse//or/./. Probed against the PR code, each of these returnedNone(clean room):grep -ri foo /home/u/dev/proj/,cat ~/dev/proj/skills/run/SKILL.md,$HOME/…,dev//proj,proj/./skills. By reading,-rn,-Rn,rg,treeandfdmiss the same way. This is the failure classa3e3d09set out to close on "the one side whose emptiness this command rests on" — a baseline that read the skill still yieldsno_signal/harmfulinstead of being voided.
Confidence 100 · logic · Fix: match recursive flags by regex (e.g.\s-\w*[rR]\w*\s), addrg/tree/fd, expand~/and$HOMEinside command strings, collapse//and/./in_normalise. -
(new) New behaviour that no test catches when broken —
behavioral.py(_inspect_room,_read_the_skill),engine/scorers.py(files_exist, judge polarity),staleness.py(_why_dirty,_decisive,_met,_seconds),cli.py:741-860
Each mutation below was applied alone and the suite stayed green:_inspect_roomletting a laterother:hit overwritewhy, andelif hit and fired is None→elif hit(both bugs this change's docstring says it fixed); the~expansion branch disabled;files_existlist-failuregraded=False→True; judge hard-wired tomust_happen=True;_why_dirty's known-over-UNKNOWN_ROOMpreference removed;_decisive's fired-attempt preference removed;_metaccepting empty claims;_secondskeeping zero elapsed.cmd_behavioraland_run_behavioral's new kwargs (install,attempts,concurrency,baseline,config_dir) are reached only by source-substring checks (tests/test_skillscope.py:2398-2406) and a mock (:6348).
Confidence 100 · logic · Fix: one targeted test per branch; thefiles_existand polarity ones guard the exact misattribution this PR fixes.
Non-blocking
- (new) One ungraded check turns a decided failure into an error —
behavioral.py:522-533
With one check graded-False and another ungraded, the attempt becomes an error andstaleness.answereddrops it. The comment's own premise ("a side passes only by meeting every positive claim") makes that attempt decidably failing. Confidence 85 · logic · Fix: error only when no graded check failed. - (new) Sweep detection is over-broad —
behavioral.py:184,:206-251
*,?,lsmatch as substrings of any string argument, and.claudeis an ancestor at floor 1, soecho "see ~/.claude docs *"(probed) voids the baseline; so would a Write of Markdown mentioning.claudewith a?. Untested in this direction. Confidence 85 · logic · Fix: apply sweeps to command-like fields only, or require token and ancestor in the same shell word. - (new) A sweep is reported with a false reason —
behavioral.py:257,:412
_sweptreturnsUNKNOWN_ROOM("the run did not record where the skill was, and it is not on this machine") when the location is known;staleness.pyrenders and compares that string as a sentinel. Confidence 85 · mechanical · Fix: a distinct reason for sweeps. - (new) Stale comment on
~expansion —behavioral.py:384-397
Says$HOMEis "the agent's own, recorded in the environment";Path.expanduser()uses the reader's HOME. Confidence 85 · mechanical. - (new) Docs say
staleness"refuses--harnessoutright"; it accepts--harness claude-code—README.md:57,docs/usage.md:451,docs/harnesses.md:151vscli.py:991-1008
The code governs:test_the_default_harness_is_not_refusedpins it. Confidence 85 · mechanical · Fix: "refuses any harness other than claude-code". - (new) No-sandbox credential refusal runs after the preflight probes —
cli.py:1022vs:1039
A run that will be refused first pays the model/API reachability probes; the check is pure environment. Confidence 85 · mechanical · Fix: move it beside the--harness/--attemptschecks. - (new) Ungraded cases are never named in the Markdown report —
staleness.py:1178-1217
Only counted; with all graded casesearns_itthe report says "None. Every graded case needed the skill…" while errored cases go unnamed. Confidence 70 · logic. - (new) Hard-coded commit hash in a comment —
staleness.py:1188
"the same bug commit542cad9fixed" goes stale on squash/rebase. Confidence 70 · mechanical.
Decisions for the author
SKILL_DIR_KEYis written to every log but nothing in production reads it — answered-by-intent
Written atbehavioral.py:124,verify.py:258;_outcomes(:535) never passes it, only tests do. Description: "inspect logs gain the skill, the side and the skill's path" and "--from-logs,--run-idand--unlabelled-logs… will come back on their own." Would otherwise be blocking as dead code. The module comment at:585-596and the_inspect_roomdocstring describe a reader that does not exist yet; confirm that is intended until the rebuild lands.stalenessexits 0 when nothing was graded — tradeoff
routing_gatefails a run withgraded == 0as infrastructure; staleness exits 0 even when every attempt errored, so a CI step stays green. "A verdict never fails a run" does not obviously cover "nothing was measured".--attempts/--max-cases/--concurrencyexist only onstaleness— non-blocking-improvement
_gradable_cases/_run_behavioralread them viagetattrdefaults while comments call them "shared so behavioral and staleness cannot drift";behavioralcannot use them. Expose them there or pass explicitly.- 4,800 lines in one PR — non-blocking-improvement
The judge/Ungradablefix and theusage.legaccounting stand on their own and could have landed separately.
Positive signals
judge.Ungradable→graded=False→ error keeps infrastructure faults out of pass/fail, andgraded()defaulting to True keeps old logs readable.usage.legswaps the global accumulator instead of subtracting snapshots, with tests pinning the snapshot-inside-a-leg and leg-raises cases._gradable_casesapplies--max-casesonce, so both halves provably grade the same cases; a test asserts they differ only ininstall.
Deployment notes
- CI gains a ruff
F841gate, which may fail other open PRs with unused locals. - New
stalenesssubcommand; on the default engine it needsANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKEN; Claude Code only. behavioralJSON and inspect logs gain fields (additive).
What this covered
All 18 files at 0a25c86…3f0dc36, read completely, as five independent workers: staleness feature; grading engine; CLI/usage/CI/docs; tests incl. mutation runs; history, prior-PR review comments (#34, #33, #19, #18) and code comments. Workers read the PR head from a detached temporary worktree; running and mutating happened only in per-worker worktrees, all removed.
Test verdicts: CI rollup — all 8 checks SUCCESS. Full suite python -m unittest discover 781 OK locally (after moving TMPDIR off a noexec /tmp and isolating global git config; TestWhatABranchChanged, pre-existing, is not hermetic against commit.gpgsign=true). The new tests fail with the change reverted; 15 of 24 single-branch mutations went red, the survivors are the second blocking finding.
Design questions and final verification were done by the driver, not independent workers. Fixes in this round were written by the PR author. Not reconciled against the PR discussion.
Two blocking findings from the third review pass. Both are about the same
thing from opposite ends: the baseline's emptiness is the whole measurement,
and neither the check that establishes it nor the branches around it were
holding.
**The contamination check matched the tidy spelling of each shape only.**
Probed against the code, each of these came back a clean room: `grep -ri foo
<above>/`, `grep -rn`, `ls -lR`, `rg`, `tree`, `fd`, `cat ~/<path>`, `cat
$HOME/<path>`, `<a>//<b>` and `<a>/./<b>`.
`-r ` was a literal, and a flag an agent almost never types alone. It is now
any single-dash cluster containing `r`, which is what `-ri`, `-rn`, `-nr` and
`-lR` all are; `rg`, `tree`, `fd` and `grep` join `find` and `ls`, and the
command words are matched on a word boundary rather than as substrings, so a
sweep is a command somebody ran rather than two letters inside another word.
`~` was expanded only for an argument that was a bare path with no whitespace
in it -- and a Bash command line is never that, which is the only place an
agent types one. `~/`, `$HOME/` and `${HOME}/` are now written out wherever
they appear, anchored so they only fire where a path can start. The expansion
resolves against this process's home, which is the agent's own on the live
path and a guess on a rebuild; the comment claiming otherwise is corrected
rather than left standing.
`_normalise` collapsed `/../` but not `//` or `/./`, which a shell joining
two pieces of a path produces without anybody typing them. All three are now
collapsed, along with a trailing separator, which costs nothing because
`_mentions` already treats the end of the string as a boundary.
**The branches nothing would have caught.** Each of these was mutated alone
and the suite stayed green; each now has a test that fails when it is:
- `_inspect_room`'s two guards -- a later `other:` hit overwriting the first
reason, and a later firing replacing the first. These are the exact bug the
function's own docstring says it fixed, and no test ever walked more than
one activating call.
- `scorers.files_exist`'s `graded=False` on a sandbox that would not list.
Flipping it to `True` turns a harness failure back into the agent's, and a
baseline whose sandbox would not list manufactures `earns_it`.
- the judge's `must_happen` polarity. Hard-wired to `True`, every
`unexpected_behavior` check inverts: an agent that stayed away from the
forbidden thing is reported as having done it.
- `_met`'s `bool(claims)`. Without it `all([])` makes a case that asserts
nothing read as a side that passed.
- `_decisive`'s preference for a passing attempt that fired, which is what
keeps `fired_with` and `checks_with` describing the same attempt.
- `_why_dirty`'s preference for a known reason over `UNKNOWN_ROOM`, which is
compared as a sentinel downstream and so decides which group a case is
reported in.
- `_seconds` discarding a zero, the one column where reporting nothing as
zero inverts the finding.
- `_run_behavioral`'s `install`, `attempts`, `concurrency`, `baseline` and
`config_dir`, which were asserted by grepping the function's source. The
stub engine calls `solver_factory` the way the real one does, because
`install` and `config_dir` reach the agent only through that closure.
802 tests pass, ruff `F401,F841` clean.
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · fbd4a03
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change.
Says it closes #6; #6 itself was not retrieved, so that claim was checked against the description only.
Blocking
-
(new) When the graded half uses up the time budget, the report's headline says the baseline ran and was slow, but it never started —
skillscope/staleness.py:776-785,skillscope/cli.py:1079-1085
If the graded half exhausts the shared--timeout, every baseline case returns_failedwithout running._spend_leadchecks onlybaseline["finished"], so it prints "The baseline never finished. The graded half took 30m00s; the baseline was still going 0s in … an agent without the skill reaches it only by exploring" — the opposite of what happened.test_a_half_the_bound_stopped_is_marked_unfinishedbuilds exactly this state but asserts only the flags; addingand graded["finished"]to the condition left the suite green.
Confidence 100 (arender_markdownprobe printed the sentence) · logic · Fix: when the graded half did not finish, say the budget ran out before the baseline could start, and pin the sentence in that test. -
(new) The baseline wiring, which the whole command depends on, is not tested end to end —
skillscope/engine/verify.py:221-343,skillscope/engine/behavioral.py:786-798,skillscope/cli.py:1074-1085
On theclaude-codeengine, theinstall=Falsepath is covered only byinspect.getsourcetext checks. Seven single-branch mutations all left the suite green: staging the skill in the baseline anyway, passingskills=[skill_dir], the-baselinetask name,epochs,max_samples, passinginstallfromrunintobuild_task, and the baseline metadata.behavioral.rundroppingbaseline=baselinealso survived. At thecmd_stalenesslevel, giving both halves one sharedconfig_dir, orNone, survived too — the dir the comment calls what "the whole command rests on". On either engine the baseline could silently become a second graded run with CI green.claude_codecan be stubbed, asTestEveryLegThinksAsHardAsItWasToldalready does elsewhere.
Confidence 100 (mutation runs) · logic · Fix: stubclaude_codeand assert what each half receives; extend the_staleness_harnessassertions toconfig_dirper half. -
(new) Several new report branches and command checks are untested —
skillscope/staleness.py:528-534, 591, 870, 922, 935, 1031, 1051,skillscope/cli.py:1016, 1107-1108
Each survived alone: themax_casesbanner, the multi-attempt comparability banner, the UNKNOWN_ROOM note and the errored-attempts note set toif False;_detail's pipe escape removed;per_skillrenamed orbaseline_unknownzeroed insummarize; themax_cases < 0refusal dropped (only--concurrency -1is tested); themax_cases/concurrencymeta keys removed fromcmd_staleness.
Confidence 100 (178 tests run per mutation) · mechanical · Fix: one assertion per branch. -
(new) Some tests cannot fail, so they claim branches they do not check —
tests/test_skillscope.py:7445, 7449, 7472,test_a_tilde_belonging_to_something_else_is_left_alone
assertNotIn("disagreed with itself", …)andassertNotIn("Every graded case either", …)look for strings that appear nowhere inskillscope/(the code says "did not settle"). Line 7449 is the only assertion intest_one_attempt_a_side_can_never_read_as_unsettled, so that test always passes. The tilde test stays green with_HOME's lookbehind removed, because the expanded text matches no mark either way.
Confidence 100 (grep, and the lookbehind mutation) · mechanical · Fix: assert on the text the code actually emits ("did not settle"); give the tilde test a mark it would hit if the anchor were removed. -
(new) Leftovers from the dropped offline rebuild: dead parameters, and comments citing things that no longer exist —
skillscope/engine/behavioral.py:205, 331-375, 371, 481-486, 653-656, 806,skillscope/behavior.py:49-53,tests/test_skillscope.py:6474, 6643
_inspect_room(skill_dir=…)and_skill_marks(recorded=…)have no production caller; the only call,behavioral.py:600, passes neither, so that path is reached only by tests. Comments justify code by astaleness._no_duplicatesguard,--from-logs, and "a reader rebuilding someone else's run", none of which exist after 90bbfb3.BehaviorOutcome.attempt's comment gives the missing guard as its reason to exist.
Confidence 85 (grep ofskillscope/) · architectural · Fix: removeskill_dir/recordedand the tests that only cover them, or keep them for the follow-up and stop citing removed code; rewrite theattemptcomment around what reads it today, the JSON output. -
(new) The docs make claims the code does not bear out —
README.md(newstalenessparagraph; "Run it from your CI"),docs/usage.md,docs/harnesses.md
The docs saystaleness"refuses--harnessoutright";cmd_stalenessrefuses onlyasked != harnesses.DEFAULT, so--harness claude-codeis accepted. The code governs: it is the behaviour users get. The README's "This will triger all graders to run directly from your repo" now sits under four test types, whilereusable.yml:226, 439runs three, andusage.mdsays "no workflow runs it for you".
Confidence 85 · mechanical · Fix: "refuses any harness other than claude-code", and exclude staleness from the CI sentence. -
(new) The description leaves out one change — PR description
.github/workflows/selftest.ymladdsF841to the ruff step and renames the job; the description says "nothing new runs in CI" and does not mention it. Harmless in practice: ruff is clean andmainhas no required checks.
Confidence 85 · mechanical · Fix: add one line to the description.
Non-blocking
- (new) The "voided by contamination" text is shown for cases voided by an error —
skillscope/staleness.py:1062-1109
A caseungradedbecause an attempt errored, whose errored attempt also carries a contamination reason, is counted under "reportedungraded: a baseline that reached another skill…". Confidence 100 (probe) · logic. - (new) One UNKNOWN_ROOM message covers three causes, and is wrong for two —
skillscope/engine/behavioral.py:325-328, 477, 545-546,skillscope/staleness.py:1043-1057
An ancestor sweep, an exception mid-walk and no marks all say "the run did not record where the skill was", and the report adds "Re-running them records it" — which cannot help a sweep. Confidence 100 (probe) · logic. - (new) Within one sample, an early UNKNOWN_ROOM hides a later definite read —
skillscope/engine/behavioral.py:530-536
ls .claude/skills/thencat <skill>/SKILL.mdyields UNKNOWN_ROOM, not "read the skill's own files"._why_dirtyprefers a known reason across attempts but not within one; the mutation reversing it survived. Confidence 100 · logic. - (new) A cell containing a newline, or a pipe escape cut at 90 characters, breaks the Markdown table —
skillscope/staleness.py:870, 1251-1253
Confidence 100 (probe) · logic. - (new) One ungraded check errors the attempt even when another positive claim already failed outright —
skillscope/engine/behavioral.py:585-598
The comment's reasoning ("leaves the result undecidable") does not hold once a claim has failed; a real failure is discarded and the case can turnungraded. Confidence 70 · logic. - (new) The sandboxed baseline can be voided for a sweep that cannot reach the skill —
skillscope/engine/behavioral.py:516
The staged mark's ancestors (.claude,.claude/skills) apply on theclaude-codeengine too, where the container never holds the skill;ls ~/.claude/skillsthere returns UNKNOWN_ROOM and voidsno_signal/harmful. Confidence 70 · logic. - (new) The
claude-codebaseline can lose its whole half to one bad sample, and--attemptsraises the odds —skillscope/engine/verify.py:330-343
verify.runlacks thefail_on_error=FalseandSystemExithandlingbehavioral.runhas. Pre-existing, but staleness now builds its baseline half on it. Confidence 70 · architectural. - (new) With
--attempts > 1, missing epochs are not reported as missing —skillscope/engine/behavioral.py:630-633
gradedis keyed by case id only, so a case with any sample at all passes the check. Confidence 70 · logic. - (new)
stalenessexits 0 even when every case errored —skillscope/cli.py:1120-1123
CI cannot tell a run that died from a clean one;routingfails when every case errored. Confidence 70 · logic. - (new) The skill-mark check asks only the default harness —
skillscope/engine/behavioral.py(_skill_marks)
Its docstring says this is "right while that is the only one registered", but Copilot already is (#34);behavioral --harness copilotnow writesactivated/contaminatedcomputed against.claude/skillsinto its JSON. Confidence 70 · logic. - (new) Stale or inaccurate comments and docstrings —
datasets.py:26says "two graded commands";no_sandbox.harness_solver's docstring still names routing as the only caller passingconfig_dir; thestaleness.pymodule docstring callsno_signal"two findings" where the docs and_advicesay three;_spend_leadsays "three shapes" and has four returns;build_tasksays "baselineonly names the task" but it also sets metadata and tags;judge.gradedoes not mention it now raisesUngradable;staleness.py:1188cites commit542cad9, which a squash merge will erase; test docstrings citeunattributed,guards_onlyandTestEarningItMeansTheSkillActuallyFired, none of which exist; about 19 "used to / the first real run" comments instaleness.pyandcli.py, and the same anecdotes inusage.md, describe history rather than current state. Confidence 85 · mechanical.
Decisions for the author
- The contamination check works by matching strings — tradeoff
It is conservative (any sweep word near an ancestor becomes UNKNOWN_ROOM, and every string argument is scanned, Write/Edit contents included) and still missescd X && cat rel/SKILL.mdandfind / -name SKILL.md(the latter by the deliberate top-of-machine floor). Confirm a string matcher with both kinds of error is acceptable for the measurement the command rests on. - Size — non-blocking-improvement
5,300 lines: a 1,256-line new module with a ~380-linerender_markdown, ajudge/behavioralbehaviour change that also affectsbehavioral, and a CI lint change. The judgeUngradablechange could land on its own first. - Log metadata keys with no reader yet (
BASELINE_KEY,SKILL_KEY,SKILL_DIR_KEY) — answered-by-intent
Description: "inspect logs gain the skill, the side and the skill's path. All additive." and "--from-logs… scoped out of this round and will come back on their own." Would otherwise have been part of the dead-code finding; the unusedskill_dir/recordedparameters remain under Blocking. Confirm the follow-up still plans to read them.
Positive signals
judge.Ungradableis an exception rather than abool | Nonereturn, which prevents themust_happen=Falsepolarity inversion; a scorer test pins it.scorers.graded()defaults a missing key to True, so logs written before this change still read correctly.usage.leg()swaps the global accumulator instead of subtracting snapshots, and a test pins the "both halves free" trap._secondsreturnsNonerather than0, so a baseline that never finished is never rendered as faster than the graded half.- Contamination voiding is driven by one constant,
VOIDED_BY_CONTAMINATION, and is deliberately asymmetric:earns_itsurvives a dirty baseline;no_signalandharmfuldo not. _gradable_casesis shared bybehavioralandstaleness, so--onlyand--max-casescannot drift between the halves.- The staleness regression tests run
_outcomesover real log objects rather than hand-built errors.
Deployment notes
- New
stalenesssubcommand; no new dependencies and no changes to existing flags.--timeoutdefaults to 1800s for it. On the host engine it refuses to run withoutANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKEN. behavioralchanges behaviour: a judge that could not answer is now an error, where it used to be a failed check.behavioralJSON gainsactivated,contaminatedandattemptper case, andgradedper check.- The selftest job is renamed "Unused imports and locals";
mainhas no required status checks, so nothing breaks.
What this covered
Every file in 0a25c86…fbd4a034 (18 files), read completely, by four independent workers: staleness/cli/usage/behavior/datasets with their tests; engine/* with its tests; docs, action.yml, workflow, pyproject.toml plus impact analysis; history, earlier merged PRs (#15, #19, #33, #34) and code comments. Workers read the head from a detached temporary worktree; running and mutating happened only in per-worker worktrees.
CI rollup: all 8 checks SUCCESS. Locally the engine and staleness subsets pass unmutated; the full suite shows 3 failures and 4 errors, all in TestTheDefaultEngineRunsACase (reads the local Claude install) and TestWhatABranchChanged (needs git commit to work), neither touching this change; both pass in CI.
Test verdicts: every new test fails with the change reverted. About 40 single-branch mutations were run with __pycache__ cleared before each; the survivors are listed under Blocking. The classify/pair/summarize, judge Ungradable, scorer graded and usage.leg tests killed theirs.
Standing blockers from the review at 3f0dc369 were not re-found blind and were settled by putting them to the code directly — anchored evidence, not blind:
- Contamination spellings: all eleven spellings it listed (
grep -ri,-rn,-Rn,rg,tree,fd,ls -lR,~/,$HOME/,//,/./) were re-probed at this head; none reads as a clean room. No longer holds. - Untested branches: the mutations it named are now caught (re-checked here:
files_existgraded=False→Trueand the~expansion disabled both go red). The untested-behaviour class still holds through different branches, which this round reports as new findings above.
Did not run: the design review was done by the driver, not independent workers; no real paid staleness run (no inspect_swe or Claude CLI auth here), so verify.build_task was judged by mutation against the existing tests rather than run; issue #6 was not retrieved. Fixes in this round were written by the PR author. Not reconciled against the PR discussion.
**A baseline that never started was reported as the slow one.** The halves
share one `--timeout`, so a graded half that spends all of it leaves the
baseline none: every baseline case returns failed without an agent ever
starting. `_spend_lead` read only the baseline's `finished` flag, so the
headline said "the baseline was still going 0s in, when the bound stopped
it ... an agent without the skill reaches it only by exploring" -- a half
that never ran, described as exactly the finding this command exists to
produce. The graded half's flag is what tells the two states apart, and
there is now a fourth sentence for the one where the budget ran out first.
`test_a_half_the_bound_stopped_is_marked_unfinished` built this state
already and asserted only the flags, so the sentence went unread; it is now
asserted, with a control for the genuine case -- graded half finished,
baseline stopped partway -- that must keep saying what it said.
**Three tests asserted the absence of strings the code never emits.**
`assertNotIn("disagreed with itself", ...)` and `assertNotIn("Every graded
case either", ...)` match nothing in `skillscope/`; the code says "did not
settle" and "Every graded case needed the skill". One of them,
`test_one_attempt_a_side_can_never_read_as_unsettled`, had no other
assertion, so it could never fail. Each now names the string the code
prints, and the one that was alone has gained the positive half.
**And one of mine from last round.** The tilde test used `cat
notes.txt~/<skill>`, which reads as a clean room whether the anchor on
`_HOME` is there or not: expanding it puts the skill's path straight after
`notes.txt`, and `_mentions` rejects a mark that starts mid-word. For a
word character the boundary check is what blocks the false positive and the
anchor is belt and braces. The test now uses `$$HOME/` and `~~/`, where
what precedes the expansion is itself a boundary and the anchor is the only
thing between the spelling and a hit. Both go red with the lookbehind
removed; the old one did not.
**Docs corrected to what the code does.** `staleness` refuses any
`--harness` other than `claude-code` rather than refusing `--harness`
outright -- `--harness claude-code` is accepted, and a test pins that.
Fixed in `README.md`, `docs/usage.md` and `docs/harnesses.md`. The README's
CI paragraph now names the three graders `reusable.yml` actually runs and
says `staleness` is not among them.
804 tests pass, ruff `F401,F841` clean. The new branch and each repaired
test were checked by mutation.
The `claude-code` leg's baseline half was asserted by reading `verify.build_task`'s source for a substring. Everything `install` controls could therefore be wired to the graded value with the suite green, and that failure is silent in the worst way: the baseline becomes a second graded run, every case comes out `no_signal`, and the report says the skill earns nothing. `inspect_swe` is not installed in the unit job, so `claude_code` is stubbed as `TestEveryLegThinksAsHardAsItWasTold` already stubs it, returning a real solver so `chain` still builds. The staging solver is recorded the same way. Both halves are built, because every assertion here reads the same on one half alone. Nine mutations, each applied alone, each now caught: - the baseline staging the skill anyway, and `skills=[skill_dir]` reaching the agent -- the two that make a baseline not a baseline; - the `-baseline` task name, the `BASELINE_KEY` metadata and the `baseline` tag, which are how the two halves are told apart in the log directory; - `verify.run` dropping `install` on its way to `build_task`, which builds a graded task out of a baseline run, and dropping `epochs`; - `behavioral.run` dropping `baseline=baseline`, the same join on the other engine; - `cmd_staleness` handing the baseline `None` for its config dir, which is what keeps the runner's own `~/.claude/skills` out of the room the whole measurement rests on. Tests only; no production code changed. 809 pass, ruff `F401,F841` clean.
The last two blocking findings. **The caveats the report carries were untested.** Nine branches, each mutated alone and each green: the `--max-cases` banner, the multi-attempt comparability banner, the unknown-room note, the errored-attempts note, `_detail`'s pipe and newline escape, `per_skill`'s keys, `baseline_unknown`, the `--max-cases` half of the negative-count refusal, and the `max_cases` and `concurrency` meta keys. None of them can make the command reach a wrong verdict; all of them are how a reader knows what the verdicts are worth. A capped run that does not say it was capped reads as a statement about the whole suite, and the comparison banner is the only thing in the report saying a three-attempt run is not comparable with a one-attempt one. The `--max-cases` refusal was the one real behaviour among them: only `--concurrency` was covered, so dropping `--max-cases` from the condition let a cap of -1 reach `_gradable_cases`, where a negative slice silently selects nothing. **And comments citing things this round removed.** `_inspect_room`'s `skill_dir` and `_skill_marks`'s `recorded` have no caller in the package: the offline reader they are for went out with `--from-logs`. They are kept rather than deleted, because `SKILL_DIR_KEY` is written to every log from now on and a reader arriving to find the seam missing would have to resolve the skill against itself, which is the one answer always wrong for somebody else's run -- but the docstrings now say that is what they are waiting for, instead of describing a reader that exists. Likewise a comment justifying `epochs` by `staleness._no_duplicates`, which no longer exists, and four test docstrings naming `--from-logs` as a current flag. 816 tests pass, ruff `F401,F841` clean. Every branch above was confirmed by mutation, including one whose first attempt failed to apply and reported a false pass until it was re-run.
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · 8bfbc56
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (18 files, prw-base...8bfbc568).
Closes #6 ("Add skill staleness test"); the description matches what the change does, apart from the docs claim in blocker 4.
This replaces the two earlier change requests with one complete position. Resolved since fbd4a034: the "baseline never started" headline (_spend_lead), the --harness refusal wording in the docs, the undeclared F841 change, and the main baseline-wiring tests. Everything below still holds at this head.
Blocking
-
(standing, narrowed) The contamination check calls the room clean when the agent sweeps with Glob/Grep tool calls —
skillscope/engine/behavioral.py:269-271(_swept),:478(_read_the_skill)
Both test each string argument on its own, so a sweep word and a skill-ancestor directory must sit in one string. Claude Code's Glob and Grep pass them as separate arguments (pattern,path). Probed on this head:_sweptreturnsFalsefor each of"**/SKILL.md","/home/u/dev/proj/skills","deploy"and".claude", and_read_the_skillreturnsNonefor{"pattern": "**/SKILL.md", "path": "/home/u/dev/proj/skills"}and for{"pattern": "deploy", "path": ".claude"}, while the same sweep typed as a Bash string is flagged. These are the tools an agent told nothing uses to look around, on the side whose emptiness the verdict rests on, so a baseline that found the skill still yieldsno_signal/harmfulinstead of being voided. The earlier Bash spellings (-rn,rg,$HOME,//) are now caught.
Confidence 100 · logic · Fix: when a call carries a sweep marker in any argument, test the ancestors across all its string arguments, or treat a Glob/Grep call whosepathis at or above an ancestor as a sweep. -
(new) A sample cut off by the time or message limit is graded as an ordinary failure —
skillscope/engine/behavioral.py:591-615(_outcomes),:135(time_limit=task_time_limit(bound))
Nothing inskillscope/reads inspect'ssample.limit(grep finds no.limituse). The baseline runs second on whatever budget the graded half left, and is the half that explores longest, so a baseline stopped by the clock arrives aspassed=False, error=None,answered()keeps it, and a case the skill did not earn can read asearns_it. A probe with a sample carrying a timelimitreturnedpassed=False, error=None. The leg-levelfinishedflag marks only that the deadline fired, not which cases were cut.
Confidence 70 · logic · Fix: map a time- or message-limited sample toerror="stopped at the time limit"so the case comes outungraded. -
(standing, narrowed) New behaviour that tests do not pin, and tests that cannot fail —
tests/test_skillscope.py
Each item below was reproduced by a reviewer worker as a single mutation, applied alone to a copy of this head, that left the module green; I confirmed the first by reading.:6440assertNotIn("every case needed the skill", …)names a string the code never emits (it says "Every graded case needed the skill"), so it cannot fail._outcomes→_inspect_room: replacing the call withNone, None, or droppingactivated=/contaminated=, stays green. Every test calls_inspect_roomdirectly or hand-buildsBehaviorOutcome.cmd_staleness:staleness.pair(with_skill, without)swapped topair(without, with_skill)(cli.py:1087),baseline_leg.finished = graded_leg.finished(:1085), and removing_fail_if_expired(:1121) all stay green._staleness_harnessreturns[]for both halves, so no verdict is ever compared, which would flipearns_itandharmfulin every real report.test_the_other_engine_records_it_too(:8169-8178) grepsinspect.getsource(build_task)forSKILL_DIR_KEY, and a comment above the line satisfies it; deleting the metadata line stays green.usage.leg: removing thefinallystays green (:8667-8674, the assertion reads the inner accumulator by accident).- Report branches with no assertion: the nothing-graded banner (
staleness.py:1005,:1211), aharmfulcase appearing under "Cases worth a look" (:1199), a case present only in the baseline (pair,:474),baseline_contaminatedexcludingUNKNOWN_ROOM(:585-590),verify.runpassingmax_samples, and the per-halfconfig_dir(gradedNoneand a shared dir are both green at:8780).
Confidence 100 for the first, 90 for the rest (mutation runs by workers) · logic · Fix: one assertion per branch; giverecorddistinguishable outcomes per half and assert the verdicts; asserttask.metadata[SKILL_DIR_KEY]instead of grepping source.
-
(standing) Write-only keys and parameters, and docs that describe a reader that does not exist —
skillscope/engine/behavioral.py:122-125, 331-376, 483-495,skillscope/engine/verify.py:254-258,docs/usage.md:365-372
SKILL_DIR_KEY,SKILL_KEYandBASELINE_KEYare written to every log and read nowhere (grep ofskillscope/)._inspect_room(skill_dir=…)and_skill_marks(recorded=…)have no caller;_outcomescalls_inspect_room(routing_engine, sample, skill)without them. The docs go further than the code comments, which admit it:usage.mdsaysskillscope_skill_dir"is what lets the baseline be judged empty" and that a log without it "is read as an unknown room rather than an empty one", a path nothing runs. Dead parameters are blocking under the rubric, and a doc describing behaviour the code does not have is the same defect as a description that overclaims.
Confidence 95 · architectural · Fix: land the reader with this change, or delete the keys, the two parameters, their tests and the docs paragraph until the reader exists, and rewrite the comments that cite--from-logsand_no_duplicatesaround what runs today.
Non-blocking
- (new) A run that graded nothing exits 0 —
cli.py:1129-1131. Every attempt can error on both halves (dead provider, missing CLI) and CI stays green;behavioralandroutingfail in that shape. Confidence 70 · logic. - (new) With
--attempts > 1, lost epochs are not reported —behavioral.py:638-641keys on case id only, so a case with one of two epochs reads as fewer attempts. Related and pre-existing:verify.runlacks thefail_on_error=Falsethatbehavioral.runhas (verify.py:327-342), so one container error drops the remaining cases on both halves. Confidence 75 · logic. - (new) The sweep match is over-broad, and still misses some spellings —
behavioral.py:230, 269.sort -r /home/u/notes.txt,echo "what?" > /home/u/q.txtandpip install -r /home/u/dev/proj/requirements.txtall come backUNKNOWN_ROOMand void the case;cd /home/u/dev/proj/skills && cat run/SKILL.mdandcat "$HOME"/dev/proj/skills/run/SKILL.mdcome back clean. Confidence 65 · logic. - (new) A multi-line expectation or an id with
|breaks the Markdown table —staleness.py:1233, 1262, 1271-1272; a probe split a row across two lines, and[:90]runs after the escape. Confidence 80 · mechanical · Fix: one_cell_text()helper that escapes|, flattens newlines and truncates first. - (new) The credential refusal runs after the preflight probes —
cli.py:1030-1043vs_prepare_graded_run; a runner with no key can be told "model not reachable" first. Confidence 60 · logic. - (new) Stale or inaccurate prose —
staleness.py:1207cites commit542cad9, which resolves to nothing in this repository;no_signalis "two findings" (staleness.py:~28, 392, 1100) but three inusage.mdand_advice(:694); comments narrate taxonomies that never shipped onmain("used to", "the oldguards_only", "earlier versions");judge.grade's docstring andtuple[bool, str]return do not mentionUngradable(judge.py:237);_skill_markssays "the only one registered" though Copilot is (behavioral.py:~337);deadline.py:7anddatasets.py:26still count graded commands withoutstaleness;--harnesshelp still lists copilot forstaleness. Confidence 85 · mechanical. - (new)
tools/sandbox_smoketest.py:~134has a dead branch — itsunlistablecheck follows theerroredcheck, which now always fires first for agraded=Falsecheck. Confidence 75 · logic. - (new) Weaker test details — a test that calls
_selecttwice on identical input (test_both_halves_get_the_same_set);assertNotIn("0s", report)fails on any duration ending in0s; refusal tests do not scrubSKILLSCOPE_HARNESS;_staleness_harnessdoes not resetusageon exit; theusage.legfold-back of output tokens, cost andwall_time_sis unasserted. Confidence 85 · mechanical. - (new) Pre-PR logs read differently from new ones —
gradeddefaults toTrue, so an old log's "judge skipped" check reads as a real failure; it bites only once an offline reader lands. Confidence 55 · logic.
Decisions for the author
- The contamination check works by matching strings — tradeoff
It is conservative in one direction (over-broad voids) and blind in another (separate-argument tool calls,cd-relative paths). Confirm a string matcher with both kinds of error is acceptable for the measurement the command rests on; blocker 1 is the part that is wrong rather than a tradeoff. - 5,600 added lines for one opt-in command — non-blocking-improvement
The judge/Ungradablefix and theusage.legaccounting stand on their own and could land separately.
Positive signals
Ungradableis an exception, not a falsy value, andanswered()is the single place infrastructure failure is separated from a verdict.usage.legswaps the accumulator instead of subtracting snapshots, and the aliasing hazard is explained and tested._gradable_casescaps once, so both halves grade the same cases, and--attemptsand--concurrencyare read once for both.- The baseline staging is exercised by driving the real
harness_solverandverify.build_taskwith a stubbedinspect_swe, replacing the earlier source-grep checks. pairreports a case missing from either side asungradedrather than shrinking the denominator, and the unknown-room result refuses to claim clean rather than claiming dirty.
Deployment notes
- New
stalenesssubcommand; on the default engine it needsANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKEN; Claude Code only. - CI gains a ruff
F841gate in this repository's selftest, which may fail other open PRs with unused locals. behavioralJSON and inspect logs gain fields (additive).
What this covered
All 18 files at prw-base...8bfbc568 (merge base 0a25c86), read completely, as five independent reviewer workers in groups: staleness, CLI and docs; the grading engine; the staleness tests; the engine tests; history, comments and cross-file impact. Workers read the checked-out tree directly and ran probes and one-at-a-time mutations only in their own copies of it, which are left under /tmp/pr35-*. I confirmed the vacuous assertion, the missing readers, the docs passage, the absent .limit handling and the Glob/Grep probes myself; the other test-mutation results rest on the workers' runs.
CI: all 8 checks on this head are SUCCESS (statusCheckRollup, both check runs and statuses read); the base-branch exception was not needed. ruff was not run locally.
Test verdicts: the workers ran the full module with 3 failures in TestTheDefaultEngineRunsACase (a real claude CLI and host skills on the review machine; one worker reports them failing on prw-base too, which I did not re-run) and 4 environment errors in TestWhatABranchChanged (pre-existing, needs git signing off). About 235 and 175 single-branch mutations were applied to the staleness and engine tests; the survivors are the third blocker.
Did not run: the design-question step was done by me from the workers' findings, not as separate workers; no workflow definitions were read. The description and #6 were read after the findings were settled, and the earlier single-context attempt this round was discarded unpublished. Fixes in this round were written by the PR author. Not reconciled against the PR discussion.
Four blockers from the fourth review pass.
**A sweep through Glob or Grep read as a clean room.** Both the sweep test
and the read test ran per string argument, so a sweep needed its word and a
skill-ancestor directory in the same string. A Bash command is one string and
was caught; Glob and Grep carry the two halves as separate arguments, and
those are the tools an agent that has been told nothing reaches for first.
Probed on the last head: `{"pattern": "**/SKILL.md", "path": <ancestor>}`
came back `None` while the identical sweep typed as a shell command was
flagged.
The sweep question is now asked of the whole call rather than of each
argument. And a search tool needs no sweep word at all: `Grep(pattern=
"deploy", path=".claude")` reads every file under `.claude` and carries no
wildcard, no `-r` and no path naming the skill, so the only evidence is which
tool was called. `_searches` says so by function name, loosely, because the
same two tools are spelled differently by different harnesses.
**A sample inspect cut short was graded as an ordinary failure.** Nothing
read `sample.limit`, which inspect sets separately from `sample.error`, so an
agent the clock stopped arrived as `passed=False, error=None` and `answered`
kept it. The baseline runs second on whatever budget the graded half left and
is the half that explores longest, so it is the one the limits cut -- and a
baseline interrupted rather than beaten reads as a case the skill earned.
Time, working-time, message and token limits now error the attempt, which
lands the case `ungraded`. `operator` and `custom` are left alone: one is
somebody cancelling and the other is a task's own rule.
**The write-only keys and their parameters are gone.** `SKILL_DIR_KEY`,
`SKILL_KEY` and `BASELINE_KEY` were written to every log by both engines and
read nowhere, and `_inspect_room(skill_dir=…)` and `_skill_marks(recorded=…)`
had no caller. Keeping them and correcting the comments was the wrong answer
last round: the `usage.md` paragraph went further than the comments did and
described a reader that does not exist. Deleted, with the tests that only
covered them and that paragraph. The tags and the task name already tell the
two halves apart in a log directory. They come back with the reader.
**And the tests that could not fail.** The sharpest: `_staleness_harness`
returned no outcomes from either half, so no verdict was ever compared and
`pair(with_skill, without)` could be called with its arguments swapped with
the suite green -- every real report would have had `earns_it` and `harmful`
the wrong way round. The halves now return outcomes that differ, and the
orientation is asserted on both the verdict and the time column. Also: a
third vacuous `assertNotIn` naming a string the code never emits; `_outcomes`
calling `_inspect_room` at all; `usage.leg`'s `finally`; and six report
branches with no assertion.
Two of those needed a second attempt. The `usage.leg` test first asserted a
total, which comes out the same either way because the fold-back adds exactly
what the leaked accumulator already holds; it asserts identity now. And the
first mutation of that `finally` was a syntax error rather than a test, so
its pass meant nothing until it was re-run properly.
829 tests pass, ruff `F401,F841` clean. Every fix above confirmed by mutation.
Closes #6.
skillscope behavioraltells you a skill's cases pass. It cannot tell you theypass because of the skill.
skillscope stalenessruns a skill's existing behavioral cases twice, once withthe skill installed and once with nothing installed, and puts the two runs side
by side. It reads the same
evals.json, so there is no new file to write and nonew assertion to learn.
What it reports
What each half spent. An agent holding the skill goes straight to the work;
an agent without it explores. Run on
rocm-doctor:Six times the wall clock and four times the money, to fail both cases.
A verdict per case.
earns_itharmfulno_signalungradedThe pass counts and an advice column sit beside each verdict, so a
no_signalwhere both sides passed is told apart from one where both sides failed.
What the skill stopped. A guard rail the baseline broke and the skill held
is credited to the skill. On
rocm-doctorthe unaided agent attempted asudowrite under
/opt/rocm/shareon a case the skill kept read-only.Running it
Two agent passes, and the baseline is the slow one, so
--timeoutdefaults totwice the usual and
--max-casesand--concurrencyare there to keep a runaffordable. Needs
ANTHROPIC_API_KEYorANTHROPIC_AUTH_TOKENon the defaultengine: each half gets its own throwaway
CLAUDE_CONFIG_DIR, which only workswhen auth comes from the environment. Claude Code only for now.
Reporting only. A verdict never fails a run, though an expired deadline or a
refused run still exits non-zero.
Effect on the existing commands
structural,routing,behavioralandselectbehave the same, and nothingnew runs in CI:
stalenessis absent fromselection.plan()and from everyworkflow.
One change to this repo's own CI, which does not reach a consumer's.
.github/workflows/selftest.ymladdsF841besideF401on the ruff step,and the job is renamed to match.
F841finds a local nothing reads, which isthe same signal as an unused import one scope down: a review found a list
still being built on every run after the report section that consumed it was
deleted, and neither
F401nor a green suite could see it. The tree is cleanunder both rules.
One behaviour change beyond this command.
engine/judge.pyreturned a failedcheck when the sandbox would not list or the grader answered in prose, which is
the run failing rather than the agent. It now raises, checks carry a
gradedflag, and the case comes out
ungraded.behavioralgets that fix too.behavioral's JSON gainsactivated,contaminatedandattemptper case andgradedper check, and inspect logs gain the skill, the side and the skill'spath. All additive.
--from-logs,--run-idand--unlabelled-logsrebuilt a comparison from logson disk. They are scoped out of this round and will come back on their own.