Repository navigation
Conversation
nowycondro
left a comment
There was a problem hiding this comment.
🤖 Automated review · review-code · head 5e885ae9
Review — needs work
Full review of the whole change.
Implements the checklist requested in issue #9 ("Create checklist showing requirements to pass each test"), which remains open (expected — it auto-closes on merge via the PR's "Closes #9").
Blocking
-
machine.yml'ssandboxkey is documented as requiringlabelstoo, but the two are independent —docs/checklist.md:112-114
The doc says: "sandboxnames a compose file sitting beside themachine.yml. You need it as well aslabelsif the skill needs network access or a device."skillscope/schema/machine.schema.json:16-28andskillscope/engine/sandbox.py(for_skill/_declared_compose) treatlabels(whichruns-onpool a job lands on) andsandbox(what the container on that pool can reach) as fully independent, separately-optional keys; nothing requireslabelsforsandboxto take effect. A skill that only needs network egress on an ordinary runner needssandboxalone — following this bullet would have it unnecessarily requestlabels, which the docs elsewhere say makes the leg "scoped" (gated, routed to a special pool). Confidence 85 · logic · Fix: drop "as well as labels"; state the two keys as independent, as the schema and the bullet's own closing sentence already do. -
The judge is said to see only what the agent did, but it is also shown what the agent said —
docs/checklist.md:168-169
"The model doing the judging is shown what the agent did, not what the agent said about it."skillscope/engine/judge.py:239-267builds every evidence packet with an explicit"--- what the agent SAID to the user (its final message) ---"section (final_message_of, judge.py:97-106) and instructs the judge to use it as evidence for requirements about what the agent told the user (judge.py's own example: "output the curl commands they need"). The narrower claim ("an agent claiming it avoided something is not evidence") is correct for action-type requirements, but the categorical sentence is false and could lead an author to believeexpected_behaviorcan never test communicated content, when the code explicitly supports that. Confidence 85 · logic · Fix: qualify the sentence — the judge sees both the actions and the final message; a claim of avoidance in the message isn't evidence for action requirements. -
"Whichever you use" states a per-grader
optional/offtoggle that only exists forreusable.yml—docs/checklist.md:239-240(heading atdocs/checklist.md:228)
.github/workflows/reusable.ymlhasstructural/routing/behavioralinputs eachrequired/optional/off..github/workflows/skill-evals.ymlhas no such per-grader toggle at all — its only grader-selection input ismode: both|routing|behavioral|references, an all-or-nothing switch with no non-gating "optional" state. Placed under a heading that reads "whichever you use," this bullet states something untrue of one of the two workflows the section is about. Confidence 85 · architectural · Fix: move the bullet underreusable.yml's own subsection, or note explicitly thatskill-evals.ymlhas no equivalent. -
reusable.yml's input list includesapi_key, which is actually a secret, not an input —docs/checklist.md:193-195
"It accepts these inputs and no others:skills,structural,routing,behavioral,runner,min_accuracy,api_key,api_key_secret,api_base_url,api_custom_headers." In.github/workflows/reusable.yml,api_keyis declared underon.workflow_call.secrets, notinputs— the actualinputs:block has 9 entries, not includingapi_key. The doc's own next two bullets correctly describe mappingapi_keyin viasecrets:, so this is confined to one sentence's wording, but "inputs ... and no others" is a specific, checkable claim and it's wrong. Confidence 75 · mechanical · Fix: say "these inputs, plus theapi_keysecret" or similar.
Non-blocking
- Pre-existing, pattern-relevant:
docs/authoring-evals.mdomits thesandboxkey frommachine.ymlentirely —docs/authoring-evals.md:134("Both keys are optional:", showing onlyos/labels)
The code'sMACHINE_KEYSis{"os", "labels", "sandbox"}(confirmed againstskillscope/datasets.py), and this PR's own newdocs/checklist.mdcorrectly lists all three. Not introduced by this change —docs/authoring-evals.md's machine.yml section isn't touched here — but this PR explicitly cross-linkschecklist.mdandauthoring-evals.mdto each other for the first time, so a reader bouncing between the two now hits a direct contradiction on this exact topic.
Decisions for the author
- Hand-written checklist vs. generated/tested one — tradeoff
The description states the rules were "taken from the code rather than restated from the prose," and the great majority genuinely trace cleanly tostructure.py,datasets.py,references.py,routing.py/engine/routing.py, and the two workflow files (verified bullet-by-bullet). But this review still found four confirmed inaccuracies in a 253-line first draft, plus a pre-existing cross-doc contradiction this PR newly links together. Worth the author's call: is periodic manual re-review the intended way to keep this in sync with the code as it evolves, or would a lightweight generated/testable check (e.g. asserting the documented key/input lists against the actual schema and workflow files) be worth the complexity here?
Positive signals
- Every subtle validation-order rule in the "
evals/evals.json" section — the exact top-level key set, the true/false-case key split including wherenotebelongs, and that id-uniqueness is repo-wide rather than per-file — traces precisely todatasets.py, which is easy to get wrong restating from memory. - The new 253-line document's internal cross-references (into
README.md,usage.md,authoring-evals.md, and both workflow files) all resolve, and none of the three small link-edits to the existing docs were found broken or mis-anchored.
Deployment notes
None — documentation-only change; no runtime or CI behavior is altered.
What this covered
Full review of all 4 changed files (README.md, docs/authoring-evals.md, docs/checklist.md [new, 253 lines], docs/usage.md), base 2a1959a9…head 5e885ae9. Verified every technical claim in the new file against the actual implementation: read completely skillscope/structure.py, skillscope/datasets.py, skillscope/references.py, skillscope/schema/evals.schema.json, skillscope/schema/machine.schema.json, skillscope/engine/routing.py, skillscope/routing.py, skillscope/selection.py, skillscope/engine/behavioral.py, skillscope/engine/scorers.py, skillscope/behavior.py, skillscope/engine/hooks.py, skillscope/engine/judge.py, .github/workflows/reusable.yml, .github/workflows/skill-evals.yml, skillscope/config.py, docs/usage.md, docs/authoring-evals.md, README.md, plus targeted reads of skillscope/cli.py and skillscope/engine/sandbox.py.
Fan-out: four independent workers grouped by grader/topic (Structural; Routing; Behavioral; CI + doc cross-references). Every worker-reported candidate was independently re-verified by the driver directly against source before inclusion. Several worker-reported items were checked and dropped as sub-70/non-issues: a "2xx" vs. the code's literal {200-206,429} wording (already present verbatim in the existing, unchanged usage.md); a routing_room: none nuance that applies only to direct misuse of the routing CLI command, not the workflow-input usage this doc describes; and a minor "error has several causes" simplification. One claim — that this PR deletes skillscope/harness/, tests/test_harness.py, and docs/harnesses.md — was checked against the PR's own diff and GitHub's own changed_files/additions/deletions counts and found false: those files exist in neither this PR's base nor head commit (they were added to main later, by an unrelated PR); it is excluded from this report entirely.
Did not run: history (docs/checklist.md is new; nothing to blame); prior-changes (no earlier review round visible on this PR).
Not reconciled against any discussion on this PR.
Not a human review. This is an automated code review; please verify findings independently before acting on them.
Review — needs workFull review of the whole change. Blocking
Non-blocking
Decisions for the author
Positive signals
Deployment notesNone. What this coveredEvery file in the diff, completely: Design review (Step 7) was done directly from evidence already gathered during the fan-out rather than via additional spawned workers, given the change's size; it produced the two items under Decisions for the author and no further open questions. Did not run: a behavioral pass tied to a specific agent-instruction file — none exists in this repository (no Not reconciled against any discussion on PR #31 — none was read, per the review method. 🤖 Automated review by an agent. |
johnl-amd
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change.
Closes #9 ("Create checklist showing requirements to pass each test"), and does so fully — the new page covers structural, routing and behavioral requirements as asked, plus a CI section not strictly requested but clearly in scope.
Blocking
-
`none` switches routing offis false whenrouting_roomis passed straight to theroutingcommand —docs/checklist.md:135(also the section's framing atdocs/checklist.md:125-135).
skillscope/cli.py's_empty_room()(~line 319) raisesSystemExit("error: \routing` asks for a routing run and `--routing-room none` empties the room...")wheneverconfig.wants_no_skills(args.routing_room)is true — i.e.skillscope routing --routing-room none(or a config/workflow value ofnonereachingcmd_routing) is a hard error, not a skip.noneonly "switches routing off" one layer up, inskillscope select's CI-plan construction (skillscope/selection.pyrouting_needed()/plan()), which decides whether theroutingjob is *scheduled* at all — it never reaches theroutinggrader this checklist section is about. A reader who setsrouting_room: noneexpecting a quiet no-op fromskillscope routingitself gets a crash. Confidence 85 · logic · Fix: scope the claim toselect/the CI plan (e.g. "nonekeepsroutingout of the CI planselectbuilds; passed toskillscope routing` itself it is a hard error, since the command is the routing run"). -
The
sandboxcompose-file check is not part of the free Structural grader —docs/checklist.md:112-116, inside "###evals/machine.yml" under "## 1. Structural", whose own intro promises "No agent, no cost, seconds" (docs/checklist.md:58-60).
skillscope/datasets.py'smachine_plan()(~line 571) only validates and returns{"os": ..., "labels": ...}— it acceptssandboxas a known key name but never reads, resolves, or checks the file it names.structural_errors()(~line 455) only surfaces aSystemExitfrommachine_plan(), which cannot be raised for a bad or missingsandboxtarget. The actual existence check lives inskillscope/engine/sandbox.py's_declared_compose(), called only fromskillscope/engine/behavioral.py'sbuild_task()(~line 119) — i.e. only when an actual behavioral task is built, which for--engine claude-codeis the costly, agent-running path section 1 exists to gate in front of. A skill whosemachine.ymlnames a nonexistent compose file passesskillscope structuralcleanly and only fails once a behavioral run is already starting.
Confidence 85 · logic · Fix: either move this bullet to section 3 (Behavioral), or state explicitly that the compose file's existence is checked lazily at run time, not for free alongside the othermachine.ymlchecks.
Non-blocking
- The
errorgloss under Routing ("the agent neither switched a skill on nor answered") describes only one path into_error_outcome(no_output, inskillscope/engine/routing.py); the sameerror/excluded-from-scoring verdict is also reached viasample.error, a missing inspect result, or a crashed task, some of which can follow partial agent activity rather than silence. The scoring behavior the checklist describes (excluded from both sides) is still correct — only the parenthetical explanation of why a case errors is narrower than reality. docs/usage.mdand the newdocs/checklist.mdnow each independently restate fine-grained behavior straight from the source (pass marks, key lists, exclusion rules). That's a reasonable design for a quick-reference page, but it is two places that can drift from the code and from each other — which is exactly how the two findings above arose in this first draft. Worth keeping in mind as both pages are maintained going forward; not something to fix in this PR.
Decisions for the author
None
Positive signals
- The numeric limits and exact key sets transcribed from the implementation —
name≤ 64 chars,description≤ 1024, body ≤ 500 lines, theanthropic/claudereserved-substring rule, the 3-true/2-false Tier 0 bar excludingextended_evals.json, theTRIGGER_CASE_KEYS/NO_TRIGGER_CASE_KEYSsplit,min_accuracydefaulting to1.0, the complete 9-input-plus-1-secret list forreusable.yml, and thefiles_exist/logs_containmatching semantics (including theplan.mdvsexamples/plan.mdsegment-match example) — are all precise, verbatim-accurate restatements of the code. The overwhelming majority of the ~40 distinct factual claims checked here held up exactly. - The cross-linking (README → checklist.md, authoring-evals.md ↔ checklist.md ↔ usage.md) is clean, with no orphaned or circular references, and the one external verification claim in the PR description (37 internal references resolved via
skillscope structural --docs) is the right tool for that specific question.
Deployment notes
None — documentation-only change, no runtime impact.
What this covered
Every changed line (README.md, docs/authoring-evals.md, docs/checklist.md, docs/usage.md) read completely, diffed 2a1959a996f9f527b16fa700f9acbe780750f3f8...5e885ae9622bb45333e6cfd14e0ff40c31fc7ea6, cross-checked against the full text of skillscope/structure.py, datasets.py, references.py, config.py, cli.py, routing.py, engine/routing.py, engine/behavioral.py, engine/judge.py, engine/scorers.py, engine/hooks.py, engine/sandbox.py, agent.py, selection.py, credentials.py, schema/evals.schema.json, schema/machine.schema.json, data/TEMPLATE.json, .github/workflows/reusable.yml, .github/workflows/skill-evals.yml, and examples/skill-with-a-device/evals/machine.yml. All CI checks on this commit are green (unit tests, unused-imports, sandbox machinery, throwaway-repo action run) — none of them exercise documentation-content accuracy, so they corroborate nothing about the two findings above.
Did not run: agent-instruction adherence (no CLAUDE.md/AGENTS.md in this repo); history/prior-changes (single-commit PR adding a new file; the two lightly-edited docs' recent history had nothing bearing on this change).
Not reconciled against any discussion on this PR.
rominf
left a comment
There was a problem hiding this comment.
Good direction for #9. It needs a rebase onto current main and a few factual fixes before merge.
Direction: one tick-list per grader is what #9 asked for, and it sits well beside usage.md. The cost is a third place that restates code facts, and it is already drifting: main gained the harness contract after this branch was cut, and #26 and #29 each change something this page states. Keep it lean, and point at usage.md for anything likely to move.
Before merge:
- Rebase on main and update for harnesses. The routing credential bullet (L139-142) names only
ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, but on main the host leg takes each harness's own variables (Copilot:COPILOT_GITHUB_TOKEN,GH_TOKEN,GITHUB_TOKEN; see harnesses.md). "That tag is the harness" (L230) now clashes with what "harness" means here, so say "that tag is the skillscope version that grades your skills". - I agree with the earlier reviews on the judge sentence (L168): judge.py also shows the final message and uses it for requirements about what the agent told the user. I also agree on "Whichever you use": the
api_keymapping andoptional/offexist only on reusable.yml. - Confirmed AgentHub's
sandboxpoint. Structural never reads that key (machine_planreturns only os/labels), and the compose file is checked only when a behavioral task is built. Move the bullet to Behavioral and say it applies only to the sandboxed leg.
Optional:
api_keyappears in reusable.yml's "inputs and no others" (L193), but it is a secret. AgentHub's summary calls this list verbatim-accurate. It isn't, but the fix is one word.- AgentHub's
noneitem is overstated as blocking. This page presentsrouting_roomas the CI input, and therenonedoes skip routing. A directskillscope routing --routing-room noneexits with a message that says what to do. Half a sentence covers it. - AgentHub's
errorgloss is a nit, as is "2xx" vs the code's 200-206 plus 429. - Whichever lands second among this PR, #29 (adds
prerequisitesto machine.yml, so L107's key list goes stale) and #26 (ungraded expectations vs "passes when every claim passes") has to update the checklist.
The bars a skill is held to were spread across usage.md, authoring-evals.md and the error strings the code raises, so there was nowhere to look before asking for a run, or after a red one. Collect them in one list, grouped by grader and written for someone meeting the harness for the first time, and link it from the README and both docs. Signed-off-by: John Lybeck <john.lybeck@amd.com>
5e885ae to
181abc4
Compare
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · 181abc4
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — no blocking findings
Full review of the whole change (4 files, docs only: docs/checklist.md new, plus link additions in README.md, docs/authoring-evals.md, docs/usage.md).
PR description and any ticket were not read or retrieved here, so what the change claims about itself is not judged. No ticket named in the material I had.
Blocking
None
Non-blocking
- The "model key reaches the run" checklist item leaves out workload identity federation, the other supported credential route —
docs/checklist.md:233-235(the "Whichever you use" list), anddocs/checklist.md:~163-167(Routing credential item)
The item says to mapapi_keyor usesecrets: inheritwithapi_key_secret. The code andskill-evals.ymlalso accept federation (federation_rule_id,federation_organization_id,federation_service_account_id, and related inputs at.github/workflows/skill-evals.yml:245-271).skillscope/credentials.pyresolves federation first, andskillscope/routing.py:168-183documents thatANTHROPIC_AUTH_TOKENcounts as a valid credential for routing. Here the checklist is narrower than the code. A reader on a federated runner following the checklist literally would think a key is required, and the checklist says federation cannot be combined with a custom base URL, which the "gateway key" item implies is the only gateway path. Fix: add a clause noting federation as an alternative onskill-evals.yml(and that it cannot be combined withapi_base_urlor custom headers), or link to the usage.md section. Confidence 85 · mechanical. - "Grades the skills you name, on one runner each" is slightly imprecise for
reusable.yml—docs/checklist.md(section 4,reusable.yml)
The workflow header (.github/workflows/reusable.yml:20-21,52-55) says structural is one job for all skills, while routing and behavioral are one runner per skill. Fix: say "routing and behavioral run on one runner per skill". Confidence 70 · mechanical.
Decisions for the author
- The checklist duplicates rules that also live in
usage.md,authoring-evals.mdand the code's error strings — tradeoff
A single list is easier to tick through, but it is a third copy of numeric limits (500 lines, 1024 characters, 3 positive and 2 negative cases) and of the input lists, which can drift. I checked these againstskillscope/structure.py(500, 1024, reserved substrings, name equals folder),skillscope/datasets.py(minimum 3 true and 2 false,TRIGGER_CASE_KEYSagainstNO_TRIGGER_CASE_KEYS, repo-wide ids,machine.ymlkeys) andreferences.py(2xx plus 429), and thereusable.ymlinput list is exact. They all match today. Confirm the duplication is intended, or consider one source of truth.
Positive signals
- The
reusable.ymlinput list is complete and exactly matches the workflow'sworkflow_callinputs, and the doc explicitly warns that the two workflows do not accept each other's inputs. - Explaining why
routing_roomhas no good default, and why a label no machine has makes a job wait forever, is useful, concrete guidance.
Deployment notes
None
What this covered
DEGRADED REVIEW: no fan-out was run. I reviewed in a single context, so this is not an independent multi-pass review, and I recommend re-running with fan-out if the stakes warrant it. I also did not read the skill's references/ files (rubric, scope-resolution, impact-analysis), so the standards were applied from the method text only. The CI rollup (statusCheckRollup) was not read, so whether CI passes is unknown. The source-reading passes (agent-instruction adherence, history, prior changes, code comments) did not run as separate passes.
Read completely: the full diff prw-base...HEAD (head 181abc4), the one commit message (it carries a Signed-off-by trailer), and the relevant code in the checkout, which was read via the checked-out tree. Specifically, I cross-checked the documented rules against skillscope/structure.py, datasets.py, credentials.py, routing.py, references.py, cli.py, and the reusable.yml and skill-evals.yml inputs. I did not execute the tests or the CLI, so nothing is scored 100.
Test verdicts: no tests are in scope, since the change is documentation only.
Not reconciled against any discussion here.
Adds
docs/checklist.md: every bar the three graders hold a skill to, in one list.SKILL.mdlimits fromstructure.py, the dataset shape and Tier 0 coverage fromdatasets.py, the four kinds of behavioral check fromengine/scorers.py, the pass mark and the two infrastructure checks fromrouting_gate.reusable.ymlandskill-evals.ymlapart and gives the former's full input list.routing_roomandinfra_pathsbelong only to the latter, which the current docs make easy to get wrong.workspaceand sends the prompt verbatim, and anerrorordegradedcase counts towards neither side of the bar.Closes #9
Verified with
skillscope structural --docs '*.md,docs/**/*.md': the 37 internal references across the four markdown files resolve.