Repository navigation
Conversation
johnl-amd
left a comment
There was a problem hiding this comment.
Review — needs work
Full review of the whole change (a single-file, 16-line addition to .github/workflows/skill-evals.yml). No ticket is named; the PR description is the only stated intent.
Blocking
- The description cites a doc file that does not exist — PR description, final paragraph:
docs/skill-quality.md.
The PR justifies not touching any documentation by saying "docs/skill-quality.mdalready lists the missing artifact as a known gap under 'What is not available yet'." There is nodocs/skill-quality.mdin this repository (thedocs/tree on this branch contains onlyauthoring-evals.md,harnesses.md, andusage.md), and a repo-wide code search for bothskill-qualityand the phraseWhat is not available yetreturns zero hits. The claim that this gap is "already" documented elsewhere is false, which means the stated reason for leaving docs untouched doesn't hold.
Confidence 85 · documentation · Fix: correct the description — either point at wherever this gap actually is tracked, or drop the claim and add a line toREADME.md/docs/usage.mdnoting the artifact now exists (and that it didn't before).
Non-blocking
None
Decisions for the author
- Uploading the whole
.skillscope/directory vs. named paths — tradeoff.
The existingroutingjob's artifact step lists exact paths (routing-report.json,routing-logs/). The new steps instead glob the entire.skillscope/directory. That's more future-proof against whatever skillscope writes there next, but less precise than the routing job's approach right next to it, and would silently sweep in anything else ever written under that path. Worth confirming this breadth is intentional rather than an inconsistency with the sibling job. - Behavioral transcripts are richer and uncapped compared to routing's — tradeoff, see Deployment notes.
Positive signals
- The new steps are a faithful, minimal mirror of the existing
routingjob's upload step — sameif: always()reasoning, sameif-no-files-found: warn, and artifact names correctly extended with the matrix keys (skill/os/environment) so two legs never collide on one name, which is required sinceupload-artifact@v4fails on a duplicate name within a run. - The claim that this "changes no behaviour" holds up: traced
cmd_behavioral→engine/behavioral.py::run(), which already writes the JSON report to.skillscope/runs/and the inspect transcript to.skillscope/logs/by default (log_dir=log_dir or str(Path(".skillscope") / "logs")) with no new CLI argument introduced by this change.
Deployment notes
- This repo is public, so these artifacts become world-downloadable the moment a run completes. That risk is not new — the
routingjob already publishes full inspect transcripts the same way — but it is worth noting that behavioral runs have no tool-call cap (routing caps at--max-tool-calls/--max-inspection-calls), run to a much higher message limit, and are not scrubbed or redacted anywhere in the codebase before being written or uploaded. Credentials themselves are not at risk (resolved only into environment variables and GitHub's own masked$GITHUB_ENV, never into the report or transcript, confirmed by readingcredentials.pyand the harness env-wiring), but whatever a test case's tool calls touch (file contents, shell output, etc.) lands in a now-public artifact exactly as routing's already does. Flagging for awareness, not as a defect — this PR is consistent with precedent already accepted in this repo.
What this covered
Read the full diff and the complete current .github/workflows/skill-evals.yml on commit 34f2e41, plus README.md, docs/usage.md, .gitignore, skillscope/cli.py, skillscope/engine/behavioral.py, and skillscope/engine/models.py to verify the PR's factual claims about default output paths and credential handling. A separate pass, run by an independent worker with no exposure to the PR description or this report, specifically investigated whether secrets or other sensitive content could end up in .skillscope/ and whether a redaction mechanism exists; its findings are folded into Deployment notes above.
Did not run: history/prior-changes passes beyond git log on the touched file (no earlier PR or review thread touches this artifact-upload pattern); agent-instruction-adherence (no CLAUDE.md/AGENTS.md in this repository). No CI check in this PR's run validates the workflow YAML itself (the file is a reusable workflow invoked by callers, not self-triggering), so YAML correctness here rests on manual reading rather than a check.
One caveat on independence: the driver (this review) read the PR's title and description together with the diff in its first lookup, before splitting into separate passes, so the direct read of the workflow diff itself was not blind to stated intent — only the delegated security-impact investigation was. Given how small and mechanical this diff is (a straight copy of an existing, adjacent block), the risk from that is low, but it is disclosed here rather than left implicit. Not reconciled against any other discussion of this PR.
|
On the blocking finding: the doc does exist, just not in this repo. It's
So the claim holds, the description just never said which repo, and a repo-wide search here was always going to come up empty. I've qualified it. The follow-up belongs on the rocm-cli side, that bullet goes once this merges. On The public-artifact note is fair and matches what routing already does, so nothing changes here on that. |
The routing job already uploads its report. The behavioral jobs do not, so when a behavioral case fails, the only record that outlives the run is the summary table: a case id, an expectation, and a truncated judge sentence. That is not enough to tell a skill bug from an environment one. A case that fails in CI and passes everywhere else leaves nothing to compare, because the transcript of what the agent actually did is written to .skillscope/ on the runner and then discarded with it. skillscope already writes both halves there by default, the JSON report under runs/ and the inspect transcript under logs/, so this adds no arguments and changes no behaviour. It keeps what is already produced. Artifact names carry the matrix keys, since one job uploads per skill and OS, and per environment on the scoped leg. Signed-off-by: John Lybeck <john.lybeck@amd.com>
34f2e41 to
d628914
Compare
rominf
left a comment
There was a problem hiding this comment.
🔴 Automated review · d628914
This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.
Review — needs work
Full review of the whole change (1 file, 16 added lines, one commit).
No ticket named. The description says both behavioral jobs now upload their report and transcripts and "adds no arguments, changes no behaviour".
Blocking
- Both new upload steps will silently upload nothing, because
.skillscope/is a hidden directory (new) —.github/workflows/skill-evals.yml:642,.github/workflows/skill-evals.yml:707actions/upload-artifact@v4resolves to v4.4.0 or later. Since v4.4.0 it excludes hidden files by default: anything starting with., or inside a directory starting with..- Naming the hidden path explicitly in
pathdoes not bypass the exclusion; the fix isinclude-hidden-files: true. if-no-files-found: warnkeeps the leg green with only a warning, so the PR's goal (keeping transcripts after a failed leg) is not met and nothing visibly fails.- Basis: upload-artifact v4.4.0 release notes and issues #602 / #610. Not run on a runner.
- The existing routing upload avoids the hidden directory: it passes
--output routing-report.json --keep-logs routing-logsand uploads those visible paths (skill-evals.yml:566-581);reusable.yml:340-352andreusable.yml:380-406follow the same pattern. The description's argument that naming the paths and taking the directory cover the same files holds for the files but misses the hidden-directory filter.
Confidence 85 · mechanical · Fix: addinclude-hidden-files: trueunderwith:in both steps, or follow the routing pattern (add--output behavioral-report.json --keep-logs behavioral-logsto the behavioralargsand upload those visible paths). Considerif-no-files-found: errorso a leg that uploads nothing is not silent.
Non-blocking
None
Decisions for the author
- Uploading raw agent transcripts as artifacts — tradeoff
For: the transcripts are what tells a skill bug from an environment bug. Against: the.evallogs can carry whatever the agent saw or printed, including, on the scoped legs, environments with their own secrets; artifacts are readable by anyone with read access to the repo. Confirm that is acceptable, or consider a shorterretention-days. - Behavioral outputs now have two conventions — non-blocking-improvement
Routing uses explicit visible paths; the new steps rely on tool defaults under a hidden directory. Explicit flags in both would be uniform and also resolve the blocking finding.
Positive signals
if: always()on the upload, so failed legs still keep their transcripts.- Artifact names include
matrix.osandmatrix.environment, so matrix legs do not collide under v4's unique-name rule.
Deployment notes
None
What this covered
- Every line of the diff (
prw-base...HEAD, head d628914), plus the surrounding job definitions inskill-evals.yml, thereusable.ymlrouting and behavioral jobs, and the--output/--keep-logsoptions inskillscope/cli.py. - The PR description was read only after the findings were settled; no comments or reviews were read.
- Sign-off check ran:
checked: 1, missing: []. No tests in scope (workflow-only change). - DEGRADED REVIEW: no fan-out; the change is one 16-line workflow file, reviewed in a single pass.
- Did not run: CI rollup read; design-question workers (considered inline); history and prior-changes passes (nothing earlier touches the added lines).
- Not reconciled against any discussion here.
The
routingjob already uploads its report. The behavioral jobs don't, so when a behavioral case fails the only record that outlives the run is the summary table: a case id, an expectation, and a judge sentence truncated at 160 characters.That isn't enough to tell a skill problem from an environment one. A case that fails in CI and passes everywhere else leaves nothing to compare against, because the transcript of what the agent actually did is written to
.skillscope/on the runner and thrown away with it.Concretely, chasing one such case this week meant re-running CI repeatedly to re-read a summary line, while the same case passed 29 times locally. The transcripts that would have settled it were produced on every one of those runs and discarded each time.
skillscope already writes both halves there by default, the JSON report under
runs/and the inspect transcript underlogs/. So this adds no arguments, changes no behaviour, and keeps what is already being produced.behaviorandbehavior-scopedeach gain anUpload behavioral reportstep, matching the oneroutingalready has.if: always(), because a failed leg is exactly when the transcript is wanted.pathis the.skillscope/directory rather than named files. Routing can name its files because it passes--output routing-report.json --keep-logs routing-logs; the behavioral calls pass neither, so their output lands in the defaults,.skillscope/runs/and.skillscope/logs/. Those two are the only paths anything writes under that directory, so naming them and taking the directory cover the same files.if-no-files-found: warn, same as routing, so a leg that produced nothing doesn't fail the run.ROCm/rocm-cli'sdocs/skill-quality.mdlists this missing artifact as a known gap under "What is not available yet". That bullet can go once this lands.