Skip to content

Let a repo set the dataset bar its skills are held to - #46

Open
johnl-amd wants to merge 1 commit into
mainfrom
configurable-dataset-bar
Open

johnl-amd wants to merge 1 commit into
mainfrom
configurable-dataset-bar

Conversation

@johnl-amd

Copy link
Copy Markdown
Collaborator

Tier 0 is three positive evaluations and two negative, as constants in datasets.py. That is the right bar for a catalogue that starts with this harness, and the wrong one for a catalogue that predates it. A repo adopting skillscope over fifty skills written before these checks existed cannot produce two hundred and fifty prompts before its first green run, so the choice on offer is to run none of the checks at all. The folder, frontmatter and reference checks are worth having on day one, and the dataset requirement was holding them hostage.

  • --min-positive and --min-negative on structural, routing and behavioral, with matching min_positive / min_negative inputs on both reusable.yml and skill-evals.yml
  • Blank means the existing default, so nothing changes for a repo that passes neither
  • Zero for both stops requiring a dataset. A skill that ships one anyway is still read against whatever the bar is, because "not required of every skill" is not "unread where present"
  • The applied bar prints at the top of every structural run, passing or failing

Where the bar may be set

It is a flag and nothing else: no file in the repo under test, none inside a skill, and no environment variable, unlike SKILLSCOPE_SKILLS.

The bar a skill is measured against is not the skill author's to set. Were it a file beside the skill, one pull request could add a thin dataset and lower the number judging it, and a reviewer would have to notice the second change to understand the first. On the command line it lives in the workflow, which is a path a repo can put behind branch protection and code owners. An environment variable would be set by whoever administers a self-hosted runner, who is not the person reviewing the workflow diff.

TestTheDatasetBarIsTheRepos asserts that property rather than leaving it to be assumed, so a later change that adds a file or an env fallback fails a test instead of quietly widening who can lower the bar.

How to test

skillscope structural --min-positive 0 --min-negative 0   # datasets not required
skillscope structural --min-positive 1 --min-negative 1   # a lower bar
skillscope structural                                      # unchanged, 3 and 2

Boundaries worth poking: one bar at zero and the other above it still requires a dataset (only both at zero turns the requirement off), and a negative or non-integer value is refused in the preflight rather than part-way through a run.

@rominf rominf left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Automated review · a2cc6e5

This automation never files a GitHub approval, so no approving review will appear here whatever the outcome — the merge decision stays with a human reviewer.

Review — needs work

DEGRADED REVIEW: single-context pass, no fan-out workers were spawned. Full review of the whole change (prw-base...HEAD, one commit).
No ticket named. The description matches the change's stated scope.

Blocking

  • (new) YAML # comments inside args: >- folded scalars become literal arguments and break every structural run — .github/workflows/reusable.yml:301-310, .github/workflows/skill-evals.yml:369-377
    In a block scalar, # lines are content, not comments. The action passes SKILLSCOPE_ARGS through shlex.split (bootstrap/launch.py:104), which does not strip comments. Splitting the parsed workflow value yields tokens '#', '`!=', 'rather', 'than', … and python3 -m skillscope structural '#' fails with skillscope: error: unrecognized arguments: #. The structural job in both workflows would fail on every run, including runs where no min_* input is set. The comment's backticks and quotes risk further shlex errors. Nothing tests the workflow wiring, so the test suite cannot catch this.
    Confidence 100 · logic · Fix: delete the # lines from both args blocks; put the explanation in a real YAML comment above the args: key, or in the input's description.

Non-blocking

  • (new) The "applied bar is printed at the top of every structural run" behaviour has no test — skillscope/cli.py:302-325
    Both the bar-on and bar-off message branches are untested, as is the CLI-to-config wiring (_configure, _add_dataset_bar_arguments). The new tests call config.build and datasets.tier0_errors directly; nothing runs --min-positive through argparse. The description promises this print as a feature.
    Confidence 75 · mechanical · Fix: add a CLI-level test running structural with --min-positive 0 --min-negative 0 and one without flags, asserting on the printed line.

Decisions for the author

  • Default 0 is accepted for one bar but not both — tradeoff
    With only one bar at zero, a skill with no dataset still fails (test_one_bar_still_standing_still_requires_a_dataset); both at zero is the only "off" switch. Coherent and tested, but confirm it is intended.

Positive signals

  • The bar is settable only via CLI flag, with no file or environment fallback, and tests assert that property rather than relying on a comment.
  • _bar rejects negative and non-numeric values with a flag-named error, and an existing dataset is still read when the bar is off.
  • stdout is flushed before the stderr failures, so the bar line appears first in the CI log.

Deployment notes

  • Two new optional inputs, min_positive and min_negative, on both reusable workflows, and two new flags on structural, routing and behavioral. Defaults are unchanged (3 and 2).
  • MIN_POSITIVE_CASES and MIN_NEGATIVE_CASES are removed from datasets in favour of config.DEFAULT_MIN_*; no other in-repo users found, but external importers would break.

What this covered

Every changed file read completely (7 files, prw-base...HEAD at a2cc6e5), plus the consumers of the changed symbols (bootstrap/launch.py, action.yml, tier0_errors callers), read from the prepared checkout. The one commit carries a Signed-off-by trailer (check_signoff.py exit 0).
Test verdicts: the new TestTheDatasetBarIsTheRepos tests pass. The full suite shows 10 failures and 68 errors, identical in count at prw-base, so environmental. No revert or single-branch mutation was run, so neither test verdict is established. The blocking finding was reproduced with shlex.split and the CLI against the actual workflow value, and re-checked by the driver against the head commit.
Did not run: fan-out workers, the history and prior-changes passes, the CI rollup read, Step 7 design workers; agent-instruction and code-comment passes were inline, not independent. Not reconciled against any discussion here.

Tier 0 was three positive evaluations and two negative, as constants. That
is the right bar for a catalogue that starts with this harness and the wrong
one for a catalogue that predates it: fifty skills written before these
checks existed cannot produce two hundred and fifty prompts before their
first green run, so the choice on offer was to run none of the checks at
all. The folder, frontmatter and reference checks are worth having on day
one, and they were being held hostage by the dataset requirement.

--min-positive and --min-negative lower the bar and raise it again as the
datasets land. Zero for both stops requiring a dataset; a skill that ships
one anyway is still read against whatever the bar is, because "not required
of every skill" is not "unread where present".

Where that bar may be set is the point, so it is a flag and nothing else.
No file in the repo under test, none inside a skill, and no environment
variable: the bar a skill is measured against is not the skill author's to
set, and a file beside the skill would let one pull request add a thin
dataset and lower the number judging it, leaving a reviewer to notice the
second change to understand the first. On the command line it lives in the
workflow, which is a path a repo can put behind branch protection and code
owners. Two tests assert that property rather than trusting it. The applied
bar is printed at the top of every structural run, passing or failing, so a
report says what it was holding skills to.

Signed-off-by: John Lybeck <john.lybeck@amd.com>
@johnl-amd
johnl-amd force-pushed the configurable-dataset-bar branch from a2cc6e5 to 25b904f Compare November 18, 2026 08:11
@johnl-amd

Copy link
Copy Markdown
Collaborator Author

All three addressed in 25b904f (amended, force-pushed).

Blocking, the # lines in the folded scalars. Confirmed exactly as described: parsing the workflow and splitting the value gives ['#', '!=', '', 'rather', ...], and structural dies on the first token. Moved the explanation to a real YAML comment above the args: key in both files. Re-parsed both after the fix, the structural args now splits to [] with blank inputs in reusable.yml and ['--external'] for the external references step, nothing stray left.

Non-blocking, the bar line had no test. Added test_the_applied_bar_is_printed_whether_it_is_on_or_off, which runs structural through cli.main with no flags and with --min-positive 0 --min-negative 0, asserting on both branches of the printed line. Good catch that every other test in the class goes in at config.build and so would keep passing if the flags stopped reaching it. Mutation checked: pulling min_positive / min_negative out of _configure's settings makes the new test fail and leaves the rest of the class green.

Decision, one bar at zero. Intended. Both at zero is the only off switch, because "this repo requires no dataset" is a different statement from "this repo wants a low bar", and only the first should turn the requirement off. A repo that genuinely wants zero positives and some negatives still has a dataset requirement, which is what test_one_bar_still_standing_still_requires_a_dataset pins.

On the suite: ran it at the head commit and at the parent, 68 failures and errors each, identical test ids both times, so they are environmental here and not from this branch.

@johnl-amd
johnl-amd requested a review from rominf November 18, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants