Skill Eval: actionable, revision-bound review guidance for skill and knowledge PRs #215

Goal

Make Skill Eval answer a practical review question: what does the current evidence support merging, what needs correction, and what remains untested? Deliver advisory, evidence-linked guidance—not automatic approval or a new required paid CI gate.

Refs #88 and #101 (accepted workflow foundation), #91 (coverage inventory), and #179 (evidence-integrity corrections). This is a bounded follow-on, not a reopening of their accepted scope or a transfer of existing ownership.

Starting point and gap

#88 acceptance established automatic addition/update comparisons and retained PR feedback. Each real comparison exercised one fixed holdout; a neutral update was valid. That acceptance explicitly did not establish whole-library efficacy or subject-PR merge readiness.

The current mapping names inference-benchmark-design and audit-fixed-workload-claim, both using the audit case file. The publisher reports outcomes and coverage gaps but does not explain their consequence for a PR decision. PLANNED, skipped execution, expired evidence and results for old revisions are not quality passes. The workflow's path filter does not cover every knowledge-only path.

Reuse the existing architecture

Verified source surfaces: .github/workflows/skill-eval-pr.yml, tools/skill_eval_pr.py, mapping, comparison controller, and usage guide. Reconcile these against the implementation's exact starting SHA.

flowchart LR
    A["PR scope and mapped cases"] --> B["Existing frozen plan"]
    B --> G["Owner-enabled label + eligibility + remaining PR budget"]
    G --> C["Revision-bound authorization and paired execution"]
    C --> D["Retained evidence and trusted regrade"]
    D --> E["Advisory review guidance and explicit gaps"]
    E --> F["Independent review and authorized merge decision"]

Keep prepare/authorize/execute/pack/publish, skill_eval_compare, the maintained mapping and one identifiable PR comment. Extend trusted report derivation and presentation; do not build a second runner, registry or dashboard. Preserve execution/publication credential separation and publication-only retry.

Bounded delivery

Phase 1 — Make existing mapped PR evidence decision-useful

Use #159 and #160 as the initial addition/update subjects, subject to their current state and exact revisions. If either is no longer suitable, record a specific replacement before using it.

• Show changed scope, selected cases, uncovered/unsupported paths, both compared SHAs, trusted implementation, plan/exam identity and retained evidence availability.
• Derive a concise advisory disposition with reasons and next actions: address observed regression, obtain missing evidence, or covered evidence supports proceeding to independent review. Existing failures remain visible. A fix does not cancel a regression; correct→correct is not incremental benefit.
• Never issue a blanket “approved” verdict. Uncovered scope needs a case or explicit reviewer disposition; no arbitrary all-files-covered requirement. Content/source review, authorship admission and merge authority stay separate.
• A new base/head or relevant plan input invalidates applicability. Bind guidance to the exact existing plan identity; do not silently reuse old runs. Missing, unexecuted, stale, expired, erroneous or incomplete evidence must not become positive quality evidence.
• Clarify the documented entry path for knowledge-only PRs: outside the supported scope must be visible, not mistaken for an eval pass. Any automatic-trigger expansion must name its bounded paths and receive focused tests; do not blanket-trigger all repository files.

Phase 2 — One justified coverage extension, not a library campaign

After Phase 1, select at most one currently unmapped task family from an actual pending skill/knowledge PR. Record the target PR, intended behavior, source/skill paths, independent expected outcomes, negative/boundary cases and supported grader before implementation. Do not force an unrelated task through the audit grader. If a new trusted task/grader adapter is necessary, record that bounded scope first; wider coverage remains separate under #91.

Owner-approved extension — label-gated automatic paid execution

The owner requested automatic paid evaluation for suitable PRs, with a USD 200 cumulative budget per PR. A dedicated opt-in label is the proposed control (for example, skill-eval:execute; final name to be confirmed against existing labels). This records the delivery requirement; the switch is not implemented or enabled by this issue edit.

The following are proposed implementation and acceptance constraints for that requirement, not a claim that the owner prescribed this exact design:

• Define suitability using supported changed scope, meaningful mapped cases, a trusted grader/provider configuration, and a current frozen plan. A label alone does not make an unsupported PR executable or quality-approved.
• Only an owner-authorized operator may enable the switch; verify the label action and current permission through trusted repository metadata. Do not treat arbitrary PR text, an author's request, or an untrusted event payload as execution authorization. Keep secrets away from untrusted PR code.
• Once enabled, automatically prepare and authorize an eligible exact-revision plan within the remaining budget, then execute and publish real evidence. Head/base/plan changes require fresh eligibility and binding, but do not reset the PR's spending allowance. Duplicate events, workflow reruns, label removal/re-addition and publication retries must not duplicate paid work.
• Account against the repository + PR number: actual charges plus outstanding/conservatively unresolved reservations across all revisions and retries must never exceed USD 200. Reserve an enforceable worst-case amount before calls, including output-token limits and the configured provider's applicable rates; concurrency must not overspend. Unknown costs, unavailable accounting, insufficient remaining budget or unsafe bounds stop execution with a visible reason. An estimate alone is not a hard cap.
• Removing the label, closing the PR or exhausting the budget prevents new calls. Re-check authorization before subsequent calls; already in-flight charges cannot be undone. Re-enabling does not replenish the allowance. Additional budget or a materially different provider requires explicit owner authorization.
• Show the label/eligibility state, actual execution state, spent/reserved/remaining USD and blocked reason in the existing PR report. An unsupported, stale or unexecuted plan stays “insufficient evidence,” not PASS.

Acceptance

• Offline controls cover fix, regression, unchanged-correct, unchanged-incorrect, missing pairs, grader/runner errors, uncovered scope, no selected cases, stale revisions and expired/missing artifacts. No aggregate score hides a regression or evidence gap.
• A controlled degraded candidate produces actionable negative guidance; a prepared-only run cannot support a positive recommendation. Publication retry makes no new model calls.
• Before relying on results, reconcile relevant fix: run state and comparison verdicts can report support the artifacts do not establish #179 plan/verdict-integrity items with the then-current code, tests and merged fixes. Record remaining blockers; do not assume every historical defect remains present or require unrelated lifecycle work merely because fix: run state and comparison verdicts can report support the artifacts do not establish #179 is open.
• Demonstrate Phase 1 on the selected addition/update PRs with exact revision-bound reports and an explicit reviewer decision rationale. Real runs must use revision-bound authorization, either the implemented owner-enabled label policy within the USD 200 per-PR ceiling or a separately approved exact-plan execution. Until that mechanism is implemented and verified, do not treat this issue edit as a live execution switch. Negative or inconclusive results can complete the demonstration without authorizing a subject PR merge.
• Validate label-gated execution with offline controls for unauthorized label actions, duplicate events, concurrent reservations, head changes, removal/re-addition, cancellation, unknown charges and budget exhaustion. Demonstrate an authorized real run and evidence publication without exceeding the cumulative PR cap; report any live-validation prerequisite explicitly.
• Record the Phase 2 selection and its bounded acceptance, or explicitly defer it with reason and destination. Do not claim expanded coverage if deferred.
• Update the existing usage guide, including obsolete readiness statements, and record revision-scoped independent review plus applicable offline/CI results.

Boundaries and ownership

One accountable issue owner coordinates implementation and independent review; no existing claim is implied. Confirm active claims before editing shared evaluator files. Preserve #91's separate metadata/content work.

The owner-approved automatic-execution scope is limited to suitable, explicitly enabled PRs and USD 200 cumulatively per PR. Publication of this requirement does not itself enable a label, dispatch a run or configure credentials. Provider/permission or protection changes, automatic promotion, merging and closing subject PRs remain outside this issue's execution authorization. No universal efficacy threshold, mandatory positive effect, or full-library evaluation is required. Close only with a scoped delivered/verified/limits summary and applicable owner acceptance.

Accepted bounded delivery and remaining limits: #215 (comment)