Repository navigation
Skill Eval v1.1: automated evaluation for skill additions and updates #88
Description
Help an agent answer: what did this skill change fix, what did it break, and what remains untested?
This is a bounded delivery plan under the InferRSI roadmap, with three implementation issues. This parent tracks the overall outcome and closes after end-to-end acceptance; it adds no fourth implementation workstream.
Starting point and goal
Skill Eval v1 (#40) is accepted and closed. It provides real-agent execution, prepare/execute separation, deterministic task grading, controlled ordering and revision-bound evidence for a bounded pilot. That pilot did not establish incremental skill benefit.
v1.1 turns that foundation into a reusable, agent-operated workflow for skill PRs. It does not rebuild the runner or create a skill-version platform.
| Change | Comparison | Question |
|---|---|---|
| Add a skill | Without target skill vs new skill | Does it help on the covered tasks, and can it mislead? |
| Update a skill | Old skill vs new skill | What improves, regresses or remains unchanged? |
flowchart LR
A[Skill change] --> B[Plan]
B --> C[Authorize]
C --> D[Compare]
D --> E[PR report]
Planning makes no model calls. Execution needs explicit, revision-bound budget authorization. The report is generated and posted automatically, not manually copied by a maintainer.
Three implementation issues
| Order / implementation issue | Increment beyond v1 | Acceptance |
|---|---|---|
| 1. Strengthen deterministic grading (#99, completed) | Independent expected answers, negative/boundary cases and offline failure-path tests for the existing verifier and execution flow | Known bad outputs are rejected; incomplete evidence and execution errors cannot become task passes. Expected answers are not generated by the implementation being tested. |
| 2. Add a configurable comparison command (#100, completed) | Replace pilot-specific wiring with small skill/task configuration; support both comparison modes and generated JSON + Markdown reports | An addition and an update use the same entry point. Independent cases detect a deliberately degraded candidate; reports distinguish observed fixes, regressions, unchanged results and inconclusive outcomes. |
| 3. Automate PR planning and feedback (#101, completed) | Map changed skills/resources to tests, run with authorization, and update a revision-bound PR summary with evidence links | Uncovered changes are visible; a new revision invalidates old results; duplicate triggers do not duplicate paid execution; publishing can retry without rerunning evaluation. |
- Skill Eval: harden deterministic grading and offline failure handling #99 — grading confidence: accepted via merged fix: harden deterministic skill grading and offline failure handling #94 (
b3ef95a); retrospective acceptance record closed. - Skill Eval: add configurable skill addition and update comparisons #100 — accepted and closed with owner confirmation on 2026-10-23; feat: add configurable offline skill addition and update comparisons #109 merged as
48e0d5894117be7c57f8b4ce793c0ec986f219e8. Offline tooling acceptance only; closure evidence. - Skill Eval: automate PR evaluation planning and feedback #101 — accepted and closed on 2026-10-27; real addition/update execution and automatic retained-evidence PR feedback verified.
1 → 2 delivers a local tool; 3 integrates it into PR work. Detailed code changes and test inventories belong in the implementation issues.
What the report should tell us
Illustrative update report below: invented values, not measured results.
| Outcome | Example count | Review action |
|---|---|---|
| Fail → pass | 1 | Inspect the apparent fix and its evidence |
| Pass → fail | 1 | Investigate the observed regression before acceptance |
| Pass → pass | 3 | Covered behavior remains correct in these runs |
| Fail → fail | 1 | The case still fails; inspect whether the failure changed |
| Uncovered | 1 | Add a case or explicitly accept the evidence gap |
| Execution/grader error | 1 | Fix the evaluation; do not call this a skill failure or pass |
Each result links to its case, outputs and grading evidence. Discovery tests and tests that explicitly provide the skill are reported separately. Model/settings, revisions, coverage and usage accompany the report; timing is an observation, not an automatic performance claim. No single aggregate score hides regressions or missing evidence.
For additions, the same paired outcomes apply to without vs with the new skill. Passing both sides is not proof of added benefit. Any conclusion is bounded by the tested tasks and model configuration.
Keep maintenance small
- Reuse Git and existing reader revisions. No new skill version fields, version registry or checked-in old/new copies. Run metadata is generated automatically.
- Maintain tests, not paperwork. Keep the minimal skill/resource-to-task mapping with evaluation configuration. New task families may need new fixtures and correctness rules; unmapped changes are uncovered, not passed.
- Keep comparisons controlled. Freeze skill snapshots and relevant resources; use the same tasks, grader, model/settings and environment, with fresh sessions and balanced ordering. Show changes to tests/grading explicitly rather than silently changing the exam.
- Reuse existing tooling. No second model runner, dashboard, generalized resume service or plugin platform. Results and raw outputs are run artifacts with a documented retention window, not routine commits to main.
- Separate execution from publication. Untrusted PR content must not execute with publication credentials. Report-only PR integration does not create a required efficacy gate.
Authorship admission remains separate. This work does not automatically approve, merge, promote or rewrite skills, and does not add automatic evaluation to local Create/Update operations.
References and actual reuse
Updated 2026-10-23. Code reuse ≠ workflow inspiration ≠ an evaluated alternative. See #40 §9 for the original pinned readsets and limitations. No upstream benchmark score is an acceptance threshold here.
| Source | Concrete reuse / relevance | Status |
|---|---|---|
| InferRSI v1 + #94 | #109 imports CodexClient, export_bundle, run_trial, summarize, KnowledgeReader/safe_path, artifact helpers, and pilot PROMPT/SCHEMA/observed_tokens; command execution and process cleanup remain on the existing infra path. |
Actual internal code reuse; inspected imports. #109 was accepted under #100; see the closure evidence above. |
| Sibling pilot / npu-skill-factory | Fresh-session, frozen-verifier, event and cost-reporting methods informed v1. The old fixed no-skill/human/distilled controller was not imported. | Workflow adaptation; recorded reuse decision, retained plan. No external evaluation-platform source was copied according to that record. |
| Anthropic skill-creator | Paired comparisons, assertion-level evidence and separate discovery/trigger assessment. | Design reference, not imported runner code. Triggering is not task correctness. |
| SkillsBench paper / code; SkillLearnBench | Matched skill/no-skill tasks, artifact verification, and development versus further/held-out instances. | Methodology references; their task suites, containers and platforms were not imported. |
| Harness-aware plugin evals | Evaluate the actual agent harness and outputs, not just a model response or tool selection. | Workflow reference; no copied adapter or production dataset claimed. |
| Promptfoo Codex skill comparison | Configuration, comparison and report layer considered during design. | Discussed, not integrated; no verified integration proof recorded. #109 extends the existing Python path. |
| pytest / Hypothesis | Independent expected outcomes and negative/boundary/property-oriented tests. | pytest used; Hypothesis proposed, not adopted in the inspected dependency configuration. |
| agent-skills-eval @ f10ae2b | Declarative cases, deterministic tool assertions, per-assertion evidence; experimental artifact/verifier isolation. | Inspected design reference, no integration or copied code. MIT attribution must be preserved if code is reused. |
Bounded role of agent-skills-eval
Its stable path also uses with/without A/B. Borrow test authoring and evidence presentation, not another runner:
- Now: use its small case structure and local assertion approach as references for task-specific checks: expected behavior → observed result → evidence. No new assertion DSL.
- Skill Eval: automate PR evaluation planning and feedback #101: publish our existing JSON/Markdown; do not add its TypeScript runner or a dashboard.
- Later file/code tasks: consider its experimental pattern of snapshotting agent artifacts before isolated validators run. Validator-created files must not count as agent output. Not an additional Skill Eval: add configurable skill addition and update comparisons #100 requirement.
- Limits: returned tool calls do not prove execution; the stable default is single-call and the experimental runtime is not a ready Codex adapter. Keep InferRSI's missing/error/uncovered semantics; do not copy empty-assertion success or omitted-error aggregation.
References do not authorize new dependencies, paid trials or automatic promotion.
Close when
- All three implementation issues are accepted.
- One addition and one update complete the agent-operated path from plan to PR report, with explicitly authorized real-agent execution.
- Offline controls validate the specified grading rules and failure paths; a controlled degradation demonstrates that the end-to-end system can flag a regression, not merely finish execution.
- Missing coverage, incomplete runs and runner/grader errors are visible and cannot be counted as passes.
- Reports retain both evaluated revisions and configuration; stale publication and duplicate paid execution are covered by tests.
- Usage guidance explains how to run, interpret results and add a test without adding skill-version metadata.
- Closure includes a concise change diagram, evidence links and remaining limitations.
Success means reliable evaluation evidence and less manual coordination—not a guarantee that every skill improves every agent. New commits require renewed execution authorization; acceptance never depends on manufacturing a positive skill effect.
Consolidated acceptance
Completed 2026-10-27: v1.1 acceptance, change diagram, verification receipts and limitations. This supersedes historical readiness statements saying #101/#88 remain open or a usable hosted provider is not yet available. The bounded workflow is accepted; wider task-family coverage, private benchmarks, skill promotion and subject-PR merges are not implied.
Activity
fix: retain safe Skill Eval publisher HTTP diagnostics (#163)
ci: add model-free Skill Eval plan dispatch (#165)
fix: retain safe publisher permission requirement diagnostics (#168)
feat: add offline bounded Messages agent contract (#174)
Dong1017 commentedon Oct 27, 2026
Skill Eval v1.1 accepted: bounded end-to-end workflow complete
All three implementation deliveries are accepted: #99 grading, #100 configurable comparisons, and #101 automatic PR integration.
flowchart LR G["Independent grading: #99"] --> C["Configured addition and update: #100"] C --> P["Plan and explicit finite authorization: #101"] P --> E["Real agent comparison"] E --> R["Retained evidence and automatic PR feedback"] R --> A["Addition: wrong to correct"] R --> U["Update: correct to correct"]Acceptance evidence
fix=1. Retained bytes and receipt hashes were independently inspected as recorded in Skill Eval: automate PR evaluation planning and feedback #101.unchanged_correct=1, with no missing pair or inconclusive outcome. This neutral result satisfies the workflow goal; no improvement is claimed.Limits retained
Only one fixed holdout case was exercised per real comparison mode. The mapping currently covers two skills; 70 addition and 65 update snapshot paths are explicitly uncovered. This is not full-library capability coverage, discovery effectiveness, hardware optimization acceptance or automatic skill promotion. Historical incomplete update execution remains recorded and was not counted as a pass. Final update artifact bytes were not independently downloaded in this acceptance pass; trusted publisher/job receipts were verified. Actual provider billing is not asserted; reservations are not invoices. Evidence expiry and new-revision authorization rules remain in force.
Decision: all seven parent closure criteria are satisfied within the agreed v1.1 scope. Close #88 as completed. Broader task-family/private benchmark coverage is future work, not an added prerequisite or an implied delivered feature. Subject PR merge readiness and #190's retrieval-performance campaign remain separate.