Repository navigation
Skill Eval: automate PR evaluation planning and feedback #101
Description
Parent: #88 · Delivery 3 of 3 · Depends on #100's local comparison/report contract
Connect the local evaluator to skill PRs without making maintainers copy reports.
flowchart LR
A[PR change] --> B[Plan]
B --> C[Authorize]
C --> D[Evaluate]
D --> E[PR report]
F[New revision] --> B
Planned increment
| Capability | Required behavior |
|---|---|
| Automatic planning | Identify changed skills/resources using a small maintained case mapping. Show selected cases, uncovered changes and estimated calls/limits; planning makes no model calls. |
| Authorized execution | Bind each execution to explicit approval of the frozen plan, PR head, compared revisions and budget. A new revision requires renewed authorization. |
| Automatic feedback | Generate/update one identifiable PR summary with outcome differences and evidence links. No manual report copying. |
| Freshness and retry | Mark old results stale after relevant input changes. Duplicate triggers do not duplicate paid work; publication retry reuses results without reevaluation. |
| Separation | Execute untrusted PR content without publication credentials; use trusted code to publish validated report data. |
Detailed trigger, authorization and artifact-retention choices should be finalized against the delivered local comparison interface, not by adding a new orchestration platform now.
Close when
- A skill addition and an update select the appropriate comparison; unmapped changes are visibly uncovered, never passed.
- New head/plan changes invalidate prior authorization/results; stale jobs cannot replace a current report.
- Offline integration tests cover duplicate triggers, publication failure/retry and untrusted report handling without real model calls.
- The tool publishes a revision-bound PR report automatically from retained evidence, with a documented retention window.
- The parent Skill Eval v1.1: automated evaluation for skill additions and updates #88 end-to-end exercise covers one addition and one update with separately authorized real-agent execution and automatic PR feedback. Neutral or negative results are valid outcomes.
- Closure records a concise change diagram, verification receipts and remaining limitations.
Boundaries
Report-only integration, not a required efficacy gate or automatic merge/promotion. Authorship admission remains separate. No new skill-version fields, manual result ledger, dashboard, runner, or automatic evaluation of local Create/Update. Issue creation grants no model budget, credentials or repository-security changes.
Activity
Dong1017 commentedon Oct 23, 2026
Next delivery after #109
The local comparison implementation is now on main at
48e0d58. #100 is accepted and closed with explicit owner confirmation (2026-10-23); its offline closure evidence is recorded. #101 is now the remaining implementation slice under #88.tools/skill_eval_compare.pyanddocs/skill-eval-comparison.mdat the merged revision; identify the existing plan hash, execute and report interfaces before designing glue.No new runner, skill-version registry, dashboard or required efficacy gate. Keep the PR-facing layer thin. Real-agent addition/update end-to-end exercises still require a separately fixed budget and authorization; this comment grants neither paid execution nor new credentials/settings. #88 remains open.
Dong1017 commentedon Oct 25, 2026
Issue owner and remaining delivery plan
Acting Issue-level owner: Codex, continuing #101 across its delivery PRs, under the user's explicit delegation. Preserve the three implementation workstreams under #88. #101/#88 remain open for real addition/update flows and automatic feedback.
Software integration is merged: #118 was squash merged as
db82ec8346364667b90f4fa97fdb190150141cf0, from accepted headd03266f59c0c2fee253090f1ff08b2f2d9867305ontobcc37c55f01cbaf1c42def8d6a3c8d6f5502d55f. The resulting tree exactly matches the calculated integration tree; main's 90-file delta and the ten-file delivery increment have no overlapping paths. The workflow and maintained mapping are now on trusted main. #145 and #146 are already merged; no duplicate merge or repair is needed.The independent reviewer approvals were not authored by this Issue owner. The squash operation used the accepted expected head, real DCO and truthful agent attribution. No merge permission for another PR, model budget, runner/credential/security change or Issue closure results from software integration.
Remaining execution plans are proposals, not approvals. Actual PR preparation must freeze both plans independently, including actual PR/base/head, trusted implementation, bundle/case/grader/schema/mapping/adapter identities and canonical hashes. Obtain explicit approval of each exact plan/version/full budget before any real call or evaluation dispatch. Proposed per-PR settings remain
gpt-6.1-sol / xhigh, one repeat, two calls, 180 seconds/trial, 600 seconds/comparison and observed-token stop 20,000; cost is unknown and the stop is not a provider hard monetary/token cap. The owner has confirmed both retained subject-source authors asgpt-6.1-sol / xhigh; that statement is source attribution, not evaluation authorization. Disclose the current integration author's actual settings separately when confirmed; unknown values remain unknown.Current model-free acceptance preparation — 2026-10-26
The owner explicitly requested continuation from current main and preparation of one real skill-addition PR plus one update PR, followed by separate exact-plan approval. The latest handoff and this plan have been read. Preparation starts at
a4a0fbd62defe2bc53ab090e1fa50bd7139663a6using the merged planner, executor and publisher.audit-fixed-workload-claimsource; regenerate all projections from the current canonical registry, preserving existing owners.inference-benchmark-design, preserving resources/provenance; regenerate source-pinned projections.9b55f0e112fd118be0897e3c4583d92ee333e52dand01d7fcc4dd733b18fcabe5fba270568b130e894ewere authored with Codex /gpt-6.1-sol / xhigh. This is owner-provided metadata, not an independent runtime attestation. The owner separately confirmed this chat's current integration settings asgpt-6.1-sol / xhigh. Both source and integration authorship declarations are now supplied truthfully against the actual PR heads.No new model call or evaluation dispatch is authorized by this preparation. Keep #101/#88 open for both real reports and independent acceptance.
Actual frozen subjects and current readiness — 2026-10-26
03d8a513a7cdc4a921b38dd8d5442555f887a0f730d41b2bd3413169400222ddb4bd04b477166c78d3a28bd895716040fb9196bce51b74ddcb8ab69798b51f3ed83ea5a84ff3d1690913f9d52039a66a83a00bcdbc3183a10c9c3b7337481b1709880d9017fc4e1bBoth automatic plans bind trusted implementation/base
0c78a7e257dfaa3573b77581a39539e8a3a5da4c, the unchanged mapping/case/grader/schema, complete selected bundles, modelgpt-6.1-sol / xhigh, and independently: one comparison, repeat1, max2 scheduled calls, 180 seconds/trial, 600 seconds/comparison, observed-token stop20,000. The token threshold is checked between trials; cost is unknown and it is not a provider hard cap. No prior suggestion or author confirmation authorizes these budgets.Both retained plan archives were downloaded, archive digests matched their Actions artifact digests, and the existing loader validated every frozen file/controller plus comparison hashes. Preparation from the a4a0fbd source base preserved all current owners; later main changed only four Flare paths with zero subject/evaluator overlap. The actual automatic plans include those full-snapshot differences as uncovered; uncovered changes are never counted as passes. Local focused catalog/index/library/reader checks, generator checks and changed-test Ruff/whitespace passed as recorded in the PR bodies. Automated review summaries found no findings, but are not independent exact-head LGTMs or whole-Issue acceptance.
New observed blocker: both automatic publisher jobs returned HTTP403 on POST to the PR conversation (addition, update). Logs declare Issues write permission; the underlying refusal reason is not established. The current CLI permission cannot read repository workflow policy. No permission/security change, synthetic green, manual outcome copying or deterministic rerun is proposed. No automatic plan/report conversation receipt exists; plan-only artifacts are retained and are not outcome reports.
The existing authorized execution environment linked to the trusted workflow/artifact publisher remains unconfirmed. An owner question about that channel is pending; local login alone is insufficient. Both plans are frozen preparation proposals with execution blocked, not approved executable plans. Exact runtime/publication readiness must be settled before requesting/consuming execution approval; changes to frozen/live identities require new plans and approval. No authorization/reservation or execution job ran, no evaluation/model call or evaluation dispatch occurred, and no attempt was consumed for these subjects. The separately consumed historical private probe remains untouched and unpublished.
Next: resolve the observed automatic publication refusal through the owner and confirm an already authorized usable runtime/publication channel; then refresh each exact identity, obtain separate approval and run only through the existing executor/publisher. Preserve consumed history, stale/duplicate rejection, retention/expiry and publication-only retries. #101/#88 remain open for both real automatic reports and independent acceptance.
Installed-wheel correction and refreshed addition plan
The preceding addition head's installed-package jobs failed because the total
knowledge inventory still expected108 instead of109. A real built-wheel check
outside the checkout reproduced that exact assertion failure. Commit
03d8a513a7cdc4a921b38dd8d5442555f887a0f7changes only that one inventory assertion; the same installed wheel then passes
the complete existing check, and Ruff/whitespace passed. The source skill,
projection source pin, mapping, cases, grader, controllers and budgets are unchanged.
Both new-head installed-package jobs succeeded in
Ready run 40309420195;
the last full-check/policy snapshot was still in progress/cancelled for some jobs,
so this is not an all-CI success claim. Update PR160's four Ready jobs passed in
40295923528,
with a later successful declarations job; its automatic publisher still failed.
Addition's current canonical PR plan is
30d41b2bd3413169400222ddb4bd04b477166c78d3a28bd895716040fb9196bc, comparison freeze8dc27b1397dc3b00f2bbae8ef65efd2c2de7170ea1a45ec1634801fdb99362ca,automatically prepared at base/trusted version
0c78a7e257dfaa3573b77581a39539e8a3a5da4c.The new downloaded archive digest and all frozen inputs/controllers passed the
existing loader. The older
8ca9da51bc13df24ba8ff632404105c847a12e33c4899e5f6d8b2f7a5d415299proposal is archived as superseded;it was never authorized or executed. Update's exact plan remains unchanged.
The new addition publisher
was inspected and again returns HTTP403 on POST. Keep both runtime/publication
readiness blockers and separate exact approval gates. Evaluation calls0,
evaluation dispatches0, consumed attempts0 for both subjects. No outcome reports
or whole-Issue acceptance are claimed; #101/#88 remain open.
Dong1017 commentedon Oct 26, 2026
Status re-check and next local-agent handoff — 2026-10-26
Software delivery is complete through merged #118 (
db82ec8346364667b90f4fa97fdb190150141cf0); whole-issue acceptance is not. This confirms, rather than replaces, the existing owner plan. Independent integration/CI review, Ready CI and targeted declarations rerun passed. Post-merge manifest/CPU smoke passed; the separate post-merge declarations run was cancelled, not successful.Next bounded task for the local agent
Keep #101 and #88 open pending that real addition/update end-to-end evidence and independent acceptance. No automatic merging, issue closure, required efficacy gate, new platform, credential grant or budget approval is included.
fix: retain safe Skill Eval publisher HTTP diagnostics (#163)