Skip to content

feat: enable bounded Yibuapi Skill Eval execution - #181

Merged
Dong1017 merged 4 commits into
mainfrom
codex/issue101-yibuapi-live
Oct 27, 2026
Merged

Dong1017 merged 4 commits into
mainfrom
codex/issue101-yibuapi-live

Conversation

@Dong1017

@Dong1017 Dong1017 commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Refs #101. Enables usable opt-in Yibuapi execution after #174.

provider_config connects the direct Messages client to an internal Python worker, the existing paired comparison runner and the hosted execute job. Complete reviewed pricing/bounds and the owner-provisioned Secret enable the path; no unconditional live blocker remains. Codex stays the default. No external agent executable/framework is added.

The frozen readiness record binds model/provider/group, effective currency/per-million input/output/cache-read/cache-write rates, request fee, expiry and all request/output/read/tool bounds. Reservation covers every possible attempted request across both arms/comparisons, using the full registered model-context input ceiling, the highest input/cache rate, full output ceiling and fee. Every request rounds upward to cents before allocation. Unknown/expired pricing, understated context, currency mismatch, over-budget totals or missing keys fail before calls; missing keys precede the local claim. Dispatch history remains durable and is not erased. No automatic retries, fallback model or extra paid probe; the first baseline request supplies compatibility evidence.

Only the conditional provider model step receives SKILL_EVAL_YIBUAPI_KEY. Worker/verifier environments exclude GitHub/Actions credentials, and the verifier receives no provider key. No key in argv, prompt, trace, artifact or plan/publish. Hosted permissions, trusted checkout, exact approval/source identity, one-shot history, artifact validation and the separate privileged publisher/regrade remain intact. New paid runs explicitly stop on unknown usage/interruption before later comparisons. Thinking/signature blocks are structurally validated and preserved opaquely without content logging; auto tool choice supports Opus5.5.

Head ffb5ce1; base 38d5883; tree e10902c81ee296ea5e3dba7f9f32515ba985793b. All nine remote files were read back equal to the inspected source.

Validation:

  • Ready 40481781107: all four jobs passed on merge b9425f2 of this exact head into the stated base. Python3.10 and3.11 each report 1325 passed, 9 skipped, 135 subtests passed; Ruff and both installed-package checks passed.
  • Draft 40481772829 and natural Skill PR 40481778336 passed. Policy attempts are separately tracked; parent owns declaration rechecks.
  • Mocks exercise worker→paired addition/update execution→retained trusted feedback, finite cost arithmetic/allocation, missing/unknown/stale facts, missing key before claim, one-shot rejection, unknown-usage stop, credential separation and thinking replay. Earlier lint/test-fixture failures were corrected; no earlier failed/skipped test is counted.
  • Local Ruff/pytest did not start: CreateProcess failed with exec-server transport disconnected. The original prototype, new isolated worktree and source snapshot are preserved. No local pass or local/remote tree equality is claimed.

Owner's current one-smoke total is USD100, superseding CNY10; no retries or extra campaigns. Parent verified public Yibuapi per1M prices8/40/0.4/10 USD for input/output/cache-read/cache-write in default/claude-code/claude-code-pro. A two-trial, max3-requests/trial, max4096-output proposal reserves USD61.02 at the full1M input ceiling (explicit zero request fee under the published token-only schedule). Confirm the chosen group applies to the owner key before freezing. No API/key probe, spending, Secret read/configuration, login or execute dispatch occurred.

This PR proposes a narrowly scoped Secret exposure; parent checks the owner's action-time approval before merge, then prepares/approves a new exact trusted plan and coordinates the real smoke. Selected public inputs/skills go to Yibuapi; held-out answers/verifier/unrelated files do not. #101/#88 remain open.

Signed-off-by: GUOGUO <xwdong1998@163.com>
Agent-assisted-by: Codex (author)
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Skill evaluation

PLANNED: execution is not authorized by knowing this hash.
Outcomes: {"fix":0,"inconclusive":0,"missing_pair":0,"regression":0,"unchanged_correct":0,"unchanged_incorrect":0}
Earlier revision results are stale for this plan.

Selected comparisons: 0 · Estimated task calls: 0.
Uncovered/unsupported changes: 9. Gaps never count as passes.

Applicable only while these revisions and inputs match the current PR.
Execution requires exact plan, versions and budget approval; a hash alone does not authorize it.
Token usage stops new trials; it is not a hard provider token or monetary cap.
Cost remains unknown.

Evidence and logs · Retention: 30 days.
Outcomes are historical; verifiable only before Unix time 1795685683 while the linked artifact is available. Expired evidence is unverified.

Pipeline evidence only; offline fixed outputs do not establish skill efficacy.

Plan bindings, budget and frozen exam

Plan SHA-256: b8c79d93234dd951566e25e7092531ec5f1db79238372877d5eed7f72b97a6cb
Base: 38d5883b8e72e352865e214194cb22dc99253dbd · Head: ffb5ce1e167cce33fb931dff27f705fc1b0a9212
Trusted implementation: 38d5883b8e72e352865e214194cb22dc99253dbd

Model/settings: {"effort":"xhigh","name":"gpt-6.1-sol"}
Budget: {"comparisons":0,"estimated_calls":0,"limits_per_comparison":{"campaign_seconds":600,"max_task_calls":2,"repeats":1,"timeout_seconds":180,"token_stop_new_trials":20000}}

Skill Comparison Cases Estimated calls
Uncovered / unsupported paths (9)
  • uncovered: ".github/workflows/skill-eval-pr.yml"
  • uncovered: "docs/skill-eval-yibuapi-preflight.md"
  • uncovered: "src/inferrsi/harness/skill_evaluation.py"
  • uncovered: "src/inferrsi/infra/executors/local.py"
  • uncovered: "src/inferrsi/model/anthropic_agent.py"
  • uncovered: "tests/test_anthropic_agent.py"
  • uncovered: "tests/test_yibuapi_runtime.py"
  • uncovered: "tools/skill_eval_compare.py"
  • uncovered: "tools/skill_eval_pr.py"

Signed-off-by: GUOGUO <xwdong1998@163.com>
Agent-assisted-by: Codex (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
Agent-assisted-by: Codex (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
Agent-assisted-by: Codex (author)
@Dong1017
Dong1017 marked this pull request as ready for review October 27, 2026 09:41
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T09:44:14.874657Z ffb5ce1 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the bounded software-delivery scope at head ffb5ce1 against base 38d5883. Independent static reviews covered cost reservation/runtime and credential/workflow isolation; the final delta rounds each request upward to cents before allocation and adds an additivity regression assertion. No verified correctness blocker remains in the inspected change.

Verified CI: all four Ready jobs passed: https://github.com/InferMatrixLab/InferRSI/actions/runs/40481781107 . Current declaration policy is still pending, so merge remains conditional on that prerequisite turning green.

The configured path uses the internal Python worker, preserves trusted publication/regrading and default Codex behavior, excludes provider credentials from the verifier and publication credentials from the worker, and bounds all attempted requests without automatic retries. This is software readiness, not evidence of live Yibuapi compatibility or skill efficacy. No paid call was made during review. The real smoke still needs an exact fresh frozen plan, applicable key-group billing facts and the owner's USD100 total bound; #101/#88 acceptance is not granted by this review.

@Dong1017
Dong1017 merged commit e21b739 into main Oct 27, 2026
15 of 17 checks passed
@Dong1017
Dong1017 deleted the codex/issue101-yibuapi-live branch October 30, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant