Repository navigation
Use native OpenAI and Anthropic credits with bounded accounting - #10
hudsonaikins wants to merge 1 commit into
Conversation
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
The account paying for this security review has reached its Codex usage limits. The payer can check the Codex usage dashboard. For personal accounts, using credits requires enabling “Use credits for security reviews” in Code review settings. If you do not manage the paying account, contact this repository's admins. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 41e3930939
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| DEFAULT_MODEL = "openai/gpt-6-luna" | ||
| DEFAULT_JUDGE_MODEL = "anthropic/claude-haiku-5-5" |
There was a problem hiding this comment.
When BACKINTEL_MODEL is unset, this native default is used by cmd_acceptance, but bench/commands.py:_budget still unconditionally constructs RunBudget, which requires a provider-reported cost. The first native completion therefore settles with None, marks the shared budget uncertain, and blocks the analyst's next turn or the judge; even a single-turn answer then has incomplete cost accounting, so the default acceptance workflow cannot complete. Choose NativeRunBudget for qualified native analyst/judge models, as the improvement-cycle path already does.
AGENTS.md reference: AGENTS.md:L54-L56
Useful? React with 👍 / 👎.
| DEFAULT_MODEL = "openai/gpt-6-luna" | ||
| DEFAULT_JUDGE_MODEL = "anthropic/claude-haiku-5-5" |
There was a problem hiding this comment.
With these native defaults, an ordinary backintel bench run creates its harness without a NativeRunBudget, so UsageTracker correctly records cost_usd=None and no native estimate. However, bench/scorecard.py:aggregate converts every missing cost to zero with m.get("cost_usd") or 0, and the Markdown then displays 0.0000, making all new default native experiments appear free even though their charges are explicitly unknown. Preserve an unknown value (or render a clear incomplete marker) when no call has provider-reported cost.
AGENTS.md reference: AGENTS.md:L54-L56
Useful? React with 👍 / 👎.
| max_retries=max_retries, | ||
| max_tokens=int(env.get("BACKINTEL_MAX_COMPLETION_TOKENS", "8192")), | ||
| timeout=timeout_s, | ||
| service_tier="default", | ||
| **({"temperature": temperature} if temperature is not None else {}), |
There was a problem hiding this comment.
When Harness.build constructs the new default analyst, it calls this Chat Completions route without a reasoning setting, so Luna uses its default reasoning effort instead of none. OpenAI documents that GPT-6 Luna's Chat Completions endpoint supports function calling only when reasoning_effort is none (GPT-6 Luna model documentation); because the analyst depends on its table/query tools, default ask and benchmark runs cannot execute the required tool-calling loop. Set reasoning_effort="none" for this route or use the Responses API.
AGENTS.md reference: AGENTS.md:L20-L25
Useful? React with 👍 / 👎.
Routes BackIntel's benchmark analyst and method judge directly to the owner's OpenAI and Anthropic APIs, using native credentials and explicit standard service tiers without gateway fallback. The separate Sol artifact specialist keeps its existing ChatGPT subscription connection.
Native usage is recorded as a conservative, versioned token-cost estimate; actual invoice amounts remain unknown. Fresh campaign plans freeze accounting identity, and missing usage, unsupported billing categories or reservation overruns stop later admission. Historical OpenRouter experiments and the original $5 cumulative allowance are preserved.
Validation on this provider commit: 232 tests passed, 15 optional skips; Ruff lint/format, Vulture, Gitleaks and diff checks passed. Hosted CI passed on 41e3930. Native structured output and usage accounting were verified live. The subsequent evaluator fix in PR8 passed 10/10 frozen controls; its original 24-response paired regression comparison is accepted for review (candidate 12/12 supported versus baseline 9/12). Development/reserved campaign acceptance is still pending in PR9. These later results qualify their own frozen revisions, not this commit alone.
Shared Doppler model selectors remain unchanged. Both native keys are already configured; no new key, login, credit purchase, merge or deployment is included. Adoption guidance and retained historical failures are in docs/native-provider-credits.md. Tracks INT-430. Dependency order: PR10, PR8, then PR9.