Skip to content

[reviewer] A/B the billing modes (subscription vs API key) + verbose agent logs - #4133

Draft
brentvatne wants to merge 1 commit into
mainfrom
brentvatne/ecr-billing-mode-ab
Draft

[reviewer] A/B the billing modes (subscription vs API key) + verbose agent logs#4133
brentvatne wants to merge 1 commit into
mainfrom
brentvatne/ecr-billing-mode-ab

Conversation

@brentvatne

Copy link
Copy Markdown
Member

Why

GPT review passes keep stalling and timing out, and nothing in the logs says why. This PR turns the reviewer into an experiment: each run randomly picks a billing mode (ChatGPT/Codex subscription vs metered API key) and records the arm plus pass outcomes, so we can compare reliability per billing mode from field data. It also turns on full agent output in the job log.

Changes

  • .expo-code-review/config.jsonc: auth mode: "random" with apiKeyEnv: OPENAI_API_KEY + oauthTokenEnv: CODEX_OAUTH_ACCESS_TOKEN. Both arms run at the same concurrency; the arm lands in the run log (authModes) with passOutcomes (completed/failed/timedOut/stalled). Ending the experiment later is a one-field change (mode: "oauth" or "api-key").
  • Workflows: ECR_VERSION fallback → ^0.7.0; auth-lock fallback (ECR_EXPECTED_TOKEN_ENV) lists both credential env names (stable across the per-run coin flip); OPENAI_API_KEY passed to the review steps; ECR_VERBOSE=1 streams each agent's reasoning, reply text, and tool inputs/outputs into the job log.

Before merging (in order)

  1. @expo/code-review-cli 0.7.0 must be published — expo/code-review-cli#45 adds mode: "random" and --verbose; this PR's config fails verify-config on 0.6.x.
  2. Add the OPENAI_API_KEY repo secret — a Restricted key with only Responses → Request and Chat completions → Request, created in a dedicated budget-capped project.
  3. If the ECR_EXPECTED_TOKEN_ENV repo variable is set, update it to CODEX_OAUTH_ACCESS_TOKEN,OPENAI_API_KEY (the variable overrides the YAML fallback).

Note: ecr ci loads config from the PR's trusted base commit, so the experiment starts for PRs opened after this merges.

Reading the results

reviews.jsonl (uploaded as the run artifact) now carries the arm per run:

jq -s 'group_by(.authModes.openai.mode) | map({mode: .[0].authModes.openai.mode,
  runs: length,
  stalled: (map(.passOutcomes.stalled // 0) | add),
  timedOut: (map(.passOutcomes.timedOut // 0) | add),
  rateLimited: (map(.rateLimitEvents // 0) | add)})' reviews.jsonl

🤖 Generated with Claude Code

…ll agent output

GPT review passes keep stalling and timing out, and the cause is unclear.
This flips .expo-code-review auth to mode "random" (@expo/code-review-cli
0.7): each run randomly picks the ChatGPT/Codex subscription (oauth) or
the metered API key, records the arm and pass outcomes in the run log,
and runs both arms at the same concurrency — so stall/timeout/rate-limit
rates can be compared per billing mode from field data.

ECR_VERBOSE turns on full agent output (reasoning, reply text, tool
inputs/outputs) in the job log, so a wedged pass shows what it was doing
instead of a bare heartbeat.

Requires the OPENAI_API_KEY repo secret (Restricted key: Responses +
Chat completions Request only, dedicated budget-capped project). The
auth lock now expects both credential env names.
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown

❌ It looks like a changelog entry is missing for this PR. Add it manually to CHANGELOG.md.
⏩ If this PR doesn't require a changelog entry, such as if it's an internal change that doesn't affect the user experience, you can add the "no changelog" label to the PR.

@codecov

codecov Bot commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.64%. Comparing base (edb47e2) to head (73fa734).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4133   +/-   ##
=======================================
  Coverage   62.64%   62.64%           
=======================================
  Files         999      999           
  Lines       45175    45175           
  Branches     9483     9483           
=======================================
  Hits        28294    28294           
  Misses      15433    15433           
  Partials     1448     1448           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant