Skip to content

[reviewer] A/B the billing modes (subscription vs API key) + verbose agent logs #4133

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
brentvatne wants to merge 1 commit into
base: main
Choose a base branch
from
+50 −24
Draft

[reviewer] A/B the billing modes (subscription vs API key) + verbose agent logs #4133

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 22 additions & 11 deletions .expo-code-review/config.jsonc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,28 @@
"breakGlass": { "marker": "/skip-review" },
"commentTag": "expo-ai-code-reviewer",

// ChatGPT/Codex subscription (OAuth): tokenEnv holds the ACCESS token from an
// `opencode auth login` ChatGPT sign-in (`ecr setup-auth` extracts it) — a
// plain bearer with no rotation involvement; re-mint before its ~10-day expiry
// (doctor + the run preflight warn first). NEVER use the refresh token here:
// it is single-use and dies on first rotation. All models above are on the
// subscription's allowlist, so reviews draw no metered spend. (To add pro-tier
// models later, see the mixed auth.providers setup in the @expo/code-review-cli
// README.)
// Billing-mode A/B experiment (needs @expo/code-review-cli >= 0.7): each run
// flips a fair coin between the ChatGPT/Codex subscription (oauth) and the
// metered API key, and records the arm in the run log (authModes) together
// with pass outcomes — so GPT stall/timeout rates can be compared per billing
// mode from field data. BOTH credentials must be set (the preflight fails
// fast); both arms run at the same concurrency.
//
// oauthTokenEnv holds the ACCESS token from an `opencode auth login` ChatGPT
// sign-in (`ecr setup-auth` extracts it) — a plain bearer with no rotation
// involvement; re-mint before its ~10-day expiry (doctor + the run preflight
// warn first). NEVER use the refresh token here: it is single-use and dies on
// first rotation. apiKeyEnv holds a Restricted key (Responses → Request +
// Chat completions → Request only) from a dedicated budget-capped project.
//
// To end the experiment, set "mode" to "oauth" or "api-key" — one field.
"auth": {
"mode": "oauth",
"provider": "openai",
"tokenEnv": "CODEX_OAUTH_ACCESS_TOKEN"
"providers": {
"openai": {
"mode": "random",
"apiKeyEnv": "OPENAI_API_KEY",
"oauthTokenEnv": "CODEX_OAUTH_ACCESS_TOKEN"
}
}
}
}
9 changes: 7 additions & 2 deletions .github/workflows/expo-code-review-command.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ permissions:

env:
# Published reviewer run via npx (override with repo variable ECR_VERSION).
ECR_VERSION: ${{ vars.ECR_VERSION || '^0.6.0' }}
ECR_VERSION: ${{ vars.ECR_VERSION || '^0.7.0' }}

concurrency:
group: ai-code-review-cmd-${{ github.event.issue.number }}
Expand Down Expand Up @@ -121,8 +121,13 @@ jobs:
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# Billing A/B (auth.mode "random"): both credentials must be set —
# each run flips between them. See expo-code-review.yml for details.
CODEX_OAUTH_ACCESS_TOKEN: ${{ secrets.CODEX_OAUTH_ACCESS_TOKEN }}
ECR_EXPECTED_TOKEN_ENV: ${{ vars.ECR_EXPECTED_TOKEN_ENV || 'CODEX_OAUTH_ACCESS_TOKEN' }}
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
ECR_EXPECTED_TOKEN_ENV: ${{ vars.ECR_EXPECTED_TOKEN_ENV || 'CODEX_OAUTH_ACCESS_TOKEN,OPENAI_API_KEY' }}
# Full agent output in the job log (see expo-code-review.yml).
ECR_VERBOSE: '1'
# Unblocked CLI headless auth (PAT or team access token). Absent =>
# the CLI stays unauthenticated; the review still runs.
UNBLOCKED_API_TOKEN: ${{ secrets.UNBLOCKED_API_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/expo-code-review-dismiss.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ permissions:

env:
# Published reviewer run via npx (override with repo variable ECR_VERSION).
ECR_VERSION: ${{ vars.ECR_VERSION || '^0.6.0' }}
ECR_VERSION: ${{ vars.ECR_VERSION || '^0.7.0' }}

concurrency:
group: ai-code-review-dismiss-${{ github.event.issue.number }}
Expand Down
30 changes: 20 additions & 10 deletions .github/workflows/expo-code-review.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,9 @@ permissions:
env:
# The reviewer is the published @expo/code-review-cli package, run via npx — the
# repo only carries the `.expo-code-review/` config, not the engine source.
# Override with a repo variable ECR_VERSION to pin/bump; defaults to 0.6.x
# (0.6.0 = trusted base-commit configuration + PR-head runtime-config scrub).
ECR_VERSION: ${{ vars.ECR_VERSION || '^0.6.0' }}
# Override with a repo variable ECR_VERSION to pin/bump; defaults to 0.7.x
# (0.7.0 = auth mode "random" billing A/B + verbose agent output).
ECR_VERSION: ${{ vars.ECR_VERSION || '^0.7.0' }}

concurrency:
group: ai-code-review-${{ github.event.pull_request.number }}
Expand Down Expand Up @@ -136,7 +136,10 @@ jobs:
- name: Guard config tokenEnv (root + routing + all scopes)
if: steps.resolve.outputs.run == 'true'
env:
ECR_EXPECTED_TOKEN_ENV: ${{ vars.ECR_EXPECTED_TOKEN_ENV || 'CODEX_OAUTH_ACCESS_TOKEN' }}
# Both names: the billing A/B auth entry declares apiKeyEnv AND
# oauthTokenEnv, and the lock covers every declared credential env
# (stable across the per-run coin flip).
ECR_EXPECTED_TOKEN_ENV: ${{ vars.ECR_EXPECTED_TOKEN_ENV || 'CODEX_OAUTH_ACCESS_TOKEN,OPENAI_API_KEY' }}
run: npx --yes -p "@expo/code-review-cli@$ECR_VERSION" ecr verify-config

- name: Run AI review
Expand All @@ -148,14 +151,21 @@ jobs:
env:
# GitHub token scoped by the permissions block above (PR comments only).
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# ChatGPT/Codex subscription ACCESS token (auth.mode "oauth" in
# .expo-code-review/config.jsonc reads this env var) — used as-is, no
# rotation involvement; re-mint before its ~10-day expiry. All reviews
# run on the subscription.
# Billing A/B (auth.mode "random" in .expo-code-review/config.jsonc):
# each run flips between the subscription token and the metered API
# key; BOTH must be set or the preflight fails fast.
# ChatGPT/Codex subscription ACCESS token — used as-is, no rotation
# involvement; re-mint before its ~10-day expiry.
CODEX_OAUTH_ACCESS_TOKEN: ${{ secrets.CODEX_OAUTH_ACCESS_TOKEN }}
# Restricted OpenAI key (Responses + Chat completions Request only),
# from a dedicated budget-capped project.
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
# Layer-1 runtime auth lock (mirrors the guard step): the CLI refuses to
# run when the tokenEnv the loaded config honors differs from this.
ECR_EXPECTED_TOKEN_ENV: ${{ vars.ECR_EXPECTED_TOKEN_ENV || 'CODEX_OAUTH_ACCESS_TOKEN' }}
# run when the credential envs the loaded config honors differ from this.
ECR_EXPECTED_TOKEN_ENV: ${{ vars.ECR_EXPECTED_TOKEN_ENV || 'CODEX_OAUTH_ACCESS_TOKEN,OPENAI_API_KEY' }}
# Full agent output (reasoning, reply text, tool I/O) in the job log —
# the observability half of diagnosing the GPT stalls.
ECR_VERBOSE: '1'
# Unblocked CLI headless auth (PAT or team access token). Absent =>
# the CLI stays unauthenticated; the review still runs.
UNBLOCKED_API_TOKEN: ${{ secrets.UNBLOCKED_API_TOKEN }}
Expand Down
Loading