Replace wpcom model choices with fast/balanced/strong capability tiers - #4727
Open
youknowriad wants to merge 8 commits into
Open
Replace wpcom model choices with fast/balanced/strong capability tiers#4727youknowriad wants to merge 8 commits into
youknowriad wants to merge 8 commits into
Conversation
The WordPress.com provider now offers three capability tiers resolved to upstream models by the proxy's studio-agent lane over the Chat Completions API. Accounts with purchased AI credits default to balanced; free-allowance accounts default to fast. Sonnet 5 / Opus 5 remain available only on the Anthropic API-key provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The strong tier's upstream errors when the reasoning switch is left implicit; fast and balanced keep sending no reasoning field at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This reverts commit 199c62e.
Its upstream is a reasoning model that rejects the Chat Completions dialect's tools-plus-reasoning combination; fast and balanced stay on Chat Completions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Balanced and Strong are marked requiresPaidAiCredits and shown disabled in the desktop and browser pickers (withheld from the CLI /model list) unless purchased credits remain. UI gating only — the proxy enforces access, and sessions already pinned to a paid tier keep showing it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two subtle surfaces sharing one line of copy: a footer item in the model picker under the disabled tiers, and a dismissible muted line above the composer (dismissal remembered per renderer via localStorage). Both open the existing AI-credits purchase flow. Shown only once the quota definitively reports a free-allowance account. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Balanced and Strong stay enabled for Automattician accounts regardless of purchased credits, and the credits nudges disappear with the lock. Show-only heuristic (a8c.com / automattic.com email) — the proxy remains the real access control. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Validated session-model reads collapse into readRecordedSessionModel, the nudge dismissal helpers move to packages/common, and narrating comments shrink. Drops a duplicate credential-error test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
youknowriad
force-pushed
the
claude/wpcom-model-choices-refactor-4771d0
branch
from
August 29, 2026 12:21
05f7ab7 to
bb57db3
Compare
Collaborator
📊 Performance Test ResultsComparing bb57db3 vs trunk app-size
site-editor
site-startup
Results are median values from multiple test runs. Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff) |
gavande1
reviewed
Aug 31, 2026
| // can't block the first turn — the free-tier default is the safe floor. | ||
| const QUOTA_FETCH_TIMEOUT_MS = 3_000; | ||
|
|
||
| async function resolveWpcomDefaultModel(): Promise< AiModelId > { |
Contributor
There was a problem hiding this comment.
If this lookup fails, a paid account runs on the fast tier and the persistSessionContext writes that into the session. When the session resumes, it never re-checks the quota. Can we skip persisting the model when the quota resolution did not succeed?
gavande1
reviewed
Aug 31, 2026
| // only applies while nothing else picked a model. | ||
| let wpcomDefaultModel: AiModelId = getAiProviderDefaultModel( DEFAULT_AI_PROVIDER ); | ||
| let quotaDefaultApplicable = ! recordedModel && currentProvider === DEFAULT_AI_PROVIDER; | ||
| const wpcomDefaultModelResolution = resolveWpcomDefaultModel() |
Contributor
There was a problem hiding this comment.
The default is resolved once at startup, so starting logged out settles it on Fast and when user logs in again, it does not re-resolve it. A user with purchased credits keeps running on Fast until they change it by hand. Can we re-resolve after a successful login?
Contributor
Author
There was a problem hiding this comment.
A user with purchased credits keeps running on Fast until they change it by hand. Can we re-resolve after a successful login?
IMO that's the right thing to do, I don't think we should update the model randomly mid session even if the user purchase credits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issues
How AI was used in this PR
Built end-to-end with Claude Code (Fable 5) in a pair-programming session: I directed the design decisions (tier names, defaults, proxy lane, gating and nudge behavior) turn by turn and reviewed each step. The agent wrote the implementation and tests, verified requests on the wire against the AI proxy, and checked the browser UI in light and dark themes.
Proposed Changes
The WordPress.com provider no longer exposes concrete model names. It now offers three capability tiers — Fast, Balanced, and Strong — sent verbatim to the proxy's
studio-agentlane, which resolves each alias to an upstream model server-side. This lets us retune what backs each tier without a client release, and replaces a picker full of vendor names with a choice users can actually reason about./modellist) unless purchased credits remain. Automatticians are exempt (email heuristic, show-only — the proxy stays the real access control).Server-side dependency: the proxy's
studio-agentfeature lane must be allowlisted and accept both/v1/chat/completionsand/v1/responses. Without it, wpcom turns fail (this is also why theidentityeval couldn't pass from the dev machine — flagging for a human eval run once the lane is fully live).Testing Instructions
npm run cli:build && node apps/cli/dist/cli/main.mjs code— a fresh session on a free-allowance account shows Fast in the composer pill;/modellists only Fast (paid tiers hidden with an info line). With purchased credits, the default is Balanced and all tiers are listed.npm run cli:build:ui && node apps/cli/dist/cli/main.mjs ui --no-open): on a free account, open the model picker — Balanced/Strong disabled with the "Add AI credits…" footer item; the dismissible nudge line sits above the composer and stays dismissed after the ×. With an@automattic.com/@a8c.comaccount everything is unlocked and no nudge shows./v1/responses.Pre-merge Checklist