[P1.59] Move the Sonnet-tier defaults to claude-sonnet-5-5 before claude-sonnet-4-5 reaches end of life (2026-11-30) #1348

Activity

1. frankbria

   added

   priority:high

   P1-high-betaHigh priority - should fix before beta for best experienceHigh priority - should fix before beta for best experience

   dependenciesPull requests that update a dependency filePull requests that update a dependency file

   needs-operatorNeeds box or production access no agent session can reachNeeds box or production access no agent session can reach

   on Oct 2, 2026
2. frankbria commented on Oct 3, 2026

   

   frankbria

   on Oct 3, 2026

   OwnerAuthor

   More actions

   Parked (needs-operator): the definition of done needs scripts/lifecycle --mode cli with a real key for the cost comparison, and the agent environment has none. Decided 2026-10-02: this ships as 0.9.5, after 0.9.4 (#1277), and still before the 2026-10-31 deadline.
3. frankbria commented on Oct 6, 2026

   

   frankbria

   on Oct 6, 2026

   OwnerAuthor

   More actions

   Operator steps: tactical, in order (2026-10-06)

   Priority 3 of 7. Blocked by #1277 (0.9.4 must ship first). Deadline: tag v0.9.5 by 2026-10-31, two weeks before claude-sonnet-4-5 retires on 2026-11-30. About 1.5 hours: two real-key lifecycle runs (about $1–3 each) and a release.

   ⚠️ scripts/lifecycle defaults to haiku and overrides the compiled-in defaults (tests/lifecycle/conftest.py). A defaults comparison must pass --model sonnet, or both runs measure haiku.

   1. Start the agent's half. Say "go on #1348". The agent opens a PR that:

   • sets DEFAULT_{PLANNING,EXECUTION,CORRECTION,SUPERVISION}_MODEL = "claude-sonnet-5-5";
   • updates tests/adapters/test_model_defaults_guard_1112.py;
   • repoints the lifecycle sonnet alias to 5.5, so step 4 measures the new model;
   • bumps the version to 0.9.5 with a CHANGELOG entry.

   Note the PR number:

   PR=<the PR number>

   2. Load the key.

   read -rs ANTHROPIC_API_KEY && export ANTHROPIC_API_KEY

   3. Baseline: Sonnet 4.5 on main.

   cd ~/projects/codeframe && git checkout main && git pull --ff-only
   scripts/lifecycle --mode cli --model sonnet 2>&1 | tee /tmp/lifecycle-sonnet45.log

   4. Candidate: Sonnet 5.5 on the PR branch.

   gh pr checkout "$PR" --repo frankbria/codeframe
   scripts/lifecycle --mode cli --model sonnet 2>&1 | tee /tmp/lifecycle-sonnet55.log

   5. Hand both logs back. Tell the agent: "lifecycle logs are at /tmp/lifecycle-sonnet45.log and /tmp/lifecycle-sonnet55.log". It posts the cost, token and gate pass-rate comparison on the PR. The bar: the pass rate is not lower than 4.5's, and the cost is acceptable to you.

   6. Approve the merge. Reply on the PR, or tell the agent, "merge #1348". It merges after CI and bot triage.

   7. Release 0.9.5. Follow #1277's steps 2–12 with 0.9.5 in place of 0.9.4:

   • cleanroom --source .;
   • set the heading date;
   • git tag -a v0.9.5 … && git push origin v0.9.5;
   • watch release.yml;
   • check the PyPI version;
   • cleanroom against PyPI.

   The release's model guard does a live check of the new default, using the staging key.

   8. Close.

   gh issue close 1348 --repo frankbria/codeframe --comment "0.9.5 ships claude-sonnet-5-5 defaults; lifecycle comparison on PR #$PR; cleanroom green on PyPI."
4. frankbria commented on Oct 6, 2026

   

   frankbria

   on Oct 6, 2026

   OwnerAuthor

   More actions

   Update (2026-10-06): the cost comparison runs in CI, not on your machine

   lifecycle.yml takes mode/model inputs on workflow_dispatch and runs in the staging environment with the working Anthropic key. It has passed every Sunday. So steps 2–5 above (the two scripts/lifecycle --model sonnet runs) are the agent's:

   gh workflow run lifecycle.yml --repo frankbria/codeframe --ref main -f mode=cli -f model=sonnet    # baseline, Sonnet 4.5
   gh workflow run lifecycle.yml --repo frankbria/codeframe --ref <PR branch> -f mode=cli -f model=sonnet   # candidate, Sonnet 5.5

   These are still billed to your Anthropic account (about $1–3 each). Your part is only to approve that spend and the merge. Order unchanged: #1443 (the discovery cap) lands first, then this, then the 0.9.5 release.
5. frankbria

   added 8 commits that reference this issue on Oct 12, 2026

   test(lifecycle): print the run's token usage and cost, and show it on…

   ...

   5189571

   test(adapters): pin temperature-on-the-wire tests to a sampling model…

   ...

   5c45cb5

   feat(llm): Sonnet-tier defaults move to claude-sonnet-5-5 (#1348)

   ...

   376aad5

   test(lifecycle): print the run's token usage and cost, and show it on…

   ...

   15cf5f1

   test(adapters): pin temperature-on-the-wire tests to a sampling model…

   ...

   fc7a7b9

   feat(llm): Sonnet-tier defaults move to claude-sonnet-5-5 (#1348) (#1458

   ...

   8efcb62

   docs(readme): beta banner reads 0.9.5 (#1348)

   ...

   0a2d879

   chore(release): 0.9.5 — version bump and CHANGELOG (#1459)

   ...

   29970ad
6. frankbria commented on Oct 12, 2026

   

   frankbria

   on Oct 12, 2026

   OwnerAuthor

   More actions

   Done: shipped in 0.9.5 (2026-10-12), seven weeks before claude-sonnet-4-5 retires on 2026-11-30.

   Definition of done:

   • Defaults: DEFAULT_{PLANNING,EXECUTION,CORRECTION,SUPERVISION}_MODEL are claude-sonnet-5-5, and the guards are updated (PR feat(llm): Sonnet-tier defaults move to claude-sonnet-5-5 (#1348) #1458). Generation stays on claude-haiku-4-5.

   • Lifecycle comparison (lifecycle.yml cli/sonnet, details on feat(llm): Sonnet-tier defaults move to claude-sonnet-5-5 (#1348) #1458). Both models pass every acceptance check:

     Model	Cost	LLM calls	Time
     Sonnet 4.5	$2.26	98	839s
     Sonnet 5.5	$0.41	28	264s

   • Release at least two weeks before EOL: v0.9.5 tagged; release.yml passed Unlocked Resolution, published to PyPI and created the GitHub Release (PR chore(release): 0.9.5 — version bump and CHANGELOG #1459).

   • quickstart-cleanroom against the published package: uv tool install codeframe-ai installed codeframe-ai==0.9.5, 17/17 steps OK, 0 findings, 409s; 4-prd-generate took 90s with 3 questions.
7. frankbria

   closed this as completedon Oct 12, 2026