Skip to content

Codify Cloud Run scaling settings into deploy config (warm floor + max-instances 8)#3768

Merged
anth-volk merged 1 commit into
masterfrom
feat/cloud-run-scaling-config-reconcile
Jul 23, 2026
Merged

Codify Cloud Run scaling settings into deploy config (warm floor + max-instances 8)#3768
anth-volk merged 1 commit into
masterfrom
feat/cloud-run-scaling-config-reconcile

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #3767

Why

Two production Cloud Run scaling settings were being managed by hand and were not reflected in the repo:

  1. Service-level warm floor (minScale=2). Set manually via gcloud --min during the PR 4 host cutover; it lived only as live service state. It survives CI deploys today only because the deploy never asserts a service-level min — invisible to anyone reading the repo, and one --min 0 or service recreation away from silently vanishing. Overnight sweeps confirm this floor is what holds the service at 2 warm instances through the bot trough with zero cold-start churn.
  2. max-instances 4. At the current 50/50 LB split the service already sits pinned at the ceiling of 4 during the daytime peak (instance_count flatlines at 4.0). The imminent 100% Cloud Run cutover roughly doubles load, so 4 would bottleneck.

What

  • New CLOUD_RUN_SERVICE_MIN_INSTANCES env (default 0 in cloud_run_env.sh; prod 2, staging 0 in push.yml). The deploy now passes --min ${CLOUD_RUN_SERVICE_MIN_INSTANCES} — the service-level floor — on every deploy.
  • Revision-level --min-instances 0 unchanged. A positive revision-level min is baked immutably onto every new revision, so each tagged no-traffic candidate would pin its own warm 16Gi instance (per-tag cost bomb). The existing invariant test guarding this stays green.
  • max-instances 4 → 8 in the production job.
  • Runbook (docs/migration/cloud-run-operations.md) updated: warm floor is now applied by CI, not manually-once; max sizing rationale recorded.
  • Deploy-script unit tests extended (service-level --min emitted, prod max 8, per-job service-min values); changelog fragment added.

Live state

The live production service was already raised to max-instances 8 (new revision 00045-2rd, serving 100%) and carries minScale 2 out-of-band, so production matches this config now. This PR makes both durable across future releases — the production Cloud Run deploy only runs on the release commit, so without it the next release would revert max to 4.

Testing

.venv/bin/python -m pytest tests/unit/test_cloud_run_deploy_scripts.py -q
34 passed

Dry-run confirms the emitted command: --min-instances 0 --min 2 --max-instances 8.

🤖 Generated with Claude Code

The production Cloud Run warm floor (service-level minScale=2) was set by
hand and lived only as live service state — invisible to the repo and
reliant on CI never asserting a service-level min. Codify it: the deploy
now passes `--min ${CLOUD_RUN_SERVICE_MIN_INSTANCES}` (production 2,
staging 0) alongside the unchanged revision-level `--min-instances 0`, so
the floor is asserted on every deploy without the per-tag cost bomb.

Also raise production max-instances 4 -> 8 for the 100% Cloud Run cutover:
at the 50/50 split the service already sat pinned at the ceiling of 4
during the daytime peak, so carrying all traffic needs the headroom.

Updates the deploy-script unit tests, the runbook, and adds a changelog
fragment.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Jul 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.03%. Comparing base (e004986) to head (e287591).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #3768   +/-   ##
=======================================
  Coverage   80.03%   80.03%           
=======================================
  Files          70       70           
  Lines        4342     4342           
  Branches      809      809           
=======================================
  Hits         3475     3475           
  Misses        648      648           
  Partials      219      219           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@anth-volk
anth-volk merged commit a21811d into master Jul 23, 2026
10 checks passed
@anth-volk
anth-volk deleted the feat/cloud-run-scaling-config-reconcile branch July 23, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codify Cloud Run scaling settings (service-level warm floor + max-instances) into deploy config

1 participant