Codify Cloud Run scaling settings into deploy config (warm floor + max-instances 8)#3768
Merged
Merged
Conversation
The production Cloud Run warm floor (service-level minScale=2) was set by
hand and lived only as live service state — invisible to the repo and
reliant on CI never asserting a service-level min. Codify it: the deploy
now passes `--min ${CLOUD_RUN_SERVICE_MIN_INSTANCES}` (production 2,
staging 0) alongside the unchanged revision-level `--min-instances 0`, so
the floor is asserted on every deploy without the per-tag cost bomb.
Also raise production max-instances 4 -> 8 for the 100% Cloud Run cutover:
at the 50/50 split the service already sat pinned at the ceiling of 4
during the daytime peak, so carrying all traffic needs the headroom.
Updates the deploy-script unit tests, the runbook, and adds a changelog
fragment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3768 +/- ##
=======================================
Coverage 80.03% 80.03%
=======================================
Files 70 70
Lines 4342 4342
Branches 809 809
=======================================
Hits 3475 3475
Misses 648 648
Partials 219 219 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3767
Why
Two production Cloud Run scaling settings were being managed by hand and were not reflected in the repo:
minScale=2). Set manually viagcloud --minduring the PR 4 host cutover; it lived only as live service state. It survives CI deploys today only because the deploy never asserts a service-level min — invisible to anyone reading the repo, and one--min 0or service recreation away from silently vanishing. Overnight sweeps confirm this floor is what holds the service at 2 warm instances through the bot trough with zero cold-start churn.max-instances 4. At the current 50/50 LB split the service already sits pinned at the ceiling of 4 during the daytime peak (instance_countflatlines at 4.0). The imminent 100% Cloud Run cutover roughly doubles load, so 4 would bottleneck.What
CLOUD_RUN_SERVICE_MIN_INSTANCESenv (default 0 incloud_run_env.sh; prod2, staging0inpush.yml). The deploy now passes--min ${CLOUD_RUN_SERVICE_MIN_INSTANCES}— the service-level floor — on every deploy.--min-instances 0unchanged. A positive revision-level min is baked immutably onto every new revision, so each tagged no-traffic candidate would pin its own warm 16Gi instance (per-tag cost bomb). The existing invariant test guarding this stays green.max-instances 4 → 8in the production job.docs/migration/cloud-run-operations.md) updated: warm floor is now applied by CI, not manually-once; max sizing rationale recorded.--minemitted, prod max8, per-job service-min values); changelog fragment added.Live state
The live production service was already raised to
max-instances 8(new revision00045-2rd, serving 100%) and carriesminScale 2out-of-band, so production matches this config now. This PR makes both durable across future releases — the production Cloud Run deploy only runs on the release commit, so without it the next release would revertmaxto 4.Testing
Dry-run confirms the emitted command:
--min-instances 0 --min 2 --max-instances 8.🤖 Generated with Claude Code