Skip to content

Staging Cloud Run smoke test intermittently fails on cold-start, skipping the prod Cloud Run deploy #3769

Description

@anth-volk

The Run Cloud Run staging integration tests job runs test_live_calculate.py first, and test_live_calculate_us_1 posts a household calculate directly (api_client.post) with a 90s client timeout and no retry. The staging Cloud Run service is scale-to-zero, so the freshly deployed candidate is cold; the ~200s US tax-benefit-system import loses the race to the 90s timeout and the request fails with httpx.ReadTimeout.

Observed on release run 30014420432 (2026-07-23): 1 failed, 6 passed, the one failure being test_live_calculate_us_1. The polling-based tests survive because _poll_live_endpoint retries on httpx.RequestError; the direct calculate calls do not.

The failure cascades: integration-tests-staging-cloud-runpromote-cloud-run-staging (skipped) → ensure-production-model-version-aligns-with-sim-api (skipped) → deploy-cloud-run-candidate (prod Cloud Run, skipped). So a cold-start flake in a staging smoke test silently blocks the production Cloud Run deploy.

Fix: add an explicit readiness gate that runs first and polls /readiness-check until the deployed revision is warm, so every subsequent test hits a ready instance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions