Proposal: harden the deploy-and-verify pipeline against cold-start and edge transients

[anth-volk]

Status: proposal, not a plan. This issue collects research on standard practice plus a root-cause analysis of a real incident, to frame a discussion about hardening the pipeline. It is deliberately not a committed work plan — each item below needs its own scoping, and several interact with the artifact-pipeline phases (#643).

Motivating incident

The first deploy after merging #644 (run 30050204366, 2026-07-23) failed two integration tests. Neither was a product bug:

• test_auth_smoke.py::test_gated_endpoint_rejects_missing_token — the gateway returned the correct 403, but logged it ~35s after the request was sent: the first real request after the deploy paid the fresh web container's cold boot, and the test's fixed 30s httpx timeout expired first. The second auth test passed seconds later on the warmed container.
• test_budget_window.py::test_budget_window_multi_year_batch_completes — the batch itself completed perfectly (verified by fetching the parent FunctionCall record directly from Modal: status=complete, progress=100, both children complete, all assertions in the test pass against the served payload). The gateway logs show polls at 22:36:17 and 22:36:22 and then a 160-second silent window before a healthy 200 — while the client was polling every 5s. Requests that die at Modal's ingress during post-deploy container churn never reach app code (nothing to log), and the poll helper hard-fails on any status outside {200, 202, 500}. One edge transient killed a test that had ~22 minutes of budget left.

Both tests passed on the previous main deploy. The difference: #644 changed image dependencies, so every layer rebuilt and every container was maximally cold when the suite hit it three minutes after deploy.

(Separate observation, worth its own follow-up: a Runner segmentation fault (SIGSEGV) in the executor app ~15s after the batch completed, in a container that had just run two year-sims back-to-back at 13.3GB RSS against the cache's 8GB warning threshold. Pre-existing accumulation behavior, not caused by #644, but a one-shot CI suite would never catch it — see layer 5.)

The anti-pattern

Deploy, then immediately fire one-shot assertions at maximally-cold infrastructure, with a test client that cannot distinguish infrastructure noise from product failure.

Standard-practice layers (ordered by leverage-per-effort for this repo)

1. Readiness gating

Poll a health/readiness endpoint until N consecutive fast successes before any functional test runs (the CI analogue of Kubernetes readiness probes). We already have /health; the integ job just doesn't gate on it. A ~10-line "3 consecutive 200s under 2s" step would have absorbed the entire gateway cold boot.

2. Explicit warmup

• Synthetic warmup traffic: throwaway requests through each critical route until latency stabilizes, then assert.
• Platform-native warm capacity: Modal's min_containers (floor above zero — the gateway web app is a lightweight always-on front door, exactly the intended use), buffer_containers, scaledown_window (up to 20 min), and memory snapshots (already used for executor model pre-loads; confirm the gateway app has snapshotting enabled). min_containers=1 on the gateway makes the auth-flake class structurally impossible.

3. Client-side error taxonomy

The flaky-test literature is unanimous: transport errors and edge 5xx (502/503/504, connection reset, timeout) are retryable with backoff and a ceiling (2 retries max); assertion failures and app-level errors are never retried. Our poll helper is currently the inverted version — it should get a retry budget for the transient class while keeping instant failure for genuine app errors. The auth smoke needs a cold-aware first-request timeout or one retry on timeout.

4. Smoke-then-integ separation

A tiny smoke gate (health, auth round-trip, one cheap request) immediately post-deploy; the expensive integ suite runs only after smoke is green. Diagnostic payoff: smoke failure = deploy/infra problem, integ failure = product problem — the suite structure itself attributes the failure.

5. Decouple deploy from verification (progressive delivery)

Per Google SRE canarying / blue-green practice: deploy without traffic, warm and verify directly, then shift traffic, then keep watching during a bake window. Our executor architecture is already blue-green-shaped (versioned apps resolved by the gateway), so verify-before-route-flip is mostly sequencing, not architecture. Post-deploy, one-shot CI should hand off to scheduled synthetic probes (we have Logfire) — that is what catches a post-test SIGSEGV or slow memory decay, which no CI moment can.

6. Flake discipline

Retry-with-tracking (e.g. pytest-rerunfailures with reporting, never silent), quarantine lane for confirmed-flaky tests (non-blocking, ticketed, fix-or-delete deadline), and flake rate tracked as a metric.

Suggested sequencing (cheap first) — again, for discussion

1. Readiness gate + warmup step in the integ job (trivial; kills the auth class).
2. Retry taxonomy in the poll helper/client for {502, 503, 504, transport} with a 2-retry ceiling (small; kills the budget-window class).
3. min_containers=1 + snapshot check on the gateway web app (config-only, small cost).
4. Split smoke job from integ job (workflow reshuffle; attribution payoff).
5. Later: verify-before-route-flip on the versioned apps; scheduled synthetic probes. Pairs naturally with the artifact pipeline (Artifact pipeline phase 1: content-addressed keys, store, runtime reader, precompute app #643) — artifact-fetch deploys become fast enough to afford richer pre-traffic verification.

Sources

• Harness — integrating smoke testing into CI/CD
• Statsig — smoke tests as pipeline gates
• Modal — cold start performance and memory snapshots
• AWS Lambda cold-start mitigation techniques; warmup vs provisioned concurrency cost analysis
• pytest — flaky tests; flaky-test detection/quarantine guide; OneUptime — fixing flaky tests in CI/CD
• Google SRE Workbook — canarying releases; Harness — blue-green & canary strategies; Bug0 — deployment testing

🤖 Generated with Claude Code