feat: harden production readiness operations

[abrichr]

What changed

• Validate all four protected backup settings before AWS credential setup or tool installation.
• Require both backup jobs to run from protected main and verify one exact custom main deployment policy before they request AWS credentials.
• Give the workflow token only the read permissions needed to inspect the GitHub environment gate.
• Upload each ciphertext with one S3 PutObject, a caller-supplied full-object SHA-256, and an enforced 5 GiB ceiling.
• Verify the S3 object size and S3-validated full-object checksum after upload and during the independent freshness check.
• Keep one durable issue for backup failures and one for production health failures.
• Validate the complete live readiness contract instead of trusting only ready: true.
• Add an hourly, read-only backup freshness workflow with a separate OIDC role.
• Add one operations runbook for deployment gates, human halt coverage, recovery, external freshness, and false-success reporting.
• Correct the public managed-runtime pin from Flow 1.23.0 to Flow 1.31.0 with the exact release commit and artifact hashes.
• State that a runtime pin proves configured artifact identity. It does not prove live deployment or hosted workflow acceptance.

Why

Issue #126 shows that the daily backup can fail before database access because the protected environment has no settings. The previous health probe could also accept an incomplete response when its top-level boolean was true. GitHub failure email alone did not provide one durable operational signal.

The previous high-level S3 upload could switch to multipart above 8 MiB. A multipart SHA-256 is composite, but the workflow compared it with the full-file SHA-256. A realistic backup could therefore upload successfully and then fail its integrity check. The new launch contract uses one PutObject, lets S3 validate the caller-supplied full-object checksum, verifies it again, and refuses an archive above the 5 GiB single-request limit before upload.

The AWS OIDC roles trust GitHub environment subjects. The environments and repository branch gate must therefore exist before the stack creates those roles. The workflows now fail closed on an unprotected ref, a broad environment policy, a wildcard policy, or more than one policy.

The public deployment matrix and related pages also had a stale Flow 1.23.0 pin. The current openadapt-cloud origin/main manifest pins Flow 1.31.0 at release commit 2d225dea9a0ad29ca84ce1b037cc0ac671367e28, with exact wheel and source archive hashes.

Validation

• python -m pytest -q tests — 184 passed on exact head efd17cc73148cbfeb55d9f896ba986a2e39fb11d
• python scripts/check_published_version_claims.py --offline — passed
• python scripts/validate_docs.py — passed
• mkdocs build --strict — passed
• focused Ruff check — passed
• actionlint .github/workflows/*.yml — passed
• python -m compileall -q scripts tests — passed
• git diff --check — passed
• independent read-only re-review of exact head efd17cc73148cbfeb55d9f896ba986a2e39fb11d — no concrete findings; 184 tests passed again
• live https://app.openadapt.ai/api/health/ready — strict contract passed with 18 required components and 19 reported components
• AWS account identity — 992382684924
• aws cloudformation validate-template — passed

The AWS checks were read-only. The backup stack does not exist. No AWS resource, GitHub environment value, secret, repository rule, or paid service changed.

Founder steps before this is ready

1. Protect main with pull requests and the applicable required status checks.
2. Create production-backup and production-backup-monitor. Give each environment one exact custom main deployment policy. Do not add a wildcard or tag policy. Do not require manual approval for these scheduled jobs.
3. Only after both GitHub gates pass, deploy ops/backup/aws-backup-target.yml in AWS account 992382684924.
4. Configure AWS_BACKUP_ROLE_ARN, AWS_BACKUP_BUCKET, SUPABASE_DB_URL, and SUPABASE_PROJECT_REF in production-backup.
5. Configure AWS_BACKUP_MONITOR_ROLE_ARN and AWS_BACKUP_BUCKET in production-backup-monitor.
6. Store the second private age key copy.
7. Run the first backup and the read-only freshness check from exact main. Confirm that the encrypted archive is no more than 5 GiB and that S3 reports the exact full-object SHA-256.
8. Complete the isolated database and private Storage recovery drill.
9. Assign primary and secondary human-halt operators, support hours, and response targets.
10. Configure one external monitor for the GitHub health and backup-freshness schedules.

Relates to #126. Do not close #126 until the successful backup and remote-checksum exit condition passes.