Await R2 readbacks across public-channel propagation lag - #32
Merged
Merged
Conversation
Promote run 36187580126 aborted because the aws s3api put for terraphim-agent-1.21.16-aarch64-unknown-linux-musl.tar.gz was acknowledged but the immediate readback through downloads.terraphim.ai returned 404; the object surfaced on the public channel roughly five minutes later with cf-cache-status DYNAMIC, so the 404s were origin responses behind a propagation lag, not edge-cache staleness or a failed write. Post-put readbacks now retry every five seconds for R2_READBACK_WAIT seconds (default 600, 0 through 3600 validated) before declaring an object or stable pointer absent. The promotion contract gains a lag-simulating curl stub mode pinning both the retried success and the bounded-timeout failure that must stop before any stable write. Refs #337
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Promote run 36187580126 (with the S3 transport from #31) aborted on
ERROR: uploaded R2 object is absent: terraphim-agent-1.21.16-aarch64-unknown-linux-musl.tar.gz. Theaws s3api put-objectwas acknowledged, the readback two seconds later returned 404, and the object surfaced ondownloads.terraphim.airoughly five minutes after the write. Response headers showcf-cache-status: DYNAMIC, so the 404s were origin responses behind a propagation lag on the public custom domain; this also retroactively explains run 36185425648, whose "vanished" wrangler upload was a readback racing the same lag.Change
scripts/promote-release.sh: post-put readbacks for immutable objects and stable pointers now retry every five seconds forR2_READBACK_WAITseconds (default 600, validated 0 through 3600) before the run declares the object absent and stops. Preflight/immediate reads keep their single-shot strict semantics, and the byte comparison of every readback is unchanged.tests/test_promotion_contract.py: the curl stub gainsLAG_KEY/LAG_READS(origin 404s for the first N reads of a key that exists). One test pins the retried success including the exact read count, and theR2_READBACK_WAIT=0bounded-timeout failure that must stop before any stable write. All 21 promotion contract tests pass locally.docs/release-operator-checklist.md: documents the lag and the wait knob.Re-dispatching after merge is safe and idempotent: live objects are preflight-compared and skipped; the two objects already written by run 36187580126 will be found identical.
Refs #337