Let deploy recovery reach reservations made by the current stable deploy - #2532
Merged
Merged
Conversation
Recovery rebuilt the pre-2026-08-05 idempotency key and payload, so every reservation made by the current reusable stable deploy came back as reservation_not_found. The action now builds the artifact-scoped key and sends deploy_reference like the original deploy did. The run-scoped key stays available by explicit deploy_key_format. A provider-evidence mode lets the reusable workflow reuse this one key builder. Refs #2531
Pin the reusable workflow's recovery steps to the action that rebuilds the current stable-deploy key, and replace the workflow's own shell copy of the provider-evidence key and payload with the action's provider-evidence mode. The dispatch path now carries the original deploy_reference. A regression test runs the real stable-deploy request steps and requires recovery to send the same key and payload. Refs #2531
Contributor
Author
|
Review by another model. OpenAI
Also noted: |
This was referenced Sep 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
RepairShopr (
repairshopr-sync/prod) has had every deploy blocked since 2026-09-12 by a fenced reservation from run 34724620086. The owner-approved recovery dry-run (run 36344798857) returned 404reservation_not_found, so the supported recovery path cannot reach it. The same failure happened on 2026-07-17 (run 29609495343); it was recovered through #2167 / #2187 only because that reservation still used the old key.Recovery looks a reservation up by its exact
Idempotency-Keyand the fingerprint of the exact original payload. Since 2026-08-05 the reusable stable deploy builds the key asgeneric-web-stable-deploy:{product}:{instance}:{artifact_id}:{deploy_reference}:{run_id}:1and sendsdeploy.deploy_reference. The recovery action and the workflow's provider-evidence step still rebuilt the older...:{instance}:{run_id}:{attempt}key withoutdeploy_reference. Neither matched any reservation made after 2026-08-05.This PR fixes the first finish-line item of #2531 only. Lock expiry, alerting on a held fence, and the ~5-minute trigger remain open there.
What changed
.github/actions/generic-web-deploy-recovery-dry-run: builds the current (artifact_scoped) key and sendsdeploy_referencein the original payload, empty when the request omits it, as the original deploy did. It accepts an optionaldeploy_reference. Reservations made before 2026-08-05 are still reachable with an explicitdeploy_key_format: run_scoped. A newmode: provider-evidencesends the same request to the provider-evidence route..github/workflows/reusable-generic-web-stable-deploy.yml: both recovery steps now use that action, pinned to 6bac61a in this PR, so the workflow no longer keeps its own shell copy of the key and payload (60 lines removed). Theworkflow_dispatchrecovery path now passes the originaldeploy_reference.docs/operations.md: documents both key formats and the attempt-1rule for rerun deploys.No service code changes. The server already looks up the exact key and fingerprint the client sends.
Safety
Recovery still names exactly one reservation: one full key plus the fingerprint of one full original payload, and the service's exact lookup is unchanged. There is no prefix, wildcard, or format fallback. A wrong format, attempt, or coordinate gets 404 and changes nothing. The key format is chosen explicitly (
artifact_scopedby default,run_scopedonly on request), and malformed values are rejected before any OIDC or network call. Apply still requires the artifact-backed, digest-bound workflow_run provenance, and provider-evidence mode refuses a digest. What a recovery can release or adopt is still decided server-side by the dry-run classification and the reviewed digest. This PR does not change that.Test evidence
test_action_reaches_reservation_created_by_current_stable_deployruns the reusable workflow's realstable-deployrequest steps: the key-building shell step, pluslaunchplane-requestwith the workflow's ownpayloadandpayload-fields. It then requires the recovery action, in dry-run and provider-evidence modes, to send the sameIdempotency-Keyand anoriginal_deployidentical to the deploy payload. It covers an emptydeploy_reference(the RepairShopr case) and a non-empty one. The expected key is never copied into the test.origin/main's action it fails:generic-web-stable-deploy:example-product:prod:34724620086:1!=generic-web-stable-deploy:example-product:prod:ghcr.io/example/product@sha256:…::34724620086:1, anddeploy_referenceis rejected as an unsupported field.run_scoped, and a new test rejects an unknown format,run_scopedwithdeploy_reference, and a paddeddeploy_referencebefore any network call.docs/operations.mdpass (237 tests). ruff, ruff format and mypy are clean on the changed test. JetBrains inspection (changed files) is GREEN.Merge note
Land with a merge commit, not squash or rebase, so the pinned action commit 6bac61a stays reachable from
main.After merge
@main, and the fix is entirely in the workflow and action.Launchplane Recovery Requestwith the same inputs as run 36344788668 (original_run_id34724620086,original_run_attempt1). It should return a recovery digest instead ofreservation_not_found.recovery-applyjob pins this action at01198841, which still builds the old key. It needs a product-repo pin bump to a commit containing this fix before a reviewed apply can reach the reservation.Refs #2531