Record a coverage backfill after DJ launches it - #2457
Merged
Conversation
✅ Deploy Preview for thriving-cassata-78ae72 canceled.
|
`_plan_coverage_backfill` wrote the record at planning time, but the launch happens after the deploy commits and never raises. A launch the query service refused therefore left a record behind, `backfill_recorded` returned true forever, and the cube was silently locked out of that span. It has happened three times in production, and the only way out was editing `coverage.from`. The record now goes in only once `run_cube_backfill` returns. The swap carries what the write needs -- the materialization's id and the partition column's name, resolved at planning time -- so the write does not depend on ORM state surviving the commit. It commits on its own, and a write that fails is logged and warned about rather than raised at a deploy that has already committed. The job url the query service hands back goes into the row's `urls`, so a person can check what a record stands for.
shangyian
force-pushed
the
backfill-record-after-launch
branch
from
August 25, 2026 12:54
392abe0 to
7bc0edc
Compare
shangyian
marked this pull request as ready for review
August 25, 2026 13:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
We should only record a backfill if it was successfully launched via the query service. If it wasn't, then DJ will never try to reconcile a failed backfill.
With this PR, we only record the backfill in the database after
run_cube_backfillreturns.Test Plan
make checkpassesmake testshows 100% unit test coverageDeployment Plan