Accept multiple materialization configs on a cube - #2460
Merged
Conversation
A cube often needs two builds rather than one. An incremental_time materialization keeps its Druid datasource fresh, and a periodic full rebuild rewrites that datasource so late-arriving data, out-of-order events and dimension backfills are eventually reflected in it. Cubes in production already run exactly that pair, configured outside the repo, and a `materialization:` block that could name only one left them undeclarable: pushing such a cube from YAML would have superseded whichever of the two builds the block did not name. The block now accepts a list as well as a single spec. The scalar form is unchanged and remains how the common case should be written -- this is a widening, so no manifest has to be rewritten as a one-element list. Entries are told apart by `strategy`, which is the only identity available to an author, since `job` is deliberately not exposed: a cube has exactly one correct job type, and the others that appear in production data are old mistakes rather than choices. A list that repeats a strategy therefore has no meaning and is rejected at parse time rather than deployed ambiguously, and so is an empty list, which cannot be told apart from the `none` teardown sentinel. Reconciliation builds every declared block and supersedes only the rows that no block built, which is the rule it always applied, read over a set of names instead of a single one. A revision swap matches each row it rebuilds to the block naming that row's strategy and falls back to the first block otherwise, so a cube declaring one behaves exactly as it did before. Export projects a cube's materializations back as a list once it has more than one, so a pull followed by a push leaves the cube as it found it, and the comparison that decides whether a deploy changed anything ignores the order the blocks were written in -- the server lists them in name order and an author is free to write them the other way round. Comparing two declarations this way also required repairing the list-field diff helper, which reduced a model to a tuple of its field values and so could not put a model carrying a nested one into a set at all -- a materialization's `coverage`, or a column's `partition`. It now compares canonical JSON. Nothing here reconciles a materialization the cube no longer declares. Dropping an entry from the list still leaves the corresponding row running, exactly as omitting the block entirely does, because deleting a materialization stops a live workflow and deserves its own decision.
✅ Deploy Preview for thriving-cassata-78ae72 canceled.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A cube often needs two materializations rather than one: an
incremental_timebuild for freshness, alongside a periodicfullrebuild that corrects late-arriving data, out-of-order events and dimension backfills. A cube with ten years of history and a slowly-changing dimension can't stay correct on incremental loads alone.With this PR, the
materialization:block now accepts a list as well as a single definition:Note that
strategyis what tells two entries apart.The declared blocks describe the materializations for their cube, so any active row with no matching YAML block is deactivated. That's a change from the single-block form, where an absent key meant "leave whatever exists alone": a materialization created through the UI or API is now superseded by a deploy that declares one. Omitting the key entirely still leaves everything alone, and none still tears everything down.
Everything downstream reconciles per entry, including declared coverage, so the full rebuild and the incremental load can maintain different spans.
Test Plan
make checkpassesmake testshows 100% unit test coverageDeployment Plan