Repository navigation
Conversation
A non-global index can store one record key in several partitions. Deleting that key without naming a partition removed only the copy tag_location picked, on copy-on-write and merge-on-read tables. Look up every stored copy for deletes. tag_location and upsert keep resolving a key to one location. Fixes apache#925
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #925.
On a table without the record index, one record key can be stored in several partitions. A non-global Java index leaves a table like that, and
appendhere produces one too. Deleting such a key without naming a partition removes only one copy. After appendingatocity=sfand tocity=nycandbtocity=sf, hudi 0.6.0 from Python gives this on copy-on-write and on merge-on-read:The
delete_keysdocstring says a bare key is looked up in every partition, and the last line shows the same rows do go away when the predicate does not take the record key path.Both delete paths resolve keys with
tag_location, which returns one location per key. Upsert needs that, because a record has to land in one place. A delete then rewrites or logs only the file group of the copy that was picked.delete_keysnow asks the index for every stored copy a key names: all of them for an empty partition path, and only those in that partition otherwise.SimpleIndexalready found every copy during its scan and then kept one, so the scan moved into a helper used by bothtag_locationand the new lookup.tag_locationand upsert behave as before.The record index is global and holds one location per key, so this change does not alter that path. With default options, where the record index is on, the same appends and
delete("id = 'a'")on 0.6.0 leave('a', 'sf'). I left that alone because finding the other copies would need the scan that index exists to avoid, and the issue is about non-global indexes.How are the changes test-covered
test_delete_without_partition_removes_every_copy_of_the_keystores a key in two partitions and twice in one of them, on both table types, and deletes it both throughdelete("id = 'a'")and throughdelete_keyswith an empty partition. On unmodified main it fails withCopyOnWrite, by_filter=true,left: 1,right: 3. Reverting only the copy-on-write or only the merge-on-read part makes it fail for that table type.crates/core/tests/table_write_tests.rspasses (93 tests), as do theindex::andwrite::rewriteunit tests (21).cargo fmt --all -- --checkandcargo clippy -p hudi-core --all-targets --no-default-features --no-deps -- -D warningsare clean.I ran these with
--no-default-features, so thespill-rocksdbanddatafusionbuilds were not exercised, and the Python binding was only run against the released 0.6.0. The test builds the duplicate key withappend; I did not run it on a table written by Java or read the result back with Spark.