Skip to content

fix(core): delete every stored copy of a key named without a partition - #1006

Open
PDGGK wants to merge 1 commit into
apache:mainfrom
PDGGK:fix-delete-all-key-copies
Open

PDGGK wants to merge 1 commit into
apache:mainfrom
PDGGK:fix-delete-all-key-copies

Conversation

@PDGGK

@PDGGK PDGGK commented Nov 9, 2026

Copy link
Copy Markdown

Description

Fixes #925.

On a table without the record index, one record key can be stored in several partitions. A non-global Java index leaves a table like that, and append here produces one too. Deleting such a key without naming a partition removes only one copy. After appending a to city=sf and to city=nyc and b to city=sf, hudi 0.6.0 from Python gives this on copy-on-write and on merge-on-read:

delete("id = 'a'")    num_deletes=1   left: ('a', 'nyc'), ('b', 'sf')
delete_keys(['a'])    num_deletes=1   left: ('a', 'nyc'), ('b', 'sf')
delete("id != 'b'")   num_deletes=2   left: ('b', 'sf')

The delete_keys docstring says a bare key is looked up in every partition, and the last line shows the same rows do go away when the predicate does not take the record key path.

Both delete paths resolve keys with tag_location, which returns one location per key. Upsert needs that, because a record has to land in one place. A delete then rewrites or logs only the file group of the copy that was picked.

delete_keys now asks the index for every stored copy a key names: all of them for an empty partition path, and only those in that partition otherwise. SimpleIndex already found every copy during its scan and then kept one, so the scan moved into a helper used by both tag_location and the new lookup. tag_location and upsert behave as before.

The record index is global and holds one location per key, so this change does not alter that path. With default options, where the record index is on, the same appends and delete("id = 'a'") on 0.6.0 leave ('a', 'sf'). I left that alone because finding the other copies would need the scan that index exists to avoid, and the issue is about non-global indexes.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

test_delete_without_partition_removes_every_copy_of_the_key stores a key in two partitions and twice in one of them, on both table types, and deletes it both through delete("id = 'a'") and through delete_keys with an empty partition. On unmodified main it fails with CopyOnWrite, by_filter=true, left: 1, right: 3. Reverting only the copy-on-write or only the merge-on-read part makes it fail for that table type.

crates/core/tests/table_write_tests.rs passes (93 tests), as do the index:: and write::rewrite unit tests (21). cargo fmt --all -- --check and cargo clippy -p hudi-core --all-targets --no-default-features --no-deps -- -D warnings are clean.

I ran these with --no-default-features, so the spill-rocksdb and datafusion builds were not exercised, and the Python binding was only run against the released 0.6.0. The test builds the duplicate key with append; I did not run it on a table written by Java or read the result back with Spark.

A non-global index can store one record key in several partitions.
Deleting that key without naming a partition removed only the copy
tag_location picked, on copy-on-write and merge-on-read tables.

Look up every stored copy for deletes. tag_location and upsert keep
resolving a key to one location.

Fixes apache#925
@PDGGK
PDGGK requested a review from xushiyan as a code owner November 9, 2026 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deleting by record key removes only one copy of a key stored in several partitions

1 participant