Repository navigation
perf(core): decode record index entries directly from Avro binary - #1025
Open
dragonknightk wants to merge 1 commit into
Open
dragonknightk wants to merge 1 commit into
dragonknightk wants to merge 1 commit into
Conversation
## Description Part of issue 908 `decode_record_index_entry` called `from_avro_datum` for every record, and that path re-resolves the full `HoodieMetadataRecord` schema, including every index payload type, on each call. Record-index entries have a fixed layout, so this change uses direct decode when the container writer schema passes the layout check, and keeps generic decode as the fallback. The fallback to generic decode has two triggers: the layout check fails, or direct decode fails. The layout check is computed once per HFile. On a release build, decoding one record-index entry drops from about 8.8 us to about 0.45 us (four sample shapes rotated for 100000 iterations). The layout check compares field names, order and types to the vendored `HoodieMetadata.avsc`. A null union branch is one byte regardless of the other branch. Record-index records leave the other payloads null, so a difference in the column-stats wrapper union still passes. An extra or reordered top-level field, or a change to `HoodieRecordIndexInfo` itself, fails the layout check, because it can change the bytes. Swapping `fileIdHighBits` and `fileIdLowBits` keeps the same field types, so direct decode would still succeed but return a file ID with its two halves swapped. The layout check rejects that schema. Column-stats encode and decode are left alone, as issue 908 step 1 asked. ## How are the changes test-covered - [ ] N/A - [x] Automated tests (unit and/or integration tests) - [ ] Manual tests - [ ] Details are described below - Direct decode and generic decode agree field by field, including null and default `HoodieRecordIndexInfo` fields and a Spark-compacted record-index HFile fixture. - Empty bytes and a null `recordIndexMetadata` stay deletes. - The layout check accepts a column-stats-only change and rejects mismatches. - `decode_record_index_entry` matches generic decode for each of those schemas, and when direct decode fails after a passing layout check. - Truncated prefixes and fixed corrupt inputs do not panic, and `decode_record_index_entry` matches generic decode. - `apply_hfile_records` matches generic decode on a schema that fails the layout check. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: dragonknightk <37928252+dragonknightk@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Part of #908
decode_record_index_entrycalledfrom_avro_datumfor every record, and that path re-resolves the fullHoodieMetadataRecordschema, including every index payload type, on each call. Record-index entries have a fixed layout, so this change uses direct decode when the container writer schema passes the layout check, and keeps generic decode as the fallback. The fallback to generic decode has two triggers: the layout check fails, or direct decode fails. The layout check is computed once per HFile. On a release build, decoding one record-index entry drops from about 8.8 us to about 0.45 us (four sample shapes rotated for 100000 iterations).The layout check compares field names, order and types to the vendored
HoodieMetadata.avsc. A null union branch is one byte regardless of the other branch. Record-index records leave the other payloads null, so a difference in the column-stats wrapper union still passes. An extra or reordered top-level field, or a change toHoodieRecordIndexInfoitself, fails the layout check, because it can change the bytes.Swapping
fileIdHighBitsandfileIdLowBitskeeps the same field types, so direct decode would still succeed but return a file ID with its two halves swapped. The layout check rejects that schema.Column-stats encode and decode are left alone, as issue 908 step 1 asked.
How are the changes test-covered
N/A
Automated tests (unit and/or integration tests)
Manual tests
Direct decode and generic decode agree field by field, including null and default
HoodieRecordIndexInfofields and a Spark-compacted record-index HFile fixture.Empty bytes and a null
recordIndexMetadatastay deletes.The layout check accepts a column-stats-only change and rejects mismatches.
decode_record_index_entrymatches generic decode for each of those schemas, and when direct decode fails after a passing layout check.Truncated prefixes and fixed corrupt inputs do not panic, and
decode_record_index_entrymatches generic decode.apply_hfile_recordsmatches generic decode on a schema that fails the layout check.AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.
(๑•̀ㅂ•́)و✧