Skip to content

perf(core): decode record index entries directly from Avro binary - #1025

Open
dragonknightk wants to merge 1 commit into
apache:mainfrom
dragonknightk:perf/record-index-direct-decode
Open

dragonknightk wants to merge 1 commit into
apache:mainfrom
dragonknightk:perf/record-index-direct-decode

Conversation

@dragonknightk

Copy link
Copy Markdown

Description

Part of #908

decode_record_index_entry called from_avro_datum for every record, and that path re-resolves the full HoodieMetadataRecord schema, including every index payload type, on each call. Record-index entries have a fixed layout, so this change uses direct decode when the container writer schema passes the layout check, and keeps generic decode as the fallback. The fallback to generic decode has two triggers: the layout check fails, or direct decode fails. The layout check is computed once per HFile. On a release build, decoding one record-index entry drops from about 8.8 us to about 0.45 us (four sample shapes rotated for 100000 iterations).

The layout check compares field names, order and types to the vendored HoodieMetadata.avsc. A null union branch is one byte regardless of the other branch. Record-index records leave the other payloads null, so a difference in the column-stats wrapper union still passes. An extra or reordered top-level field, or a change to HoodieRecordIndexInfo itself, fails the layout check, because it can change the bytes.

Swapping fileIdHighBits and fileIdLowBits keeps the same field types, so direct decode would still succeed but return a file ID with its two halves swapped. The layout check rejects that schema.

Column-stats encode and decode are left alone, as issue 908 step 1 asked.

How are the changes test-covered

  • N/A

  • Automated tests (unit and/or integration tests)

  • Manual tests

    • Details are described below
  • Direct decode and generic decode agree field by field, including null and default HoodieRecordIndexInfo fields and a Spark-compacted record-index HFile fixture.

  • Empty bytes and a null recordIndexMetadata stay deletes.

  • The layout check accepts a column-stats-only change and rejects mismatches.

  • decode_record_index_entry matches generic decode for each of those schemas, and when direct decode fails after a passing layout check.

  • Truncated prefixes and fixed corrupt inputs do not panic, and decode_record_index_entry matches generic decode.

  • apply_hfile_records matches generic decode on a schema that fails the layout check.

AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

(๑•̀ㅂ•́)و✧

## Description

Part of issue 908

`decode_record_index_entry` called `from_avro_datum` for every record, and that path re-resolves the full `HoodieMetadataRecord` schema, including every index payload type, on each call. Record-index entries have a fixed layout, so this change uses direct decode when the container writer schema passes the layout check, and keeps generic decode as the fallback. The fallback to generic decode has two triggers: the layout check fails, or direct decode fails. The layout check is computed once per HFile. On a release build, decoding one record-index entry drops from about 8.8 us to about 0.45 us (four sample shapes rotated for 100000 iterations).

The layout check compares field names, order and types to the vendored `HoodieMetadata.avsc`. A null union branch is one byte regardless of the other branch. Record-index records leave the other payloads null, so a difference in the column-stats wrapper union still passes. An extra or reordered top-level field, or a change to `HoodieRecordIndexInfo` itself, fails the layout check, because it can change the bytes.

Swapping `fileIdHighBits` and `fileIdLowBits` keeps the same field types, so direct decode would still succeed but return a file ID with its two halves swapped. The layout check rejects that schema.

Column-stats encode and decode are left alone, as issue 908 step 1 asked.

## How are the changes test-covered

- [ ] N/A
- [x] Automated tests (unit and/or integration tests)
- [ ] Manual tests
  - [ ] Details are described below

- Direct decode and generic decode agree field by field, including null and default `HoodieRecordIndexInfo` fields and a Spark-compacted record-index HFile fixture.
- Empty bytes and a null `recordIndexMetadata` stay deletes.
- The layout check accepts a column-stats-only change and rejects mismatches.
- `decode_record_index_entry` matches generic decode for each of those schemas, and when direct decode fails after a passing layout check.
- Truncated prefixes and fixed corrupt inputs do not panic, and `decode_record_index_entry` matches generic decode.
- `apply_hfile_records` matches generic decode on a schema that fails the layout check.

## AI Disclosure

Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.

Signed-off-by: dragonknightk <37928252+dragonknightk@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant