Skip to content

feat(core): support native Arrow string and binary append types - #1024

Open
GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:feat/864-arrow-view-write-types
Open

GayathriSrividya wants to merge 1 commit into
apache:mainfrom
GayathriSrividya:feat/864-arrow-view-write-types

Conversation

@GayathriSrividya

Copy link
Copy Markdown
Contributor

Closes #864

Description

Preserve Utf8View, LargeUtf8, BinaryView, and LargeBinary arrays through the core append normalization path instead of casting them to canonical Arrow types. Remove the corresponding Python-side casts, and accept equivalent string/binary types when matching append schemas. Extend partition-path handling and add Rust and Python regression coverage, including multiple appends and readback.

How are the changes test-covered

  • N/A
  • Automated tests (unit and/or integration tests)
  • Manual tests
    • Details are described below

Validation:

  • cargo test --offline --locked -p hudi-core --no-default-features --lib write:: — passed (89 tests).
  • cargo test --offline --locked -p hudi-core --no-default-features --lib keygen::record_key::tests::test_render_key_field_scalar_types — passed.
  • ruff format --check python/tests/test_table_write.py && ruff check python/tests/test_table_write.py — passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Accept Utf8View, LargeUtf8, BinaryView and LargeBinary columns in the write path

1 participant