Skip to content

Align Rust Arrow lifecycle with chdb-core rc.3 contract - #68

Merged
wudidapaopao merged 2 commits into
mainfrom
codex/arrow-contract-rc3
Nov 12, 2026
Merged

wudidapaopao merged 2 commits into
mainfrom
codex/arrow-contract-rc3

Conversation

@ShawnChen-Sirius

@ShawnChen-Sirius ShawnChen-Sirius commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add an owned, scoped Arrow reader registration that keeps its C Stream alive through lazy queries and unregisters on close/drop, while preserving the existing raw registration interface.
  • Preserve the schema for empty streaming Arrow results, distinguish schema failures from EOF, cancel on fetch/conversion errors, and retain the original insert error when cleanup also fails.
  • Advance the default chdb-core engine pin from v26.11.0-rc.2 to v26.11.0-rc.3 and reject a fetched engine being shadowed by a different runtime libchdb version. Clarify that direct Arrow avoids IPC but does not guarantee buffer-level zero-copy.

Compatibility

No existing Rust Arrow method signatures change. The scoped registration is additive; raw-pointer registration keeps its existing caller-managed lifetime preconditions. The default downloaded engine changes to rc.3 on new builds. This does not publish a new crate release.

Testing

  • macOS arm64 with the v26.11.0-rc.3 release libchdb.so explicitly selected at runtime: 60 library tests, 6 Arrow insert, 11 Arrow query, 4 scoped registration, 3 Arrow stream roundtrip, and 5 version tests passed (89 total).
  • cargo test --no-default-features --lib --offline -- --test-threads=1: 43 passed with the same runtime library.
  • cargo fmt --check, sh .github/scripts/check-engine-pin.sh, and git diff --check origin/main...HEAD passed.

Linux and the other platform architectures were not run locally; CI should validate them. A first local run without an explicit macOS runtime library path loaded an older 26.7.3 library despite linking against rc.3, which motivated the new version mismatch error. The results above used the rc.3 library and passed the version equality test.

Note

Add scoped ArrowRegistration and align engine pin with chdb-core rc.3

  • Adds ArrowRegistration, a scoped handle that owns a boxed RecordBatchReader and its FFI stream for the lifetime of a registration. It offers query, query_arrow, query_stream_arrow, query_stream, sql_expr, and an explicit close; Drop unregisters and leaks the stream if unregistration fails. Created via the new Connection::register_arrow_reader and re-exported at the crate root under the arrow feature
  • ArrowQueryStream now stores the fetched schema (exposed by a new schema() accessor), cancels the query on fetch or read errors, and treats a failed schema callback as EOF only when the next batch is empty with no stream error
  • Registered Arrow insert paths route through finish_registered_insert, so a failing insert reports the query error while also reporting a cleanup failure when both occur
  • Connection::open now calls verify_loaded_engine before connecting, erroring when the loaded library version differs from EXPECTED_ENGINE_VERSION; the engine pin moves to v26.11.0-rc.3 across Cargo.toml, build.rs, and update_libchdb.sh
  • Risk: version mismatch now rejects Connection::open with InvalidData (see version.rs); streams that fail schema retrieval but still have data may now be classified as EOF only per the probe in is_eos_without_schema (arrow_query_stream.rs)

Macroscope summarized 207f213.

Add an owned scoped registration for Arrow readers, preserve empty-result schemas and stream errors, and verify the loaded engine matches the fetched release. Cover cancellation, cleanup, and existing input/output paths.
@ShawnChen-Sirius

Copy link
Copy Markdown
Contributor Author

@wudidapaopao, please review this PR

@wudidapaopao wudidapaopao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wudidapaopao
wudidapaopao merged commit 96ddbec into main Nov 12, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants