Skip to content

feat: implement native Arrow A1-A6 on chdb-core rc.3 - #104

Merged
wudidapaopao merged 4 commits into
mainfrom
codex/arrow-a1-a6-rc3
Nov 12, 2026
Merged

wudidapaopao merged 4 commits into
mainfrom
codex/arrow-a1-a6-rc3

Conversation

@ShawnChen-Sirius

@ShawnChen-Sirius ShawnChen-Sirius commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • A1/A2: register Arrow tables, batches, and bounded one-shot readers as SQL sources with lifecycle-safe cancellation and cleanup.
  • A3/A4: return typed Arrow tables and incremental batches through the direct C Data/C Stream ABI, including bound queries.
  • A5: insert materialized batches or readers with name-based column mapping and cleanup.
  • A6: require chdb-core v26.11.0-rc.3, map supported types/options, preserve the existing IPC API, and provide the optional chdb/arrow type entry.
  • Exercise the chdb-core #321/#322 registration ownership regression fixed by #323 unconditionally on rc.3. Direct does not imply end-to-end zero-copy.

Validation

  • Official macOS arm64 rc.3 asset SHA-256 verified; the rc.3 tag is the #323 merge commit.
  • Local Node 24 build and version-scheme check passed.
  • v2: 24/24; v3: 540 passed, 11 expected skips.
  • Fresh install from locally packed main and macOS arm64 native tarballs passed scripts/smoke-arrow-e2e.mjs. Other platforms remain CI gates.

Release follow-up

The four @chdb/lib-* 26.11.0-rc.3.1 versions are not on npm yet. The lockfile records the optional dependency target versions but npm cannot add resolved/integrity package entries until publication. Regenerate it after publication and verify registry-installed packages on all supported platforms. See docs/arrow-rc3-conformance.md for supported type boundaries and remaining stress/platform validation. No new core defect was observed; #321/#322 already cover the fixed ownership issue.

Note

Implement native Arrow A1-A6 support on chdb-core v26.11.0-rc.3

  • Adds direct Arrow C Data input and output: registration of column descriptors, record batches, multi-batch tables, and one-shot readers, plus typed query results via queryArrow, queryArrowStream, and bound variants in index.js and arrow-direct.ts
  • Adds Arrow bulk insert via insertArrow and incremental insertArrowReader in arrow-insert.ts, with field-name mapping, quoting, and duplicate-name validation
  • Adds cross-isolate ownership tracking in chdb_node.cpp: connections and Arrow registrations record their owning Node environment, conflicting registrations throw, and environment teardown closes only its own connections
  • Adds lifecycle coordination: registration tokens prevent stale handles from unregistering newer registrations, and teardown defers until tracked Arrow work is idle
  • Exposes a typed chdb/arrow subpath in package.json and pins native platform packages to 26.11.0-rc.3.1
  • Risk: direct Arrow APIs require chdb-core v26.11.0-rc.3; _assertArrowCore in index.js rejects earlier versions (including rc.1/rc.2) with CHDB_ABI, and the rc.3.1 platform packages are not yet published to npm

Macroscope summarized 495560a.

Comment thread lib/chdb_node.cpp Resolved
Comment thread lib/chdb_node.cpp Resolved
Comment thread lib/chdb_node.cpp Resolved
Comment thread lib/chdb_node.cpp Resolved
Comment thread index.js Resolved
Comment thread src/layer3/execute/arrow-reader.ts Resolved
Comment thread src/arrow-direct.ts Resolved
Comment thread src/layer3/execute/arrow-reader.ts Resolved
Comment thread lib/chdb_node.cpp Resolved
Comment thread lib/chdb_node.cpp Resolved
Comment thread lib/chdb_node.cpp Resolved
@ShawnChen-Sirius

Copy link
Copy Markdown
Contributor Author

@wudidapaopao, please review this PR

@wudidapaopao wudidapaopao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@wudidapaopao
wudidapaopao merged commit 718640e into main Nov 12, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants