Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .agents/skills/chdb-arrow/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
name: chdb-arrow
description: Choose chdb-java optional Arrow module versus row-oriented or standard Arrow drivers for VectorSchemaRoot/ArrowReader input, typed results, or bulk insert.
---

# Java Arrow path selection

Java Database Connectivity (JDBC) returns row-oriented `ResultSet` values. Arrow Database Connectivity (ADBC) is a separate standard driver API. `VectorSchemaRoot` holds an Arrow batch; `ArrowReader` supplies batches on demand.

- Keep row-oriented applications on `chdb-jdbc`. For Java Arrow objects use optional `chdb-arrow` and `ArrowSession`; a caller-owned `VectorSchemaRoot` uses `registerBatch(root, allocator)` and `source.fromSql()`. A one-shot `ArrowReader` uses `registerOwnedReader(reader, allocator)`, which takes ownership of the reader. Keep the allocator alive after the source and query are closed.
- Typed output: `queryArrowMaterialized(sql, allocator)` returns a closable reader after the query; `queryArrow(sql, allocator)` returns `ArrowQuery`, whose `loadNextBatch()` reuses its current root. Bulk insert into an existing table: `insertBatch(table, root, allocator)` or `insertOwnedReader(table, reader, allocator)`. Prefer try-with-resources.
- Use a verified Java ADBC driver-manager pairing when an application already requires standard ADBC result streams or bulk ingest. General ADBC `BindStream` binds parameters, not a temporary `FROM` source; it does not replace `registerOwnedReader`. Do not force Arrow dependencies on JDBC-only consumers. Start with `chdb-arrow/README.md` for this binding's documentation map and examples.
20 changes: 15 additions & 5 deletions chdb-arrow/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,19 @@
# chDB Arrow (in development)

This optional artifact currently contains experimental A1 batch/A2 reader registration,
A3 materialized output, A4 incremental output, and A5 synchronous insert. It is not an implementation of the complete
A1–A6 native Arrow contract.
It does not use Arrow IPC, and it does not promise zero copies inside ClickHouse.
This optional Java module lets an application query an in-memory Arrow batch or one-shot reader, read typed query results all at once or in batches, and insert Arrow data into an existing table. It is experimental; the full ownership, error, compatibility and type tests described by the native Arrow contract remain a separate acceptance gate.
It exchanges Arrow data through the native C Data/C Stream interfaces rather than Arrow Inter-Process Communication (IPC) bytes. C Data shares a schema, buffers and release callbacks; C Stream supplies successive batches. Internal conversion may still copy data.

## Arrow documentation in this Java binding

| What you need | Where to look |
| --- | --- |
| Install the optional Arrow module and matching native package | [Getting started](#getting-started) and [Testing from source](#testing-from-source) below |
| Query an in-memory Java Arrow batch or reader | [Query a caller-owned Arrow batch](#query-a-caller-owned-arrow-batch) and [API and ownership notes](#api-and-ownership-notes) below |
| Read typed results or insert a batch | [Read SQL results as Arrow batches](#read-sql-results-as-arrow-batches), [Insert a batch](#insert-a-batch-into-an-existing-table), and [`ArrowSessionTest`](src/test/java/org/chdb/arrow/ArrowSessionTest.java) |
| Check output type options | [`ArrowOptions`](src/main/java/org/chdb/arrow/ArrowOptions.java) and [its tests](src/test/java/org/chdb/arrow/ArrowOptionsTest.java) |
| Decide on Arrow Database Connectivity (ADBC) instead | An existing, verified Java ADBC client can use standard Arrow results or bulk ingest. It does not replace `registerOwnedReader` for a temporary SQL `FROM` source; Java Database Connectivity (JDBC) `ResultSet` is a separate row-oriented API. |

This module README is the Java binding's Arrow entry point. See the [core ownership contract](https://github.com/chdb-io/chdb-core/blob/main/docs/NATIVE_ARROW_BINDING_CONTRACT.md) for native ABI details; the standard JDBC docs are not Arrow usage instructions.

## Getting started

Expand Down Expand Up @@ -248,7 +258,7 @@ platform, and run its `scripts/build-native.sh <platform-id>` first.

The earlier core `26.11.0-rc.2` omitted a `get_schema` release during registration:
a value-correct Java scan left 148 bytes in the Arrow allocator. The rc.3 release
includes that fix, so **A1/A2/A5 tests are not skipped** on this baseline. CI exercises the
includes that fix, so the Arrow input and insert tests run on this baseline. CI exercises the
same suite on all four native platforms and Java 11/17/21/25 without a version
override. No production deployment should use a local `--local-engine` package or
`chdb.skipEngineVersionCheck`.
Loading