Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/chdb-arrow/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ Arrow Inter-Process Communication (IPC) is a serialized byte format; the direct

- For an `apache-arrow.Table` / `RecordBatch` as a temporary SQL source, use `registerArrowSource(input, { session })` and query `arrowstream('${source.name}')`; close the source in `finally`. For a one-shot sync/async batch iterable, `await registerArrowReader(iterable, { session })` and later `await handle.close()`. Use async queries while a JS producer refills the native queue.
- Typed results: `await session.queryArrow(sql)` returns a whole `Table`; `session.queryArrowStream(sql)` yields batches via `for await` and needs `reader.cancel()` if stopped early. Bulk insert: `await insertArrow(table, input, { session })` or `await insertArrowReader(table, iterable, { session })`.
- `queryAsync(sql, { format: 'arrow' }).toArrow()` remains an Arrow IPC compatibility route, not direct C Data. Prefer a verified ADBC client only if the application already uses ADBC for standard results or bulk ingest; general ADBC `BindStream` does not create a temporary SQL source. Install the optional `apache-arrow` peer. Start with `docs/arrow.md` for this binding's documentation map, examples, types, and isolate constraints.
- `queryAsync(sql, { format: 'arrow' }).toArrow()` remains an Arrow IPC compatibility route, not direct C Data. Prefer a verified ADBC client only if the application already uses ADBC for standard results or bulk ingest; general ADBC `BindStream` does not create a temporary SQL source. Install the optional `apache-arrow` peer. Start with the [Node Arrow guide](../../../docs/arrow.md) for examples and supported types, then its release-validation link for the tested core and platform matrix.
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ unsupported (use WSL2).
Run SQL you have, or anything ClickHouse-specific? -> raw SQL (query / sql``)
Build queries programmatically, type-safe + bound? -> fluent (selectFrom / insertInto) [src/layer3/AGENTS.md]
Read/JOIN external sources (S3/Postgres/…)? -> connect({ url }) [src/layer3/AGENTS.md]
Query/insert in-memory Arrow or read typed batches? -> direct Arrow [docs/arrow.md]
Migrating from @clickhouse/client (drop-in)? -> chdb/connection [src/connection/AGENTS.md]
Need the database to survive losing the machine? -> chdb/durable (experimental) [docs/design/durable-control-plane.md]
```
Expand Down Expand Up @@ -68,4 +69,5 @@ const rows = await session('./db').selectFrom('events').selectAll().where('ts',
- **No SQL injection** with `queryBind`, the `sql` tag, or the fluent builder (values bound server-side).

Detail: fluent + federation → `src/layer3/AGENTS.md`; `@clickhouse/client` compat →
`src/connection/AGENTS.md`; full API reference → `llms-full.txt`.
`src/connection/AGENTS.md`; direct Arrow → `docs/arrow.md` and
`.agents/skills/chdb-arrow/SKILL.md`; full API reference → `llms-full.txt`.
28 changes: 12 additions & 16 deletions docs/arrow-rc3-conformance.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
# Native Arrow validation against chdb-core v26.11.0-rc.3
# Historical Arrow validation against chdb-core v26.11.0-rc.3

This is a pre-release snapshot, not the current status of npm packages or
platform validation. For the released `chdb@3.6.0` and `v26.11.1` engine, see
the [current Arrow release validation](arrow-release-validation.md).

The development baseline is the `macos-arm64-libchdb.tar.gz` asset from
`v26.11.0-rc.3` (SHA-256 `7b0ea049efcf75a851e744c7d74325d7a9b143227f9b72097e5df252564e3d58`).
The manifest and lockfile target `@chdb/lib-*` version `26.11.0-rc.3.1`, but those npm packages
have **not yet been published**. npm omitted their package metadata from the
lockfile; regenerate it after publication to capture `resolved` and `integrity`.
The local macOS arm64 tests used the official rc.3
tarball with the previous ignored `libchdb.so` backed up under
`/tmp/chdb-node-rc3-verify/`. A cleanroom install from locally packed main and
At the time of this snapshot, the manifest targeted `@chdb/lib-*`
`26.11.0-rc.3.1` and the packages had not yet been published. The local
macOS arm64 tests used the official rc.3 tarball with the previous ignored
`libchdb.so` backed up under `/tmp/chdb-node-rc3-verify/`. A cleanroom install from locally packed main and
arm64 subpackage tarballs passed `smoke-arrow-e2e.mjs`. The other three
platforms and registry-installed packages remain CI/release gates.
platforms and registry-installed packages were then outstanding release gates.

The shared contract is the [core native Arrow binding contract](https://github.com/chdb-io/chdb-core/blob/main/docs/NATIVE_ARROW_BINDING_CONTRACT.md). In
The shared contract is the [rc.3 core native Arrow binding contract](https://github.com/chdb-io/chdb-core/blob/v26.11.0-rc.3/docs/NATIVE_ARROW_BINDING_CONTRACT.md). In
this binding, **direct** means Arrow's in-memory C Data/C Stream interfaces
without Inter-Process Communication (IPC) serialization. Input and output
may copy buffers; no end-to-end zero-copy claim is made. An Arrow schema names
Expand All @@ -23,7 +25,7 @@ queue and release sequence see the [Node Arrow design](design/native-arrow.md).
The optional `chdb/arrow` subpath supplies TypeScript `Table`/`RecordBatch`
return types without making the base `chdb` entry require `apache-arrow`.

| Capability | Implemented now | Further validation / documented scope |
| Capability | Implemented at the rc.3 snapshot | Outstanding at the rc.3 snapshot / documented scope |
| --- | --- | --- |
| In-memory SQL source | `registerArrowTable` accepts supported column descriptors, `RecordBatch`, and materialized multi-batch `Table`; C Stream callbacks use pinned JS buffers. `registerArrowSource` generates opaque process-unique names. Primitive, Timestamp, Decimal128/256, sliced buffers/bitmaps, and one-chunk Dictionary<Int32/UInt32, Utf8/Binary> input are tested. Borrowed sources cannot coexist with a Session in another Node isolate. | Nested types and multi-chunk dictionary input explicitly reject; more platform/isolate stress remains. |
| Reader as SQL source | `registerArrowReader` accepts sync/async RecordBatch iterables. The JS producer awaits capacity in a two-batch native queue; worker-thread `get_next` never calls JS. Empty schema, producer error, schema mismatch, cancellation, one-release-per-delivered-batch, one-shot behavior and worker-isolate teardown are tested. | ASan/RSS stress and platform matrix. Core may retain batches after pulling them (e.g. sorting); queue capacity counts only batches not yet handed to core. `close()` cancels native promptly even if a user producer never resolves `next()`; that producer must cooperate to release its own resources. |
Expand All @@ -39,9 +41,3 @@ failed. The rc.3 engine fixes both ownership paths. The registration-schema rele
now runs unconditionally on rc.3. An ASan/RSS *failed-registration leak budget*
is still a future stress gate, not a test claimed here. Node's own close/eviction
path unregisters from core before dropping pinned buffers.

If another **new** rc.3 core defect appears: keep a small reproducer, file a
core issue before designing around its proposed fix, link the issue from a
clearly skipped/conditional test, and describe the required follow-up. Filing
an issue is a remote write and requires separate per-action confirmation; other
local implementation and unaffected tests can continue meanwhile.
58 changes: 58 additions & 0 deletions docs/arrow-release-validation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# Direct Arrow release validation for chdb 3.6.0

`chdb@3.6.0` pins chdb-core `v26.11.1` through the four platform packages
`@chdb/lib-{darwin-arm64,darwin-x64,linux-arm64-gnu,linux-x64-gnu}@26.11.1-stable.1`.
Install the optional `apache-arrow` peer to use typed Arrow objects:

```sh
npm install chdb@3.6.0 apache-arrow
```

The [v26.11.1 native Arrow binding contract](https://github.com/chdb-io/chdb-core/blob/v26.11.1/docs/NATIVE_ARROW_BINDING_CONTRACT.md)
defines direct C Data/C Stream transfer, ownership, and compatibility. "Direct"
excludes Arrow IPC serialization at the binding boundary; it does **not** promise
end-to-end zero-copy. Node copies output buffers before releasing the C result.
The older `queryAsync(sql, { format: 'arrow' }).toArrow()` remains an IPC route.

| User capability | Node API and behavior |
| --- | --- |
| Query an in-memory table or batch | `registerArrowSource` (or named `registerArrowTable`) supplies `arrowstream(name)` and a closable registration. Refresh a materialized source before a second scan. |
| Query a one-shot batch producer | `registerArrowReader` accepts sync/async `RecordBatch` iterables. Await registration, query asynchronously, and await `close()`; its native queue bounds batches not yet handed to core. |
| Read a complete typed result | `Session.queryArrow` / `queryArrowBind` return an `apache-arrow.Table` without an IPC conversion. |
| Pull typed result batches | `Session.queryArrowStream` / `queryArrowStreamBind` yield `RecordBatch` values. Cancel a partially consumed result. |
| Insert batches | `insertArrow` accepts a batch or materialized multi-batch table; `insertArrowReader` consumes an iterable. Both map destination columns by name and use engine-side `INSERT … SELECT`. `rowsSubmitted` counts input rows, not committed storage rows. |
| Reject incompatible paths | Direct APIs check the loaded engine (minimum `v26.11.0-rc.3`) and return `CHDB_ABI` for older builds. Unmappable types error unless an explicit binary option is chosen; IPC is not silently substituted. |

Registration names are process-wide. A registration's callbacks and borrowed
buffers stay alive while a scan or lazy result is active; closing during a scan
defers unregister and release until query cleanup. Borrowed Arrow sources cannot
coexist with a Session in another Node isolate. Input readers are one-shot and
their producers must cooperate with cancellation to release their own resources.
See the [Node Arrow guide](arrow.md) for examples and the supported type subset.
Nested input and multi-chunk dictionary layouts are explicitly rejected;
large-result memory stress and an AddressSanitizer/resident-memory leak budget
are not claimed as completed validation.

## Release evidence

- The [core v26.11.1 compatibility run](https://github.com/chdb-io/chdb-node/actions/runs/43935462757)
passed against the released engine on Linux x64/arm64 and macOS x64/arm64.
- The [Node v3.6.0 publication and verification run](https://github.com/chdb-io/chdb-node/actions/runs/43960573434)
completed with all 20 jobs successful on its failed-job rerun. The four native
packages and main package were already published in the first attempt;
several early clean installs had skipped optional native packages while npm
processed them. Fresh verification jobs then passed on all four platforms
with Node 18/20/22, plus Debian bookworm on both Linux architectures.
- The installed-package checks load `chdb` by package name, verify the exact
native subpackage, and run both the ordinary install tests and
[`smoke-arrow-e2e.mjs`](https://github.com/chdb-io/chdb-node/blob/v3.6.0/scripts/smoke-arrow-e2e.mjs).
The smoke covers an in-memory dictionary source, typed and bound output,
incremental batches, reader-backed insert, and a dictionary result option.
The [direct Arrow](https://github.com/chdb-io/chdb-node/blob/v3.6.0/test/v3/arrow-direct.test.ts)
and [input/reader/insert](https://github.com/chdb-io/chdb-node/tree/v3.6.0/test/v3/layer3)
tests cover empty schemas, multiple batches, release counts, errors,
cancellation, and deferred cleanup.

The earlier [rc.3 validation](https://github.com/chdb-io/chdb-node/blob/v3.6.0/docs/arrow-rc3-conformance.md)
is retained as a historical snapshot, not as a statement of the current
published package status.
8 changes: 4 additions & 4 deletions docs/arrow.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ npm install chdb apache-arrow
| What you need | Where to look |
| --- | --- |
| Choose and use the direct Arrow input, output, or insert API | [Choose a path](#choose-a-path) and the examples below |
| Run an installed-package end-to-end example | [`scripts/smoke-arrow-e2e.mjs`](../scripts/smoke-arrow-e2e.mjs) |
| Check supported types, ownership, and release validation | [rc.3 conformance notes](arrow-rc3-conformance.md), [direct API tests](../test/v3/arrow-direct.test.ts), and [reader tests](../test/v3/layer3/arrow-reader.test.ts) |
| Understand native worker/isolate and queue design | [Native Arrow design](design/native-arrow.md) |
| Run an installed-package end-to-end example | [`scripts/smoke-arrow-e2e.mjs`](https://github.com/chdb-io/chdb-node/blob/v3.6.0/scripts/smoke-arrow-e2e.mjs) |
| Check supported types, ownership, and release validation | [v3.6.0 release validation](arrow-release-validation.md), [direct API tests](https://github.com/chdb-io/chdb-node/blob/v3.6.0/test/v3/arrow-direct.test.ts), and [reader tests](https://github.com/chdb-io/chdb-node/blob/v3.6.0/test/v3/layer3/arrow-reader.test.ts) |
| Understand native worker/isolate and queue design | [Native Arrow design](https://github.com/chdb-io/chdb-node/blob/v3.6.0/docs/design/native-arrow.md) |
| Use the older in-memory Arrow IPC result route | [`format: 'arrow'` + `toArrow()`](../README.md); this is distinct from the direct typed API below |
| Use Arrow Database Connectivity (ADBC) for standard results or bulk ingest | Use a verified ADBC client/driver pairing separately. General `BindStream` binds query parameters; it does not make an async iterable an arbitrary SQL `FROM` source. |

Expand Down Expand Up @@ -173,7 +173,7 @@ the JS event loop.
LowCardinality UTF-8/Binary dictionaries. Options include
`stringAsString`, `lowCardinalityAsDictionary`, and `unsupportedAsBinary`;
without the latter, unmappable types fail rather than silently falling back
to IPC. See [validated type scope](arrow-rc3-conformance.md).
to IPC. See [current release validation](arrow-release-validation.md).
- Keep each `Session` open until its scans and readers finish. Cancel a
partially consumed output reader, close input registrations, and await
input reader cleanup. Error paths release the native handles.
Expand Down
17 changes: 9 additions & 8 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -147,21 +147,22 @@ Codegen / introspection:
during active queries/streams; finish or cancel a lazy stream to let it drain.
registerArrowSource(columns|batch|table, {session?}) same input path, with a
generated process-unique name returned on its closable handle.
registerArrowReader(asyncIterable<RecordBatch>, {session}) one-shot Arrow
reader as an engine SQL source, with a bounded native prefetch queue and
registerArrowReader(sync/async Iterable<RecordBatch>, {session}) one-shot
Arrow reader as an engine SQL source, with a bounded native prefetch queue and
async close(). The first batch establishes schema; empty sources need
source.schema. Reader errors propagate to SQL; no IPC or readAll().
insertArrow(table, columns|batch|table, {session}) bulk insert one Arrow batch
using an internal scoped registration and INSERT SELECT. Destination columns
are matched by name. rowsSubmitted is input size, not a storage write ledger.
insertArrow(table, columns|batch|table, {session}) bulk insert a batch or
materialized multi-batch Table using a scoped registration and INSERT SELECT.
Destination columns are matched by name. rowsSubmitted counts input rows,
not a storage write ledger.
insertArrowReader(table, reader, {session}) same field-name mapping and
INSERT SELECT over the bounded reader source; rowsSubmitted is not a
storage-side write ledger.
Session.queryArrow(sql) direct C Data output as an apache-arrow Table (not IPC).
Primitive signed/unsigned integers, Float32/64, Bool, Utf8, Binary,
Timestamp and Decimal128 are
supported; buffers are copied before C stream release, so the result
outlives the Session. stringAsString:false exports String as Binary.
Timestamp and Decimal128/256 are supported; buffers are copied before
C stream release, so the result outlives the Session.
stringAsString:false exports String as Binary.
Session.queryArrowStream(sql, options?) pull one typed apache-arrow
RecordBatch per core C Data fetch; cancel() on partial consumption.
schema is available after the first fetch, including zero-row results.
Expand Down
3 changes: 3 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,9 @@
"example.js",
"AGENTS.md",
"llms-full.txt",
"docs/arrow.md",
"docs/arrow-release-validation.md",
".agents/skills/chdb-arrow/SKILL.md",
"src/layer3/AGENTS.md",
"src/connection/AGENTS.md",
"integrations"
Expand Down
Loading