Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions .github/workflows/chdb-node-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,9 @@ jobs:
# installed `chdb` package — the Node analogue of chDB Python's "pip install
# the built wheel, then run the suite". Proves the published shape actually
# loads and runs, not just the in-repo source.
runs-on: ubuntu-latest
# Keep the portable-builder host stable; the rolling ubuntu-latest image
# (26.04) cancelled the AlmaLinux re-link step before packaging on #104.
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
Expand Down Expand Up @@ -246,7 +248,8 @@ jobs:
# subpackage from the registry — the cleanroom then tests the released
# binary instead of the one just built, which is how a non-relocatable
# binary (chdb-io/chdb-node#50) shipped past a green cleanroom.
export CHDB_LIB_VERSION="$(grep -E '^LIBCHDB_NPM_VERSION=' update_libchdb.sh | cut -d= -f2)"
CHDB_LIB_VERSION="$(grep -E '^LIBCHDB_NPM_VERSION=' update_libchdb.sh | cut -d= -f2)"
export CHDB_LIB_VERSION
npm run build:platform
MAIN_TGZ="$PWD/$(npm pack --silent)"
SUB_DIR="$PWD/npm/@chdb/lib-linux-x64-gnu"
Expand All @@ -257,14 +260,15 @@ jobs:
run: |
mkdir -p /tmp/cleanroom && cd /tmp/cleanroom
npm init -y >/dev/null
npm install --no-save "$SUB_TGZ" "$MAIN_TGZ"
npm install --no-save "$SUB_TGZ" "$MAIN_TGZ" apache-arrow@21.1.0
# The exact binary that will be dlopen'd must be relocatable. Assert it
# directly on the INSTALLED file — this also fails loudly (instead of a
# cryptic dlopen error) if npm resolved the subpackage from the registry
# rather than the local tgz.
INSTALLED_NODE="node_modules/@chdb/lib-linux-x64-gnu/chdb_node.node"
bash "$GITHUB_WORKSPACE/scripts/assert-relocatable.sh" "$INSTALLED_NODE"
cp "$GITHUB_WORKSPACE/scripts/smoke-e2e.mjs" ./smoke-e2e.mjs
cp "$GITHUB_WORKSPACE/scripts/smoke-arrow-e2e.mjs" ./smoke-arrow-e2e.mjs
# Relocation-faithful e2e: delete the build-machine libchdb.so so a
# baked absolute/build-dir library reference (chdb-io/chdb-node#50)
# can no longer be satisfied from the build tree. The installed package
Expand All @@ -273,9 +277,12 @@ jobs:
# passes on the build runner even when the binary is unshippable.
rm -f "$GITHUB_WORKSPACE/libchdb.so"
node smoke-e2e.mjs
node smoke-arrow-e2e.mjs
- name: The same install has to run on Debian bookworm
# The symbol assert reads the binary; this runs it. bookworm is older
# than every GitHub runner image and is the base of the node:22 /
# node:20 / node:lts images, so it is where users first meet a binary
# that demands a newer libstdc++ or glibc than they have.
run: docker run --rm -v /tmp/cleanroom:/w -w /w node:22-bookworm node smoke-e2e.mjs
run: |
docker run --rm -v /tmp/cleanroom:/w -w /w node:22-bookworm node smoke-e2e.mjs
docker run --rm -v /tmp/cleanroom:/w -w /w node:22-bookworm node smoke-arrow-e2e.mjs
9 changes: 7 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,15 +144,20 @@ somewhere later.
| Server-side parameter binding (`{name:Type}`) | ✅ |
| Insert (object / positional rows) | ✅ |
| Streaming results (`AsyncIterable`) | ✅ |
| Arrow **output** (`format: 'arrow'` + `toArrow()`) | ✅ |
| Arrow IPC **output** (`format: 'arrow'` + `toArrow()`) | ✅ |
| AbortSignal / timeout | ✅ (single-shot is honest: rejects early; native runs to completion) |
| Arrow **scan** (`registerArrowTable`, Arrow input) | ⏳ follow-up |
| Direct Arrow **scan**, reader, typed results, and bulk insert | ✅ in PR #104 (rc.3 native packages pending) |
| Arrow zero-copy (M2, `{ zeroCopy: true }`) | ⏳ follow-up |
| chDB ↔ `@clickhouse/client` integration (`chdb/connection`, **experimental**) | ✅ |
| Durable V1 control plane (`chdb/durable`, **experimental**) | ✅ pure TS, engine injected |
| Default durable engine over the addon (`chdb/durable/node`) | ✅ |
| Remote object storage for durable (`chdb/durable/s3`) | ✅ verified on AWS S3 and MinIO; R2 untested |

See the [Arrow guide](docs/arrow.md) for examples, lifecycle rules, and the
difference between direct C Data/C Stream and the existing Arrow IPC path.
The direct APIs require chdb-core v26.11.0-rc.3; they are not in the currently
published npm platform packages.

### chDB ↔ `@clickhouse/client` integration (`chdb/connection`, experimental)

> **Status**: this integration uses the experimental
Expand Down
12 changes: 12 additions & 0 deletions arrow.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
// ESM bridge for the optional typed Arrow subpath. Its runtime only delegates
// to Session; apache-arrow is loaded when the result is converted.
import { createRequire } from 'node:module'
const mod = createRequire(import.meta.url)('./dist/arrow-entry.js')
export const queryArrow = mod.queryArrow
export const queryArrowBind = mod.queryArrowBind
export const queryArrowStream = mod.queryArrowStream
export const queryArrowStreamBind = mod.queryArrowStreamBind
export const registerArrowSource = mod.registerArrowSource
export const registerArrowReader = mod.registerArrowReader
export const insertArrow = mod.insertArrow
export const insertArrowReader = mod.insertArrowReader
47 changes: 47 additions & 0 deletions docs/arrow-rc3-conformance.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Native Arrow A1–A6 against chdb-core v26.11.0-rc.3

The development baseline is the `macos-arm64-libchdb.tar.gz` asset from
`v26.11.0-rc.3` (SHA-256 `7b0ea049efcf75a851e744c7d74325d7a9b143227f9b72097e5df252564e3d58`).
The release tag is #323's merge commit `574f212fb5ee`. The manifest and
lockfile target `@chdb/lib-*` version `26.11.0-rc.3.1`, but those npm packages
have **not yet been published**. npm omitted their package metadata from the
lockfile; regenerate it after publication to capture `resolved` and `integrity`.
The local macOS arm64 tests used the official rc.3
tarball with the previous ignored `libchdb.so` backed up under
`/tmp/chdb-node-rc3-verify/`. A cleanroom install from locally packed main and
arm64 subpackage tarballs passed `smoke-arrow-e2e.mjs`. The other three
platforms and registry-installed packages remain CI/release gates.

The shared contract is `chdb-core/docs/NATIVE_ARROW_BINDING_CONTRACT.md`. In
this binding, **direct** means C Data/C Stream without Arrow IPC. Input and
output may copy buffers; no end-to-end zero-copy claim is made.
For application examples see the [Arrow guide](arrow.md); for the bounded
queue and release sequence see the [Node Arrow design](design/native-arrow.md).
The optional `chdb/arrow` subpath supplies TypeScript `Table`/`RecordBatch`
return types without making the base `chdb` entry require `apache-arrow`.

| Capability | Implemented now | Further validation / documented scope |
| --- | --- | --- |
| A1 SQL source | `registerArrowTable` accepts supported column descriptors, `RecordBatch`, and materialized multi-batch `Table`; C Stream callbacks use pinned JS buffers. `registerArrowSource` generates opaque process-unique names. Primitive, Timestamp, Decimal128/256, sliced buffers/bitmaps, and one-chunk Dictionary<Int32/UInt32, Utf8/Binary> input are tested. Borrowed sources cannot coexist with a Session in another Node isolate. | Nested types and multi-chunk dictionary input explicitly reject; more platform/isolate stress remains. |
| A2 reader source | `registerArrowReader` accepts sync/async RecordBatch iterables. The JS producer awaits capacity in a two-batch native queue; worker-thread `get_next` never calls JS. Empty schema, producer error, schema mismatch, cancellation, one-release-per-delivered-batch, one-shot behavior and worker-isolate teardown are tested. | ASan/RSS stress and platform matrix. Core may retain batches after pulling them (e.g. sorting); queue capacity counts only batches not yet handed to core. `close()` cancels native promptly even if a user producer never resolves `next()`; that producer must cooperate to release its own resources. |
| A3 typed result | `Session.queryArrow` calls `chdb_query_arrow_n`, copies C buffers before release, and constructs an `apache-arrow.Table`; `queryArrowBind` materializes the bound A4 stream. Empty schema and SQL errors tested. | Nested types and large-result memory stress. |
| A4 typed batches | `Session.queryArrowStream` pulls C Data per batch; `queryArrowStreamBind` uses `chdb_stream_query_arrow_with_params_n` with server-side binding. Early-break, in-flight cancel and bound values are tested. | Nested types, repeated cancellation and memory stress. |
| A5 bulk insert | `insertArrow` handles materialized batches; `insertArrowReader` uses A2's bounded source. Both map destination fields by name and use engine-side `INSERT … SELECT`. | Storage-side write ledger (current `rowsSubmitted` is only the input count), settings/mapping matrix and cancellation under write load. |
| A6 ABI/types | New direct Arrow paths verify the loaded `EngineVersion()` is at least `v26.11.0-rc.3` and reject older builds with `CHDB_ABI`. Basic numeric, Bool, Utf8, Binary, Timestamp, Decimal128/256 and LowCardinality Utf8/Binary dictionary output are mapped. `unsupportedAsBinary` is tested for an otherwise unsupported AggregateFunction; IPC `toArrow()` remains unchanged. | Four-platform build/runtime CI; publish all four rc.3 npm packages and regenerate the lockfile afterwards. Additional nested/type-option cases are outside this supported subset. |

## Core ownership regression and release follow-up

rc.2 predates `chdb-core#323`, which is included in rc.3. Core issues
[`#321`](https://github.com/chdb-io/chdb-core/issues/321) and
[`#322`](https://github.com/chdb-io/chdb-core/issues/322) already record its
`get_schema` release and failed array-registration wrapper leaks. Do not open
duplicate issues. The previously conditional registration-schema release test
now runs unconditionally on rc.3. An ASan/RSS *failed-registration leak budget*
is still a future stress gate, not a test claimed here. Node's own close/eviction
path unregisters from core before dropping pinned buffers.

If another **new** rc.3 core defect appears: keep a small reproducer, file a
core issue before designing around its proposed fix, link the issue from a
clearly skipped/conditional test, and describe the required follow-up. Filing
an issue is a remote write and requires separate per-action confirmation; other
local implementation and unaffected tests can continue meanwhile.
151 changes: 151 additions & 0 deletions docs/arrow.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# Arrow with chdb in Node.js

The direct Arrow APIs let you query in-memory Arrow data, read typed results,
and insert batches without Arrow IPC. They are implemented in [PR #104](https://github.com/chdb-io/chdb-node/pull/104)
against chdb-core v26.11.0-rc.3; the matching npm platform packages are not
published yet. Once released, install both `chdb` and its optional peer:

```sh
npm install chdb apache-arrow
```

## Choose a path

| Task | API | Result |
| --- | --- | --- |
| Query an Arrow `Table` or `RecordBatch` | `registerArrowSource` + `arrowstream(name)` | SQL result in your chosen format |
| Feed batches as they arrive | `registerArrowReader` + `arrowstream(name)` | One-shot SQL source |
| Read a small typed result | `session.queryArrow(sql)` | `apache-arrow` `Table` |
| Read a large typed result | `session.queryArrowStream(sql)` | Async iterable of `RecordBatch` |
| Insert a table or a batch | `insertArrow(table, input, { session })` | Insert summary |
| Insert a batch iterable | `insertArrowReader(table, source, { session })` | Insert summary |

The existing `queryAsync(sql, { format: 'arrow' }).toArrow()` still uses Arrow
IPC. The direct APIs use Arrow C Data/C Stream instead; **direct is not an
end-to-end zero-copy guarantee**. chdb may copy input while converting columns,
and Node copies output buffers before releasing the native result.

## Query an in-memory Arrow table

```js
import { Session, registerArrowSource } from 'chdb'
import { Table, Int32, vectorFromArray } from 'apache-arrow'

const session = new Session()
const input = new Table({ id: vectorFromArray([1, 2, 3], new Int32()) })
const source = registerArrowSource(input, { session }) // unique, opaque name
try {
const sql = `SELECT sum(id) FROM arrowstream('${source.name}')`
console.log((await session.queryAsync(sql, { format: 'CSV' })).text()) // 6

// A materialized source is consumed by a scan. Refresh only after the
// previous query has completed, before using it in another query.
source.refresh()
console.log((await session.queryAsync(sql, { format: 'CSV' })).text()) // 6
} finally {
source.close()
session.close()
}
```

`registerArrowTable(name, input, { session })` is available when you need to
choose the name. Names are process-wide; prefer generated names unless you have
a reason to coordinate them. Close the registration after the last scan. If a
query or lazy stream is active, unregister and buffer release are deferred
until that scan finishes or is cancelled.

## Insert Arrow data

```js
import { Session, insertArrow, insertArrowReader } from 'chdb'
import { Table, Int32, vectorFromArray } from 'apache-arrow'

const session = new Session()
try {
await session.queryAsync('CREATE TABLE events (id Int32) ENGINE=Memory')
const input = new Table({ id: vectorFromArray([1, 2], new Int32()) })
await insertArrow('events', input, { session })

async function* batches() {
yield new Table({ id: vectorFromArray([3], new Int32()) }).batches[0]
yield new Table({ id: vectorFromArray([4], new Int32()) }).batches[0]
}
const summary = await insertArrowReader('events', batches(), { session })
console.log(summary.rowsSubmitted) // 2; not a storage-side commit ledger
} finally {
session.close()
}
```

Destination fields are matched by name, not table position. Both insert
functions unregister their temporary source on success or error. For a custom
SQL transform over an incoming reader, register it with
`await registerArrowReader(batches(), { session })`, query
`arrowstream(handle.name)` once, then `await handle.close()` in `finally`.
The native queue holds at most two batches not yet handed to
core; producing another waits for capacity. Core can retain delivered batches,
so this is not a bound on the entire query's memory. A cancelled producer
should cooperate with iterator `return()` to release its own resources.
Use `queryAsync` or another async query API while a JS Arrow reader is
registered: synchronous `query`/`queryBind` on its connection are rejected
because blocking the event loop would prevent the producer from refilling
the queue.

## Read typed results

```js
import { Session } from 'chdb'

const session = new Session()
try {
const small = await session.queryArrow('SELECT toInt32(number) AS id FROM numbers(3)')
console.log(small.toArray().map(row => row.id)) // [0, 1, 2]

const reader = session.queryArrowStream('SELECT toInt32(number) AS id FROM numbers(1000)')
try {
for await (const batch of reader) {
console.log(batch.numRows)
// Stop early if needed.
}
} finally {
reader.cancel()
}
} finally {
session.close()
}
```

Use `queryArrowBind` and `queryArrowStreamBind` for server-side
`{name:Type}` parameters, for example
`await session.queryArrowBind('SELECT {id:Int32} AS id', { id: 7 })`.
For TypeScript return types without making the base `chdb` import require
`apache-arrow`, use the helpers from `chdb/arrow`.
Only one iterator may consume a given Arrow result reader at a time; cancel
it before closing its Session if you stop early.
Await earlier operations on the same Session before opening a direct Arrow
stream: its synchronous startup rejects a busy connection instead of blocking
the JS event loop.

## Limits and errors

- The direct path requires a loaded chdb-core >= v26.11.0-rc.3; an older
binary raises `CHDB_ABI`. Installing `apache-arrow` alone does not upgrade
the native platform package.
- Supported input includes primitive scalars, Boolean, UTF-8/Binary,
Timestamp, Decimal128/256, slices, null bitmaps, and a single-chunk
dictionary of UTF-8/Binary values with Int32/UInt32 indices. Nested input
and multi-chunk dictionaries are rejected rather than silently converted.
- Supported output includes basic scalars, Timestamp, Decimal128/256, and
LowCardinality UTF-8/Binary dictionaries. Options include
`stringAsString`, `lowCardinalityAsDictionary`, and `unsupportedAsBinary`;
without the latter, unmappable types fail rather than silently falling back
to IPC. See [validated type scope](arrow-rc3-conformance.md).
- Keep each `Session` open until its scans and readers finish. Cancel a
partially consumed output reader, close input registrations, and await
input reader cleanup. Error paths release the native handles.
- Arrow source registrations are isolate-local. While one isolate owns a
registration, another Node worker cannot open a Session; close the source
before transferring work between isolates. The engine's registry names are
process-wide, but a borrowed JS buffer cannot safely outlive its isolate.

For the thread and ownership model, see the [Node Arrow design](design/native-arrow.md).
1 change: 1 addition & 0 deletions docs/design/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ Current PR #43 Layer 1 capabilities include:
- CJS and ESM entry points.

See [Layer 1 Native Binding Design and Implementation](./layer1-native-binding.md) for the full reviewer guide.
For the newer direct Arrow input/output path, see [Node native Arrow: threads and ownership](./native-arrow.md).

## Layer 2 - ClickHouse Client Compatibility

Expand Down
8 changes: 3 additions & 5 deletions docs/design/durable-control-plane.md
Original file line number Diff line number Diff line change
Expand Up @@ -724,11 +724,9 @@ this package:

1. Publish `@chdb/lib-<platform>` packages built from a tree that binds the
durable ABI. `optionalDependencies` and `update_libchdb.sh` already name
`26.11.0-rc.2.1`, and the release workflow derives the subpackage version
from the latter — but nothing has been tagged since, so that version is not
on npm at all. The newest published is `26.7.0-stable.1` (main `chdb@3.3.0`
pins it), which carries an addon from before this work and exports none of
the four entry points.
`26.11.0-rc.3.1`, and the release workflow derives the subpackage version
from the latter — but this version has not yet been published on npm. The
currently published rc.2.1 packages do not contain the rc.3 ownership fix.

Two different failures follow, depending on what is installed. A fresh
install resolves no subpackage — an unsatisfiable *optional* dependency is
Expand Down
Loading
Loading