diff --git a/.agents/skills/chdb-arrow/SKILL.md b/.agents/skills/chdb-arrow/SKILL.md index aa584cc..7e7b36c 100644 --- a/.agents/skills/chdb-arrow/SKILL.md +++ b/.agents/skills/chdb-arrow/SKILL.md @@ -9,4 +9,4 @@ Arrow Inter-Process Communication (IPC) is a serialized byte format; the direct - For an `apache-arrow.Table` / `RecordBatch` as a temporary SQL source, use `registerArrowSource(input, { session })` and query `arrowstream('${source.name}')`; close the source in `finally`. For a one-shot sync/async batch iterable, `await registerArrowReader(iterable, { session })` and later `await handle.close()`. Use async queries while a JS producer refills the native queue. - Typed results: `await session.queryArrow(sql)` returns a whole `Table`; `session.queryArrowStream(sql)` yields batches via `for await` and needs `reader.cancel()` if stopped early. Bulk insert: `await insertArrow(table, input, { session })` or `await insertArrowReader(table, iterable, { session })`. -- `queryAsync(sql, { format: 'arrow' }).toArrow()` remains an Arrow IPC compatibility route, not direct C Data. Prefer a verified ADBC client only if the application already uses ADBC for standard results or bulk ingest; general ADBC `BindStream` does not create a temporary SQL source. Install the optional `apache-arrow` peer. Start with `docs/arrow.md` for this binding's documentation map, examples, types, and isolate constraints. +- `queryAsync(sql, { format: 'arrow' }).toArrow()` remains an Arrow IPC compatibility route, not direct C Data. Prefer a verified ADBC client only if the application already uses ADBC for standard results or bulk ingest; general ADBC `BindStream` does not create a temporary SQL source. Install the optional `apache-arrow` peer. Start with the [Node Arrow guide](../../../docs/arrow.md) for examples and supported types, then its release-validation link for the tested core and platform matrix. diff --git a/AGENTS.md b/AGENTS.md index caa5022..672f526 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,6 +20,7 @@ unsupported (use WSL2). Run SQL you have, or anything ClickHouse-specific? -> raw SQL (query / sql``) Build queries programmatically, type-safe + bound? -> fluent (selectFrom / insertInto) [src/layer3/AGENTS.md] Read/JOIN external sources (S3/Postgres/…)? -> connect({ url }) [src/layer3/AGENTS.md] +Query/insert in-memory Arrow or read typed batches? -> direct Arrow [docs/arrow.md] Migrating from @clickhouse/client (drop-in)? -> chdb/connection [src/connection/AGENTS.md] Need the database to survive losing the machine? -> chdb/durable (experimental) [docs/design/durable-control-plane.md] ``` @@ -68,4 +69,5 @@ const rows = await session('./db').selectFrom('events').selectAll().where('ts', - **No SQL injection** with `queryBind`, the `sql` tag, or the fluent builder (values bound server-side). Detail: fluent + federation → `src/layer3/AGENTS.md`; `@clickhouse/client` compat → -`src/connection/AGENTS.md`; full API reference → `llms-full.txt`. +`src/connection/AGENTS.md`; direct Arrow → `docs/arrow.md` and +`.agents/skills/chdb-arrow/SKILL.md`; full API reference → `llms-full.txt`. diff --git a/docs/arrow-rc3-conformance.md b/docs/arrow-rc3-conformance.md index 7a05a50..5cddbbd 100644 --- a/docs/arrow-rc3-conformance.md +++ b/docs/arrow-rc3-conformance.md @@ -1,17 +1,19 @@ -# Native Arrow validation against chdb-core v26.11.0-rc.3 +# Historical Arrow validation against chdb-core v26.11.0-rc.3 + +This is a pre-release snapshot, not the current status of npm packages or +platform validation. For the released `chdb@3.6.0` and `v26.11.1` engine, see +the [current Arrow release validation](arrow-release-validation.md). The development baseline is the `macos-arm64-libchdb.tar.gz` asset from `v26.11.0-rc.3` (SHA-256 `7b0ea049efcf75a851e744c7d74325d7a9b143227f9b72097e5df252564e3d58`). -The manifest and lockfile target `@chdb/lib-*` version `26.11.0-rc.3.1`, but those npm packages -have **not yet been published**. npm omitted their package metadata from the -lockfile; regenerate it after publication to capture `resolved` and `integrity`. -The local macOS arm64 tests used the official rc.3 -tarball with the previous ignored `libchdb.so` backed up under -`/tmp/chdb-node-rc3-verify/`. A cleanroom install from locally packed main and +At the time of this snapshot, the manifest targeted `@chdb/lib-*` +`26.11.0-rc.3.1` and the packages had not yet been published. The local +macOS arm64 tests used the official rc.3 tarball with the previous ignored +`libchdb.so` backed up under `/tmp/chdb-node-rc3-verify/`. A cleanroom install from locally packed main and arm64 subpackage tarballs passed `smoke-arrow-e2e.mjs`. The other three -platforms and registry-installed packages remain CI/release gates. +platforms and registry-installed packages were then outstanding release gates. -The shared contract is the [core native Arrow binding contract](https://github.com/chdb-io/chdb-core/blob/main/docs/NATIVE_ARROW_BINDING_CONTRACT.md). In +The shared contract is the [rc.3 core native Arrow binding contract](https://github.com/chdb-io/chdb-core/blob/v26.11.0-rc.3/docs/NATIVE_ARROW_BINDING_CONTRACT.md). In this binding, **direct** means Arrow's in-memory C Data/C Stream interfaces without Inter-Process Communication (IPC) serialization. Input and output may copy buffers; no end-to-end zero-copy claim is made. An Arrow schema names @@ -23,7 +25,7 @@ queue and release sequence see the [Node Arrow design](design/native-arrow.md). The optional `chdb/arrow` subpath supplies TypeScript `Table`/`RecordBatch` return types without making the base `chdb` entry require `apache-arrow`. -| Capability | Implemented now | Further validation / documented scope | +| Capability | Implemented at the rc.3 snapshot | Outstanding at the rc.3 snapshot / documented scope | | --- | --- | --- | | In-memory SQL source | `registerArrowTable` accepts supported column descriptors, `RecordBatch`, and materialized multi-batch `Table`; C Stream callbacks use pinned JS buffers. `registerArrowSource` generates opaque process-unique names. Primitive, Timestamp, Decimal128/256, sliced buffers/bitmaps, and one-chunk Dictionary input are tested. Borrowed sources cannot coexist with a Session in another Node isolate. | Nested types and multi-chunk dictionary input explicitly reject; more platform/isolate stress remains. | | Reader as SQL source | `registerArrowReader` accepts sync/async RecordBatch iterables. The JS producer awaits capacity in a two-batch native queue; worker-thread `get_next` never calls JS. Empty schema, producer error, schema mismatch, cancellation, one-release-per-delivered-batch, one-shot behavior and worker-isolate teardown are tested. | ASan/RSS stress and platform matrix. Core may retain batches after pulling them (e.g. sorting); queue capacity counts only batches not yet handed to core. `close()` cancels native promptly even if a user producer never resolves `next()`; that producer must cooperate to release its own resources. | @@ -39,9 +41,3 @@ failed. The rc.3 engine fixes both ownership paths. The registration-schema rele now runs unconditionally on rc.3. An ASan/RSS *failed-registration leak budget* is still a future stress gate, not a test claimed here. Node's own close/eviction path unregisters from core before dropping pinned buffers. - -If another **new** rc.3 core defect appears: keep a small reproducer, file a -core issue before designing around its proposed fix, link the issue from a -clearly skipped/conditional test, and describe the required follow-up. Filing -an issue is a remote write and requires separate per-action confirmation; other -local implementation and unaffected tests can continue meanwhile. diff --git a/docs/arrow-release-validation.md b/docs/arrow-release-validation.md new file mode 100644 index 0000000..be8ce6b --- /dev/null +++ b/docs/arrow-release-validation.md @@ -0,0 +1,58 @@ +# Direct Arrow release validation for chdb 3.6.0 + +`chdb@3.6.0` pins chdb-core `v26.11.1` through the four platform packages +`@chdb/lib-{darwin-arm64,darwin-x64,linux-arm64-gnu,linux-x64-gnu}@26.11.1-stable.1`. +Install the optional `apache-arrow` peer to use typed Arrow objects: + +```sh +npm install chdb@3.6.0 apache-arrow +``` + +The [v26.11.1 native Arrow binding contract](https://github.com/chdb-io/chdb-core/blob/v26.11.1/docs/NATIVE_ARROW_BINDING_CONTRACT.md) +defines direct C Data/C Stream transfer, ownership, and compatibility. "Direct" +excludes Arrow IPC serialization at the binding boundary; it does **not** promise +end-to-end zero-copy. Node copies output buffers before releasing the C result. +The older `queryAsync(sql, { format: 'arrow' }).toArrow()` remains an IPC route. + +| User capability | Node API and behavior | +| --- | --- | +| Query an in-memory table or batch | `registerArrowSource` (or named `registerArrowTable`) supplies `arrowstream(name)` and a closable registration. Refresh a materialized source before a second scan. | +| Query a one-shot batch producer | `registerArrowReader` accepts sync/async `RecordBatch` iterables. Await registration, query asynchronously, and await `close()`; its native queue bounds batches not yet handed to core. | +| Read a complete typed result | `Session.queryArrow` / `queryArrowBind` return an `apache-arrow.Table` without an IPC conversion. | +| Pull typed result batches | `Session.queryArrowStream` / `queryArrowStreamBind` yield `RecordBatch` values. Cancel a partially consumed result. | +| Insert batches | `insertArrow` accepts a batch or materialized multi-batch table; `insertArrowReader` consumes an iterable. Both map destination columns by name and use engine-side `INSERT … SELECT`. `rowsSubmitted` counts input rows, not committed storage rows. | +| Reject incompatible paths | Direct APIs check the loaded engine (minimum `v26.11.0-rc.3`) and return `CHDB_ABI` for older builds. Unmappable types error unless an explicit binary option is chosen; IPC is not silently substituted. | + +Registration names are process-wide. A registration's callbacks and borrowed +buffers stay alive while a scan or lazy result is active; closing during a scan +defers unregister and release until query cleanup. Borrowed Arrow sources cannot +coexist with a Session in another Node isolate. Input readers are one-shot and +their producers must cooperate with cancellation to release their own resources. +See the [Node Arrow guide](arrow.md) for examples and the supported type subset. +Nested input and multi-chunk dictionary layouts are explicitly rejected; +large-result memory stress and an AddressSanitizer/resident-memory leak budget +are not claimed as completed validation. + +## Release evidence + +- The [core v26.11.1 compatibility run](https://github.com/chdb-io/chdb-node/actions/runs/43935462757) + passed against the released engine on Linux x64/arm64 and macOS x64/arm64. +- The [Node v3.6.0 publication and verification run](https://github.com/chdb-io/chdb-node/actions/runs/43960573434) + completed with all 20 jobs successful on its failed-job rerun. The four native + packages and main package were already published in the first attempt; + several early clean installs had skipped optional native packages while npm + processed them. Fresh verification jobs then passed on all four platforms + with Node 18/20/22, plus Debian bookworm on both Linux architectures. +- The installed-package checks load `chdb` by package name, verify the exact + native subpackage, and run both the ordinary install tests and + [`smoke-arrow-e2e.mjs`](https://github.com/chdb-io/chdb-node/blob/v3.6.0/scripts/smoke-arrow-e2e.mjs). + The smoke covers an in-memory dictionary source, typed and bound output, + incremental batches, reader-backed insert, and a dictionary result option. + The [direct Arrow](https://github.com/chdb-io/chdb-node/blob/v3.6.0/test/v3/arrow-direct.test.ts) + and [input/reader/insert](https://github.com/chdb-io/chdb-node/tree/v3.6.0/test/v3/layer3) + tests cover empty schemas, multiple batches, release counts, errors, + cancellation, and deferred cleanup. + +The earlier [rc.3 validation](https://github.com/chdb-io/chdb-node/blob/v3.6.0/docs/arrow-rc3-conformance.md) +is retained as a historical snapshot, not as a statement of the current +published package status. diff --git a/docs/arrow.md b/docs/arrow.md index fd57f2a..c59ade0 100644 --- a/docs/arrow.md +++ b/docs/arrow.md @@ -17,9 +17,9 @@ npm install chdb apache-arrow | What you need | Where to look | | --- | --- | | Choose and use the direct Arrow input, output, or insert API | [Choose a path](#choose-a-path) and the examples below | -| Run an installed-package end-to-end example | [`scripts/smoke-arrow-e2e.mjs`](../scripts/smoke-arrow-e2e.mjs) | -| Check supported types, ownership, and release validation | [rc.3 conformance notes](arrow-rc3-conformance.md), [direct API tests](../test/v3/arrow-direct.test.ts), and [reader tests](../test/v3/layer3/arrow-reader.test.ts) | -| Understand native worker/isolate and queue design | [Native Arrow design](design/native-arrow.md) | +| Run an installed-package end-to-end example | [`scripts/smoke-arrow-e2e.mjs`](https://github.com/chdb-io/chdb-node/blob/v3.6.0/scripts/smoke-arrow-e2e.mjs) | +| Check supported types, ownership, and release validation | [v3.6.0 release validation](arrow-release-validation.md), [direct API tests](https://github.com/chdb-io/chdb-node/blob/v3.6.0/test/v3/arrow-direct.test.ts), and [reader tests](https://github.com/chdb-io/chdb-node/blob/v3.6.0/test/v3/layer3/arrow-reader.test.ts) | +| Understand native worker/isolate and queue design | [Native Arrow design](https://github.com/chdb-io/chdb-node/blob/v3.6.0/docs/design/native-arrow.md) | | Use the older in-memory Arrow IPC result route | [`format: 'arrow'` + `toArrow()`](../README.md); this is distinct from the direct typed API below | | Use Arrow Database Connectivity (ADBC) for standard results or bulk ingest | Use a verified ADBC client/driver pairing separately. General `BindStream` binds query parameters; it does not make an async iterable an arbitrary SQL `FROM` source. | @@ -173,7 +173,7 @@ the JS event loop. LowCardinality UTF-8/Binary dictionaries. Options include `stringAsString`, `lowCardinalityAsDictionary`, and `unsupportedAsBinary`; without the latter, unmappable types fail rather than silently falling back - to IPC. See [validated type scope](arrow-rc3-conformance.md). + to IPC. See [current release validation](arrow-release-validation.md). - Keep each `Session` open until its scans and readers finish. Cancel a partially consumed output reader, close input registrations, and await input reader cleanup. Error paths release the native handles. diff --git a/llms-full.txt b/llms-full.txt index 3650a6d..e42e61f 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -147,21 +147,22 @@ Codegen / introspection: during active queries/streams; finish or cancel a lazy stream to let it drain. registerArrowSource(columns|batch|table, {session?}) same input path, with a generated process-unique name returned on its closable handle. - registerArrowReader(asyncIterable, {session}) one-shot Arrow - reader as an engine SQL source, with a bounded native prefetch queue and + registerArrowReader(sync/async Iterable, {session}) one-shot + Arrow reader as an engine SQL source, with a bounded native prefetch queue and async close(). The first batch establishes schema; empty sources need source.schema. Reader errors propagate to SQL; no IPC or readAll(). - insertArrow(table, columns|batch|table, {session}) bulk insert one Arrow batch - using an internal scoped registration and INSERT SELECT. Destination columns - are matched by name. rowsSubmitted is input size, not a storage write ledger. + insertArrow(table, columns|batch|table, {session}) bulk insert a batch or + materialized multi-batch Table using a scoped registration and INSERT SELECT. + Destination columns are matched by name. rowsSubmitted counts input rows, + not a storage write ledger. insertArrowReader(table, reader, {session}) same field-name mapping and INSERT SELECT over the bounded reader source; rowsSubmitted is not a storage-side write ledger. Session.queryArrow(sql) direct C Data output as an apache-arrow Table (not IPC). Primitive signed/unsigned integers, Float32/64, Bool, Utf8, Binary, - Timestamp and Decimal128 are - supported; buffers are copied before C stream release, so the result - outlives the Session. stringAsString:false exports String as Binary. + Timestamp and Decimal128/256 are supported; buffers are copied before + C stream release, so the result outlives the Session. + stringAsString:false exports String as Binary. Session.queryArrowStream(sql, options?) pull one typed apache-arrow RecordBatch per core C Data fetch; cancel() on partial consumption. schema is available after the first fetch, including zero-row results. diff --git a/package.json b/package.json index 7c7ce47..f973be8 100644 --- a/package.json +++ b/package.json @@ -103,6 +103,9 @@ "example.js", "AGENTS.md", "llms-full.txt", + "docs/arrow.md", + "docs/arrow-release-validation.md", + ".agents/skills/chdb-arrow/SKILL.md", "src/layer3/AGENTS.md", "src/connection/AGENTS.md", "integrations"