Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions .github/workflows/connection.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
name: connection

# Pluggable Connection CI for chdb-node.
#
# Two jobs:
# - contract: runs the in-repo Connection contract suite
# (test/v3/connection/contract.test.ts) against ChdbConnection.
# Fast, fully self-contained, no network.
# - parity: clones @clickhouse/client at the configured ref (see
# tests/clickhouse-js/skip_list.json#syncedAgainst), links
# this checkout into it, and runs the upstream integration
# suite with ChdbConnection injected. tests/clickhouse-js/
# skip_list.json filters out the ~23 cases that are real
# chdb capability gaps (HTTP transport, auth/RBAC, server
# runtime). A new failure here is either a real new chdb
# gap (add to skip_list) or a bug in ChdbConnection.

on:
pull_request:
branches: ["main"]
paths-ignore: ["**/*.md"]
push:
branches: ["main"]
paths-ignore: ["**/*.md"]
workflow_dispatch: {}

concurrency:
group: connection-${{ github.ref }}
cancel-in-progress: true

jobs:
contract:
runs-on: ubuntu-latest
env:
TZ: UTC
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22.x
- name: Install patchelf
run: sudo apt-get update && sudo apt-get install -y patchelf
- run: npm install --ignore-scripts
- run: npm run libchdb
- run: npm run build
- run: npm run test:v3 -- test/v3/connection/

parity:
runs-on: ubuntu-latest
env:
TZ: UTC
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22.x
- name: Install patchelf
run: sudo apt-get update && sudo apt-get install -y patchelf
- run: npm install --ignore-scripts
- run: npm run libchdb
- run: npm run build
- name: Cross-suite parity (clone @clickhouse/client, inject ChdbConnection, apply skip_list)
run: node tests/clickhouse-js/runner.mjs
40 changes: 40 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,51 @@ Errors are typed (`ChdbSyntaxError`, `ChdbQueryError`, `ChdbConnectionError`,
| AbortSignal / timeout | ✅ (single-shot is honest: rejects early; native runs to completion) |
| Arrow **scan** (`registerArrowTable`, Arrow input) | ⏳ follow-up |
| Arrow zero-copy (M2, `{ zeroCopy: true }`) | ⏳ follow-up |
| chDB ↔ `@clickhouse/client` integration (`chdb/connection`, **experimental**) | ✅ |

### chDB ↔ `@clickhouse/client` integration (`chdb/connection`, experimental)

> **Status**: this integration uses the experimental
> `createClient({ connection })` hook in `@clickhouse/client`
> ([clickhouse-js#879](https://github.com/ClickHouse/clickhouse-js/pull/879)
> merged; framing follow-up
> [#880](https://github.com/ClickHouse/clickhouse-js/pull/880) merged).
> Upstream considers this a deliberately narrow chDB-only hook — not a
> public plugin system — and the shape may change. We'll keep
> `chdb/connection` working against whatever the upstream hook evolves
> into.

For users coming from `@clickhouse/client`, chdb-node ships a
**Connection** implementation under the `chdb/connection` subpath that
plugs into `@clickhouse/client`'s `createClient({ connection })`
injection point (tracking issue:
[clickhouse-js#865](https://github.com/ClickHouse/clickhouse-js/issues/865)).

```ts
import { createChdbConnection } from 'chdb/connection'

const conn = createChdbConnection({ path: ':memory:' })
const r = await conn.query({ query: 'SELECT * FROM numbers(5)', format: 'JSONEachRow' })
let body = ''
for await (const chunk of r.stream) body += Buffer.from(chunk).toString('utf8')
console.log(JSON.parse(`[${body.trim().split('\n').join(',')}]`))
await conn.close()

// chDB-specific escape hatches (raw ChdbResult, raw insert, session info)
conn.chdb.queryAsync('SELECT 1', { format: 'arrow' }) // bytes/text/json/toArrow
conn.chdb.session.path // bound on-disk path
```

See [docs/design/pluggable-connection.md](docs/design/pluggable-connection.md)
for the full design, the `Connection` interface, the `.chdb` extension
namespace, the `tests/clickhouse-js/skip_list.json` parity blacklist,
and the sync policy with `@clickhouse/client`.

### Design docs

- [Layered API design](docs/design/architecture.md): the Layer 1 / Layer 2 / Layer 3 architecture, package shape, and intended user-facing surfaces.
- [Layer 1 native binding reviewer guide](docs/design/layer1-native-binding.md): the PR #43 design and implementation map, organized by commit and review feedback.
- [chDB ↔ `@clickhouse/client` integration (experimental)](docs/design/pluggable-connection.md): the `chdb/connection` surface, the `Connection` interface chdb-node implements, the `.chdb` extension namespace, and the parity-test sync policy.

### Runtimes

Expand Down
225 changes: 225 additions & 0 deletions docs/design/pluggable-connection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
# chDB ↔ `@clickhouse/client` integration (experimental)

> **Status**: implemented in this branch (`feat/layer2-pluggable-backend`).
> Upstream tracking issue:
> [ClickHouse/clickhouse-js#865](https://github.com/ClickHouse/clickhouse-js/issues/865).
> Upstream injection point:
> [ClickHouse/clickhouse-js#879](https://github.com/ClickHouse/clickhouse-js/pull/879)
> (merged 2026-06-23 as `c2e28b9`); upstream framing:
> [ClickHouse/clickhouse-js#880](https://github.com/ClickHouse/clickhouse-js/pull/880)
> (merged 2026-06-23 as `274844f`).
> Parallel Python proposal:
> [ClickHouse/clickhouse-connect#809](https://github.com/ClickHouse/clickhouse-connect/issues/809).
>
> Upstream treats `createClient({ connection })` as a **deliberately
> narrow, internal experiment to unblock the chDB integration** — not
> the start of a public plugin system. The hook's shape may change.
> This document describes the chdb-side implementation that uses it
> today; it does NOT promise a stable third-party plugin contract.

## What this delivers

A `chdb/connection` subpath export that lets users plug in-process
chDB into `@clickhouse/client` with a one-line change at construction:

```ts
import { createClient } from "@clickhouse/client";
import { createChdbConnection } from "chdb/connection";

const client = createClient({
connection: createChdbConnection({ path: ":memory:" }),
});
```

All downstream code (`client.query` / `client.insert` / `client.exec` /
`client.command` / `client.ping` / `client.close`) runs unchanged
against the in-process backend.

## Ownership split

```
┌──────────────────────────────────────┐ ┌──────────────────────────────────────┐
│ @clickhouse/client (upstream) │ │ chdb-node (this repo) │
│ │ │ │
│ • Connection<Stream> interface │ │ • ChdbConnection implements │
│ (internal, experimental) │ │ Connection<Readable> verbatim │
│ • createClient({ connection }) │ ◄── │ • chdb/connection subpath export │
│ one experimental option │ │ • tests/clickhouse-js/skip_list.json │
│ │ │ (chdb-owned blacklist) │
│ ❌ zero chdb code │ │ • tests/clickhouse-js/runner.mjs │
│ ❌ zero chdb tests │ │ clones upstream, injects chdb, │
│ ❌ zero chdb CI dependency │ │ applies skip_list │
└──────────────────────────────────────┘ └──────────────────────────────────────┘
```

The only upstream change is the `createClient({ connection })`
injection point (≈3 lines wrapping `NodeConfigImpl`). All chdb-side
logic (the adapter, the parity sync policy, the skip list, the CI
matrix) lives in chdb-node so the upstream codebase stays
backend-agnostic.

## Public surface (this repo)

`chdb/connection` exports:

| name | purpose |
|---|---|
| `createChdbConnection(opts)` | factory; default `path: ':memory:'` (refcounted shared temp dir across same-process memory connections) |
| `ChdbConnection` | class implementing `Connection<Readable>` |
| `ChdbExtension` | type for the `.chdb` namespace |
| `Connection`, `ConnQueryResult`, `ConnInsertParams`, … | RE-EXPORTED from `@clickhouse/client-common` (chdb-node does not define its own) |

### Why re-export, not redefine

ChdbConnection must match `@clickhouse/client-common`'s
`Connection<Stream.Readable>` interface EXACTLY for the injection to
work. Defining our own "compatible" interface here would silently
drift. Instead we add `@clickhouse/client-common` as a devDependency,
re-export the real types, and let TypeScript prove conformance at
compile time. Bumping the upstream package forces a deliberate
decision in chdb-node about which contract version we conform to.

### `.chdb` extension namespace

`ChdbConnection.chdb` exposes chDB-specific operations that do NOT
belong on the cross-backend Connection contract:

```ts
conn.chdb.session.path // bound on-disk path
conn.chdb.session.isTemp // process-managed temp dir?
conn.chdb.queryAsync(sql, opts) // native ChdbResult (bytes/text/json/toArrow)
conn.chdb.queryStream(sql, opts) // chunked streaming
conn.chdb.rawInsert(params) // Buffer-passthrough insert
conn.chdb.streamInsert(params) // backpressured streaming insert
```

The exposed set is intentionally limited to what chdb-node actually
supports today; aspirational items (`Python()` table function, UDF
registration) are NOT exposed because chdb-node does not implement
them yet.

### `supportsZeroCopyStreaming`

A ChdbConnection-only property (NOT part of the upstream Connection
contract) flagging whether the connection can surface result buffers
to JS without copying. **`false` today**: chdb-node copies each result
chunk into a JS Buffer (see `lib/chdb_node.cpp:678–680` "true zero-copy
is a later optimization"; `src/result.ts:56–58` "bytes are owned by
JS (copied off the engine)"). The flag will flip to `true` once
chdb-node lands the N-API external-ArrayBuffer Arrow path. Downstream
code that wants to make a routing decision today can read it as
`Boolean((conn as any).supportsZeroCopyStreaming)`.

## Test sync policy

Cross-suite parity (running `@clickhouse/client`'s integration suite
against `ChdbConnection`) is governed by `tests/clickhouse-js/`. The
upstream `@clickhouse/client` repo has NO chdb-related test
infrastructure — no markers, no skip-profile loader, no chdb test
runner. Everything lives here.

### Which upstream ref we test against

`tests/clickhouse-js/skip_list.json#syncedAgainst.ref` records the
`@clickhouse/client` ref this skip list is calibrated against:

| `@clickhouse/client` state | Runner targets |
|---|---|
| `createClient({ connection })` PR open on personal fork | `ShawnChen-Sirius/clickhouse-js feat/pluggable-connection` |
| **PR merged to upstream (current state — #879)** | **`ClickHouse/clickhouse-js main`** |
| Released | the latest released tag |

### When chdb-node re-runs parity

**Only when `@clickhouse/client` cuts a new release.** Between
releases, the suite is frozen against the recorded ref; the
`skip_list.json` is stable. When a new release lands:

1. Bump `syncedAgainst.ref` in `skip_list.json`.
2. Run `node tests/clickhouse-js/runner.mjs --refresh`.
3. For each NEW failure, decide: real capability gap (add to
`skipFiles`) vs ChdbConnection bug (fix in `src/connection/`).
4. Commit the updated `skip_list.json` and any ChdbConnection fixes.

This means chdb-node does **not** participate in upstream CI on every
upstream commit — that would impose maintenance cost on
`@clickhouse/client` (and contradict the "minimal upstream
modification" principle). The skip_list is a snapshot, refreshed on
release boundaries.

### Initial categories on the skip list

The first `skip_list.json` (calibrated against `ClickHouse/clickhouse-js main`
post-#879) carries 25 file-level skips grouped as:

- **HTTP transport** — no socket, no keep-alive, no agents, no
HTTP-level compression / headers / `/ping`.
- **Auth / RBAC / TLS** — chdb has no auth layer.
- **Server runtime / cluster** — `system.query_log`, multi-client
cluster scenarios.
- **HTTP session_id semantics** — chdb persistence is its
`Session.path`, not an HTTP session.
- **Abort divergence** — chdb's single-shot abort rejects early while
the underlying computation runs to completion (C ABI has no
interrupt); documented divergence.

Each entry has a `reason` so reviewers can tell at a glance whether a
skip is a real chdb capability gap or an HTTP-only-feature mismatch.
When `@clickhouse/client` cuts a new release and a previously-passing
test starts failing under the runner, the triage rule is: fixable in
`ChdbConnection` → patch the adapter, real capability gap → add to
`skipFiles` / `skipTests` with a one-line reason.

## Implementation notes

### Memory model

`createChdbConnection({ path: ':memory:' })` doesn't ask Layer 1 for
a fresh temp dir each time — that would fail libchdb's
single-active-data-directory rule. Instead chdb-node maintains a
process-shared, refcounted memory dir; all `:memory:` connections
share it. The multi-connection model from
[chdb-node#51](https://github.com/chdb-io/chdb-node/pull/51) lets N
native connections coexist on the same bound path, so this gives the
same shared-state semantics as clickhouse-js's `chdb://memory` while
keeping each ChdbConnection's query state independent.

### Eager-buffered `query` / `exec`

Both materialize via `Session.queryAsync` and surface the result as a
one-shot Readable. clickhouse-js's `Client.query` callers expect
errors at `await connection.query(...)`, not mid-stream — eager
buffering preserves that. The day chdb-node lands the N-API
external-ArrayBuffer reader, ChdbConnection will route streaming
through `queryStream`-equivalents and `supportsZeroCopyStreaming` will
flip to `true`.

### `insert`: inline body

clickhouse-js sends `INSERT INTO t (...) FORMAT X` with the body as a
SEPARATE `values` parameter. chDB's parser accepts the body inline
after the statement, so ChdbConnection materializes the values stream
into a string and concatenates with the query before executing as a
single statement. Callers that want chdb's zero-copy
Buffer-passthrough insert can reach for the `.chdb.rawInsert` escape
hatch.

### Synthetic `query_id` / `response_headers` / `http_status_code`

The Connection contract requires these on every result. chdb has no
server-side query_id, no HTTP layer; ChdbConnection synthesizes a
UUIDv4, returns `{}` for headers and `200` for status. Test code that
relies on `query_id` for log correlation should expect a unique value
but cannot map it back to any cross-process log entry.

## What this design deliberately does NOT do

- **No chdb-specific code in `@clickhouse/client`.** The only upstream
change is the one `connection?:` option on `createClient`.
- **No per-feature test markers in upstream tests.** Backends own
their own blacklists.
- **No HTTP loopback inside chdb-node.** Going through a wire format
would erase chdb's reason for existing (in-process, no marshalling).
- **No client-web support.** chdb is N-API; browsers cannot run it. A
future web-runtime embedded backend would repeat this pattern in
`client-web` symmetrically.
Loading