Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/chdb-node-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,15 @@ jobs:
run: npm run libchdb
- name: Build (addon + dist)
run: npm run build
- name: Exercise the freshly built addon (not the published prebuilt)
# The loader prefers the published @chdb/lib-<platform> subpackage over a
# local build (see dist/loader.js). That published binary lags behind
# in-repo native source — e.g. the Layer 3 Arrow C Data Interface methods
# (ArrowRegisterColumns/…) only exist once republished. Drop the installed
# subpackage so loadNative() falls through to the just-built
# build/Release/chdb_node.node and the suite tests this PR's native source.
shell: bash
run: rm -rf node_modules/@chdb/lib-*
- name: Test (v2 mocha + v3 vitest)
run: npm run test:all

Expand Down
95 changes: 95 additions & 0 deletions AGENT.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# AGENT.md — working on chdb-node

Orientation and gotchas for agents (and humans) editing this repo. Pairs with
`README.md` (user-facing) and `docs/design/` (architecture).

## Layered surface

- **Layer 1** — `index.js` (CommonJS entrypoint): the process-wide session,
`queryAsync` / `queryBindAsync` (server-side parameter binding), `Session`,
inserts, `queryStream`, and the native addon bindings. It owns the
pending-op and session registries.
- **Layer 3** — `src/layer3/` (TypeScript fluent builder): compiles a chain to
`{ sql, parameters }` and forwards execution to Layer 1. **It never touches
the native addon directly** and never puts values into SQL text — values are
pre-serialized and bound server-side (`runtime.ts`). Keep that invariant.

### One connection per process

libchdb allows exactly **one active data directory per process**. All v3 tests
share a single fork and run serially (`vitest.config.ts`); a leaked session
cascades into unrelated files. The global `afterEach` in `test/v3/setup.ts`
force-closes anything a test left open. Every importer must share one `index.js`
instance — that's why vitest externalizes `…/index.js` imports.

## Arrow input — `registerArrowTable` lifecycle

`registerArrowTable(name, columns, { session })` pins JS columnar data as an
`arrowstream('<name>')` table the engine scans with zero IPC copy
(`src/layer3/execute/arrow-input.ts`). The native side holds the JS buffers via
`Napi::Reference` until the table is unregistered.

**Always `close()` the handle — ideally in `try/finally`:**

```ts
const t = registerArrowTable('events', [
{ name: 'id', type: 'Int32', data: new Int32Array([1, 2, 3]) },
])
try {
const rows = await selectFrom(chTable.arrowstream('events').as('e'))
.select('id').execute()
} finally {
t.close() // releases the native-side buffer pins
}
```

- **Single-pass semantics.** The engine consumes the underlying Arrow array on
each scan, so a *second* query against the same handle reads zero rows. Call
`t.refresh()` between repeated reads of the same data, or build a new handle
per query.
- **GC safety net (best-effort, non-deterministic).** A handle dropped without
`close()` is unregistered by a `FinalizationRegistry` fallback so native stops
pinning its buffers. This is *not* a substitute for `close()`: the JS spec
gives no guarantee finalizers run at all (or when). Treat it purely as a
backstop against forgotten handles, and still `close()` explicitly.
- A long-lived `session` that registers tables in a loop **without** closing
them grows memory monotonically — the connection only sweeps registrations on
close.

## Streaming large results — follow-up, not yet shipped

`.execute()` buffers the whole result (peak ≈ 3–4×N for the row view). For very
large result sets a lazy `.stream()` terminal on the SELECT builder is the right
tool, but it is **deliberately not in the builder yet**:

- Layer 1 streaming (`queryStream` → `query_conn_streaming_n`) has **no params
argument**, so a streaming terminal could not bind parameters server-side
without either throwing on bound queries (a footgun shaped by a temporary ABI
gap) or rendering literals into SQL text (violates the no-values-in-SQL
invariant).
- The clean fix is upstream: add server-side binding to the streaming C ABI,
then build `.stream()` once with the same safety as `.execute()`. Tracked as a
follow-up; don't add a half-feature in the meantime.

When `.stream()` does land, document here: when to prefer it over `.execute()`,
the async-iterator usage, and that a session allows only one active stream at a
time.

## Build & test

- **TypeScript only** (Layer 3, the common case): `npm run build:ts`, then
`npm run test:v3` (`vitest run`).
- **Native addon** (anything under `lib/`, or running the Arrow tests, which
exercise the real addon): rebuild with `npm run build` (`node-gyp configure
build` + `fixloaderpath` + `build:ts`). Use the repo's node-gyp, not a global
one. `test/v3/layer3/arrow-input.test.ts` needs the rebuilt addon.
- The GC fallback test needs `--expose-gc`; it's wired via `execArgv` in
`vitest.config.ts` and self-skips if unavailable.
- Legacy v2 byte-compat tests stay on mocha (`npm test`) as the untouched
regression anchor.

## Conventions

Commit/PR titles: capitalized imperative verb, **no prefix tags** (ClickHouse
style) — e.g. `Add a FinalizationRegistry safety net for Arrow input`, not
`feat: …`.
6 changes: 6 additions & 0 deletions index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -356,3 +356,9 @@ export function version(): {
arch: string;
napi?: number;
};

// Layer 3: the fluent, immutable query builder, a sibling of the pluggable
// Connection surface (`chdb/connection`), both sitting on Layer 1. `selectFrom`,
// `database`, `insertInto`, `chFn`, `chTable`, the builders, the compile output
// types, and ChdbCompileError. Generated declarations live in dist/layer3.
export * from './dist/layer3';
28 changes: 27 additions & 1 deletion index.js
Original file line number Diff line number Diff line change
Expand Up @@ -728,4 +728,30 @@ function version() {
};
}

module.exports = { query, queryBind, queryAsync, queryBindAsync, insert, Session, version, _closeAllSessions, _drainPendingOps };
// Arrow C Data Interface — low-level binding. The high-level helper lives in
// Layer 3 (`src/layer3/execute/arrow-input.ts`); this module just routes the
// call to the native addon. The connection arg is a `Session._handle` (the
// External returned by CreateConnection) or null for the process-wide default.
function _arrowRegisterColumns(connection, tableName, columns) {
return chdbNode.ArrowRegisterColumns(connection ?? null, tableName, columns);
}
function _arrowUnregister(connection, tableName) {
return chdbNode.ArrowUnregister(connection ?? null, tableName);
}

module.exports = {
query, queryBind, queryAsync, queryBindAsync, insert,
Session, version,
_closeAllSessions, _drainPendingOps,
_arrowRegisterColumns, _arrowUnregister,
};

// Layer 3: the fluent, immutable query builder. It sits on Layer 1, a sibling of
// the pluggable Connection surface (`chdb/connection`). Required at the BOTTOM,
// after module.exports is populated, so the lazy Layer 1 accessor in dist/layer3
// sees a fully-formed export object. ChdbCompileError (the only net-new error)
// rides along via dist/layer3's own re-export.
const layer3 = require('./dist/layer3/index.js');
for (const name of Object.keys(layer3)) {
if (module.exports[name] === undefined) module.exports[name] = layer3[name];
}
6 changes: 6 additions & 0 deletions index.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,10 @@ export const insert = mod.insert
export const Session = mod.Session
export const version = mod.version

// Layer 3: ChdbCompileError is the only net-new error class; export it as a
// named ESM binding so it is catchable by class. The fluent builder surface
// (selectFrom / database / insertInto / connect / chFn / chTable / sql / eb) is
// reached through the default export (`import chdb from 'chdb'`).
export const ChdbCompileError = mod.ChdbCompileError

export default mod
Loading
Loading