diff --git a/.github/workflows/chdb-node-test.yml b/.github/workflows/chdb-node-test.yml index 0e0565b..e23d4e4 100644 --- a/.github/workflows/chdb-node-test.yml +++ b/.github/workflows/chdb-node-test.yml @@ -34,6 +34,15 @@ jobs: run: npm run libchdb - name: Build (addon + dist) run: npm run build + - name: Exercise the freshly built addon (not the published prebuilt) + # The loader prefers the published @chdb/lib- subpackage over a + # local build (see dist/loader.js). That published binary lags behind + # in-repo native source — e.g. the Layer 3 Arrow C Data Interface methods + # (ArrowRegisterColumns/…) only exist once republished. Drop the installed + # subpackage so loadNative() falls through to the just-built + # build/Release/chdb_node.node and the suite tests this PR's native source. + shell: bash + run: rm -rf node_modules/@chdb/lib-* - name: Test (v2 mocha + v3 vitest) run: npm run test:all diff --git a/AGENT.md b/AGENT.md new file mode 100644 index 0000000..72d6af5 --- /dev/null +++ b/AGENT.md @@ -0,0 +1,95 @@ +# AGENT.md — working on chdb-node + +Orientation and gotchas for agents (and humans) editing this repo. Pairs with +`README.md` (user-facing) and `docs/design/` (architecture). + +## Layered surface + +- **Layer 1** — `index.js` (CommonJS entrypoint): the process-wide session, + `queryAsync` / `queryBindAsync` (server-side parameter binding), `Session`, + inserts, `queryStream`, and the native addon bindings. It owns the + pending-op and session registries. +- **Layer 3** — `src/layer3/` (TypeScript fluent builder): compiles a chain to + `{ sql, parameters }` and forwards execution to Layer 1. **It never touches + the native addon directly** and never puts values into SQL text — values are + pre-serialized and bound server-side (`runtime.ts`). Keep that invariant. + +### One connection per process + +libchdb allows exactly **one active data directory per process**. All v3 tests +share a single fork and run serially (`vitest.config.ts`); a leaked session +cascades into unrelated files. The global `afterEach` in `test/v3/setup.ts` +force-closes anything a test left open. Every importer must share one `index.js` +instance — that's why vitest externalizes `…/index.js` imports. + +## Arrow input — `registerArrowTable` lifecycle + +`registerArrowTable(name, columns, { session })` pins JS columnar data as an +`arrowstream('')` table the engine scans with zero IPC copy +(`src/layer3/execute/arrow-input.ts`). The native side holds the JS buffers via +`Napi::Reference` until the table is unregistered. + +**Always `close()` the handle — ideally in `try/finally`:** + +```ts +const t = registerArrowTable('events', [ + { name: 'id', type: 'Int32', data: new Int32Array([1, 2, 3]) }, +]) +try { + const rows = await selectFrom(chTable.arrowstream('events').as('e')) + .select('id').execute() +} finally { + t.close() // releases the native-side buffer pins +} +``` + +- **Single-pass semantics.** The engine consumes the underlying Arrow array on + each scan, so a *second* query against the same handle reads zero rows. Call + `t.refresh()` between repeated reads of the same data, or build a new handle + per query. +- **GC safety net (best-effort, non-deterministic).** A handle dropped without + `close()` is unregistered by a `FinalizationRegistry` fallback so native stops + pinning its buffers. This is *not* a substitute for `close()`: the JS spec + gives no guarantee finalizers run at all (or when). Treat it purely as a + backstop against forgotten handles, and still `close()` explicitly. +- A long-lived `session` that registers tables in a loop **without** closing + them grows memory monotonically — the connection only sweeps registrations on + close. + +## Streaming large results — follow-up, not yet shipped + +`.execute()` buffers the whole result (peak ≈ 3–4×N for the row view). For very +large result sets a lazy `.stream()` terminal on the SELECT builder is the right +tool, but it is **deliberately not in the builder yet**: + +- Layer 1 streaming (`queryStream` → `query_conn_streaming_n`) has **no params + argument**, so a streaming terminal could not bind parameters server-side + without either throwing on bound queries (a footgun shaped by a temporary ABI + gap) or rendering literals into SQL text (violates the no-values-in-SQL + invariant). +- The clean fix is upstream: add server-side binding to the streaming C ABI, + then build `.stream()` once with the same safety as `.execute()`. Tracked as a + follow-up; don't add a half-feature in the meantime. + +When `.stream()` does land, document here: when to prefer it over `.execute()`, +the async-iterator usage, and that a session allows only one active stream at a +time. + +## Build & test + +- **TypeScript only** (Layer 3, the common case): `npm run build:ts`, then + `npm run test:v3` (`vitest run`). +- **Native addon** (anything under `lib/`, or running the Arrow tests, which + exercise the real addon): rebuild with `npm run build` (`node-gyp configure + build` + `fixloaderpath` + `build:ts`). Use the repo's node-gyp, not a global + one. `test/v3/layer3/arrow-input.test.ts` needs the rebuilt addon. +- The GC fallback test needs `--expose-gc`; it's wired via `execArgv` in + `vitest.config.ts` and self-skips if unavailable. +- Legacy v2 byte-compat tests stay on mocha (`npm test`) as the untouched + regression anchor. + +## Conventions + +Commit/PR titles: capitalized imperative verb, **no prefix tags** (ClickHouse +style) — e.g. `Add a FinalizationRegistry safety net for Arrow input`, not +`feat: …`. diff --git a/index.d.ts b/index.d.ts index 4a8d793..5dadda5 100644 --- a/index.d.ts +++ b/index.d.ts @@ -356,3 +356,9 @@ export function version(): { arch: string; napi?: number; }; + +// Layer 3: the fluent, immutable query builder, a sibling of the pluggable +// Connection surface (`chdb/connection`), both sitting on Layer 1. `selectFrom`, +// `database`, `insertInto`, `chFn`, `chTable`, the builders, the compile output +// types, and ChdbCompileError. Generated declarations live in dist/layer3. +export * from './dist/layer3'; diff --git a/index.js b/index.js index f2aacfc..bfb006e 100644 --- a/index.js +++ b/index.js @@ -728,4 +728,30 @@ function version() { }; } -module.exports = { query, queryBind, queryAsync, queryBindAsync, insert, Session, version, _closeAllSessions, _drainPendingOps }; +// Arrow C Data Interface — low-level binding. The high-level helper lives in +// Layer 3 (`src/layer3/execute/arrow-input.ts`); this module just routes the +// call to the native addon. The connection arg is a `Session._handle` (the +// External returned by CreateConnection) or null for the process-wide default. +function _arrowRegisterColumns(connection, tableName, columns) { + return chdbNode.ArrowRegisterColumns(connection ?? null, tableName, columns); +} +function _arrowUnregister(connection, tableName) { + return chdbNode.ArrowUnregister(connection ?? null, tableName); +} + +module.exports = { + query, queryBind, queryAsync, queryBindAsync, insert, + Session, version, + _closeAllSessions, _drainPendingOps, + _arrowRegisterColumns, _arrowUnregister, +}; + +// Layer 3: the fluent, immutable query builder. It sits on Layer 1, a sibling of +// the pluggable Connection surface (`chdb/connection`). Required at the BOTTOM, +// after module.exports is populated, so the lazy Layer 1 accessor in dist/layer3 +// sees a fully-formed export object. ChdbCompileError (the only net-new error) +// rides along via dist/layer3's own re-export. +const layer3 = require('./dist/layer3/index.js'); +for (const name of Object.keys(layer3)) { + if (module.exports[name] === undefined) module.exports[name] = layer3[name]; +} diff --git a/index.mjs b/index.mjs index 65055e2..b67b235 100644 --- a/index.mjs +++ b/index.mjs @@ -16,4 +16,10 @@ export const insert = mod.insert export const Session = mod.Session export const version = mod.version +// Layer 3: ChdbCompileError is the only net-new error class; export it as a +// named ESM binding so it is catchable by class. The fluent builder surface +// (selectFrom / database / insertInto / connect / chFn / chTable / sql / eb) is +// reached through the default export (`import chdb from 'chdb'`). +export const ChdbCompileError = mod.ChdbCompileError + export default mod diff --git a/lib/chdb_node.cpp b/lib/chdb_node.cpp index 409282f..58b2788 100644 --- a/lib/chdb_node.cpp +++ b/lib/chdb_node.cpp @@ -4,8 +4,12 @@ #include #include #include +#include #include #include +#include +#include +#include #include typedef void * ChdbConnection; @@ -90,10 +94,19 @@ static chdb_connection *open_raw(const std::string &path) { return (conn_ptr && *conn_ptr) ? conn_ptr : nullptr; } +// Arrow-table registry cleanup (defined alongside g_live_arrow_tables below). +// A connection's pinned JS buffers live in that registry keyed by the raw +// chdb_connection*, so they must be dropped when the connection closes — else +// they leak for the life of the process, the engine retains dangling refs, and +// a later reused connection address could collide with the stale key. +static void erase_arrow_tables_for_conn(chdb_connection *conn); +static void clear_all_arrow_tables(); + // Close every live connection. atexit backstop (the JS layer closes sessions on // 'exit' too); also used so a leaked connection never outlives the process. static void hard_close_all() { std::lock_guard lk(g_reg_mu); + clear_all_arrow_tables(); for (chdb_connection *c : g_reg.conns) { if (c) chdb_close_conn(c); } @@ -146,6 +159,7 @@ static chdb_connection *acquire_session_conn(const std::string &path, char **err // Only the in-memory default is up (boundKey ""). It is transient and must // yield so this session can bind a real data directory. if (g_reg.defaultConn) { + erase_arrow_tables_for_conn(g_reg.defaultConn); chdb_close_conn(g_reg.defaultConn); g_reg.conns.erase(g_reg.defaultConn); g_reg.defaultConn = nullptr; @@ -176,6 +190,9 @@ static void release_session_conn(chdb_connection *conn) { std::lock_guard lk(g_reg_mu); auto it = g_reg.conns.find(conn); if (it == g_reg.conns.end()) return; // already released / unknown handle + // Drop any Arrow tables pinned on this connection before closing it, so the + // JS buffer references aren't orphaned in g_live_arrow_tables. + erase_arrow_tables_for_conn(conn); chdb_close_conn(conn); g_reg.conns.erase(it); if (conn == g_reg.defaultConn) g_reg.defaultConn = nullptr; @@ -877,6 +894,417 @@ Napi::String QueryWithConnectionWrapper(const Napi::CallbackInfo & info) { return output; } +//===---------------------------------------------------------------------===// +// Arrow C Data Interface — caller-built struct layout. +// +// The standard Arrow ABI structs are reproduced here so the addon doesn't have +// to depend on `arrow/c/abi.h` (chdb ships with Arrow linked inside libchdb but +// does not re-expose the header). Layout matches the spec at +// https://arrow.apache.org/docs/format/CDataInterface.html so a chdb-side +// `reinterpret_cast` matches our pointers byte-for-byte. +//===---------------------------------------------------------------------===// + +struct AbiSchema { + const char * format; + const char * name; + const char * metadata; + int64_t flags; + int64_t n_children; + AbiSchema ** children; + AbiSchema * dictionary; + void (*release)(AbiSchema *); + void * private_data; +}; + +struct AbiArray { + int64_t length; + int64_t null_count; + int64_t offset; + int64_t n_buffers; + int64_t n_children; + const void ** buffers; + AbiArray ** children; + AbiArray * dictionary; + void (*release)(AbiArray *); + void * private_data; +}; + +// No-op release callbacks. chdb-arrow.cpp saves each child's release into a +// vector before scanning, then restores it; passing NULL would segfault there. +static void AbiSchemaNoopRelease(AbiSchema * s) { s->release = nullptr; } +static void AbiArrayNoopRelease(AbiArray * a) { a->release = nullptr; } + +//===---------------------------------------------------------------------===// +// Arrow C Data Interface bindings. +// +// chdb_arrow_scan / chdb_arrow_array_scan let the engine read a caller-built +// Arrow buffer as if it were a regular table (the table function is registered +// under a chosen name; SELECT * FROM reads it). The wire types declared +// in chdb.h are opaque, but chdb-arrow.cpp inside libchdb just reinterprets +// them as the standard ArrowSchema * / ArrowArray * / ArrowArrayStream * from +// the Arrow C Data Interface — so JS hands the addon a pointer (BigInt) to a +// caller-owned, correctly-laid-out struct, and the addon forwards it. +// +// Memory: the caller owns the struct and its referenced buffers until the +// matching Unregister. chdb keeps a stable reference to the data in its own +// ArrowStreamRegistry; on Unregister chdb releases its hold and we free the +// caller-side wrapper (typed-array references included). +//===---------------------------------------------------------------------===// + +// Resolve the connection arg: undefined/null -> default in-process connection; +// External -> a Session-bound chdb_connection*. Throws a JS exception +// and returns false on a misuse / no-default-available error. +static bool resolveArrowConn(Napi::Env env, Napi::Value handle, chdb_connection ** out) { + if (handle.IsExternal()) { + *out = static_cast(handle.As>().Data()); + if (*out == nullptr) { + Napi::Error::New(env, "Connection handle is null (closed?)").ThrowAsJavaScriptException(); + return false; + } + return true; + } + if (handle.IsNull() || handle.IsUndefined()) { + char * err = nullptr; + chdb_connection * c = get_default_conn(&err); + if (c == nullptr) { + std::string msg = err ? err : "no default connection available"; + if (err) free(err); + Napi::Error::New(env, msg).ThrowAsJavaScriptException(); + return false; + } + *out = c; + return true; + } + Napi::TypeError::New(env, "Connection must be a Session handle or null").ThrowAsJavaScriptException(); + return false; +} + +// Pull a `BigInt` off the call info as a raw pointer. JS hands us 64-bit +// pointers losslessly through BigInt; anything else is a misuse. +static bool pointerArg(Napi::Env env, Napi::Value v, const char * field, uint64_t * out) { + if (!v.IsBigInt()) { + Napi::TypeError::New(env, std::string(field) + " must be a BigInt pointer").ThrowAsJavaScriptException(); + return false; + } + bool lossless = false; + *out = v.As().Uint64Value(&lossless); + if (!lossless) { + Napi::TypeError::New(env, std::string(field) + " BigInt did not convert losslessly").ThrowAsJavaScriptException(); + return false; + } + if (*out == 0) { + Napi::Error::New(env, std::string(field) + " pointer is null").ThrowAsJavaScriptException(); + return false; + } + return true; +} + +// nativeArrowRegisterArray(conn, tableName, schemaPtr, arrayPtr) +// schemaPtr / arrayPtr are JS-owned ArrowSchema * / ArrowArray * (BigInt). +// chdb takes a snapshot via chdb_arrow_array_scan and registers the table. +Napi::Value ArrowRegisterArrayWrapper(const Napi::CallbackInfo & info) { + Napi::Env env = info.Env(); + if (info.Length() < 4 || !info[1].IsString()) { + Napi::TypeError::New(env, "Usage: connection, tableName, schemaPtr, arrayPtr").ThrowAsJavaScriptException(); + return env.Undefined(); + } + chdb_connection * conn = nullptr; + if (!resolveArrowConn(env, info[0], &conn)) return env.Undefined(); + std::string tableName = info[1].As().Utf8Value(); + uint64_t schemaPtr = 0, arrayPtr = 0; + if (!pointerArg(env, info[2], "schemaPtr", &schemaPtr)) return env.Undefined(); + if (!pointerArg(env, info[3], "arrayPtr", &arrayPtr)) return env.Undefined(); + + chdb_state st = chdb_arrow_array_scan( + *conn, + tableName.c_str(), + reinterpret_cast(static_cast(schemaPtr)), + reinterpret_cast(static_cast(arrayPtr))); + if (st != CHDBSuccess) { + Napi::Error::New(env, "chdb_arrow_array_scan failed for table " + tableName).ThrowAsJavaScriptException(); + return env.Undefined(); + } + return env.Undefined(); +} + +// nativeArrowRegisterStream(conn, tableName, streamPtr) — for multi-batch data. +Napi::Value ArrowRegisterStreamWrapper(const Napi::CallbackInfo & info) { + Napi::Env env = info.Env(); + if (info.Length() < 3 || !info[1].IsString()) { + Napi::TypeError::New(env, "Usage: connection, tableName, streamPtr").ThrowAsJavaScriptException(); + return env.Undefined(); + } + chdb_connection * conn = nullptr; + if (!resolveArrowConn(env, info[0], &conn)) return env.Undefined(); + std::string tableName = info[1].As().Utf8Value(); + uint64_t streamPtr = 0; + if (!pointerArg(env, info[2], "streamPtr", &streamPtr)) return env.Undefined(); + + chdb_state st = chdb_arrow_scan( + *conn, + tableName.c_str(), + reinterpret_cast(static_cast(streamPtr))); + if (st != CHDBSuccess) { + Napi::Error::New(env, "chdb_arrow_scan failed for table " + tableName).ThrowAsJavaScriptException(); + return env.Undefined(); + } + return env.Undefined(); +} + +//===---------------------------------------------------------------------===// +// High-level table registration. JS hands the addon a list of column +// descriptors (`name`, Arrow format string, plain Buffers / TypedArrays for +// validity / data / offsets), and the addon assembles the ArrowSchema and +// ArrowArray children + struct root so chdb can read the data with no IPC +// round-trip and no Arrow header dependency on the JS side. +// +// Lifetime: chdb keeps a reference inside its own ArrowStreamRegistry, so the +// underlying byte buffers (held by JS) must stay alive until Unregister is +// called. We keep both the C structs and Napi::Reference handles to each JS +// Buffer pinned in a per-(connection, tableName) live-registration record. +//===---------------------------------------------------------------------===// + +struct ArrowColumnBuffers { + std::vector> jsRefs; // pin each JS buffer + std::vector> bufferPtrs; // one per column (validity/data/offsets) +}; + +struct LiveArrowTable { + std::string format; // "+s" + child format strings (owned strings below) + std::vector formatStorage; + std::vector nameStorage; + // Children are stored contiguously and pointed at via pointer-arrays below. + std::vector childSchemas; + std::vector childArrays; + std::vector childSchemaPtrs; + std::vector childArrayPtrs; + AbiSchema rootSchema{}; + AbiArray rootArray{}; + std::vector rootBuffers; // root struct: one null bitmap (NULL) + ArrowColumnBuffers buffers; +}; + +// Key: (connection pointer, table name). Lookup on unregister to free. +using LiveArrowKey = std::pair; +struct LiveArrowKeyHash { + size_t operator()(const LiveArrowKey & k) const noexcept { + return std::hash{}(k.first) ^ std::hash{}(k.second); + } +}; +static std::unordered_map, LiveArrowKeyHash> g_live_arrow_tables; +static std::mutex g_live_arrow_mu; + +// Drop every registered Arrow table belonging to `conn`, releasing the pinned +// JS buffer references and C-side struct storage. Called from the connection +// close paths so a session closed without an explicit Unregister doesn't leak. +static void erase_arrow_tables_for_conn(chdb_connection *conn) { + std::lock_guard lock(g_live_arrow_mu); + for (auto it = g_live_arrow_tables.begin(); it != g_live_arrow_tables.end();) { + if (it->first.first == conn) it = g_live_arrow_tables.erase(it); + else ++it; + } +} + +// Drop the entire registry (process teardown backstop). +static void clear_all_arrow_tables() { + std::lock_guard lock(g_live_arrow_mu); + g_live_arrow_tables.clear(); +} + +// nativeArrowRegisterColumns(conn, tableName, columns) +// columns: Array<{ name, format, length, nullCount, buffers: Array }> +// - format: an Arrow C Data Interface format string ("i"/"l"/"g"/"b"/"u"/...) +// - buffers[0] = validity bitmap (null if nullCount === 0) +// - buffers[1] = data +// - buffers[2] = offsets (utf8 only) +Napi::Value ArrowRegisterColumnsWrapper(const Napi::CallbackInfo & info) { + Napi::Env env = info.Env(); + if (info.Length() < 3 || !info[1].IsString() || !info[2].IsArray()) { + Napi::TypeError::New(env, "Usage: connection, tableName, columns[]").ThrowAsJavaScriptException(); + return env.Undefined(); + } + chdb_connection * conn = nullptr; + if (!resolveArrowConn(env, info[0], &conn)) return env.Undefined(); + std::string tableName = info[1].As().Utf8Value(); + Napi::Array columns = info[2].As(); + uint32_t ncols = columns.Length(); + if (ncols == 0) { + Napi::TypeError::New(env, "registerArrowTable: at least one column required").ThrowAsJavaScriptException(); + return env.Undefined(); + } + + auto live = std::make_unique(); + live->formatStorage.reserve(ncols); + live->nameStorage.reserve(ncols); + live->childSchemas.resize(ncols); + live->childArrays.resize(ncols); + live->childSchemaPtrs.resize(ncols); + live->childArrayPtrs.resize(ncols); + live->buffers.bufferPtrs.resize(ncols); + + int64_t rootLength = 0; + bool rootLengthSet = false; + + for (uint32_t i = 0; i < ncols; i++) { + Napi::Value colv = columns[i]; + if (!colv.IsObject()) { + Napi::TypeError::New(env, "Each column must be an object").ThrowAsJavaScriptException(); + return env.Undefined(); + } + Napi::Object col = colv.As(); + Napi::Value nameV = col.Get("name"); + Napi::Value formatV = col.Get("format"); + Napi::Value lengthV = col.Get("length"); + Napi::Value nullCountV = col.Get("nullCount"); + Napi::Value buffersV = col.Get("buffers"); + if (!nameV.IsString() || !formatV.IsString() || !lengthV.IsNumber() || !buffersV.IsArray()) { + Napi::TypeError::New(env, "Column needs { name, format, length, buffers }").ThrowAsJavaScriptException(); + return env.Undefined(); + } + live->nameStorage.emplace_back(nameV.As().Utf8Value()); + live->formatStorage.emplace_back(formatV.As().Utf8Value()); + int64_t length = lengthV.As().Int64Value(); + int64_t nullCount = nullCountV.IsNumber() ? nullCountV.As().Int64Value() : 0; + // Reject negatives explicitly: a negative length flows straight into + // ArrowArray.length, and a negative nullCount mismatches the validity bitmap. + if (length < 0 || nullCount < 0) { + Napi::TypeError::New(env, "Column length and nullCount must be non-negative").ThrowAsJavaScriptException(); + return env.Undefined(); + } + // A separate flag (not a -1 sentinel) so a column whose length is legitimately + // any value, including 0, sets the root length on the first column. + if (!rootLengthSet) { + rootLength = length; + rootLengthSet = true; + } else if (rootLength != length) { + Napi::TypeError::New(env, "All columns must have the same length").ThrowAsJavaScriptException(); + return env.Undefined(); + } + + Napi::Array bufs = buffersV.As(); + uint32_t nbufs = bufs.Length(); + std::vector ptrs; + ptrs.reserve(nbufs); + for (uint32_t b = 0; b < nbufs; b++) { + Napi::Value bv = bufs[b]; + if (bv.IsNull() || bv.IsUndefined()) { + ptrs.push_back(nullptr); + continue; + } + if (!bv.IsBuffer()) { + Napi::TypeError::New(env, "Column buffers must be Node Buffers or null").ThrowAsJavaScriptException(); + return env.Undefined(); + } + Napi::Buffer buf = bv.As>(); + ptrs.push_back(buf.Data()); + // Pin the JS Buffer so its underlying data stays alive until Unregister. + live->buffers.jsRefs.push_back(Napi::Reference::New(bv, 1)); + } + live->buffers.bufferPtrs[i] = std::move(ptrs); + + // Child ArrowSchema. + AbiSchema & cs = live->childSchemas[i]; + cs = {}; + cs.format = live->formatStorage.back().c_str(); + cs.name = live->nameStorage.back().c_str(); + cs.metadata = nullptr; + cs.flags = 2; // ARROW_FLAG_NULLABLE + cs.n_children = 0; + cs.children = nullptr; + cs.dictionary = nullptr; + cs.release = AbiSchemaNoopRelease; + cs.private_data = nullptr; + live->childSchemaPtrs[i] = &cs; + + // Child ArrowArray. + AbiArray & ca = live->childArrays[i]; + ca = {}; + ca.length = length; + ca.null_count = nullCount; + ca.offset = 0; + ca.n_buffers = static_cast(live->buffers.bufferPtrs[i].size()); + ca.n_children = 0; + ca.buffers = live->buffers.bufferPtrs[i].data(); + ca.children = nullptr; + ca.dictionary = nullptr; + ca.release = AbiArrayNoopRelease; + ca.private_data = nullptr; + live->childArrayPtrs[i] = &ca; + } + + // Root struct schema. + live->format = "+s"; + live->rootSchema = {}; + live->rootSchema.format = live->format.c_str(); + live->rootSchema.name = ""; + live->rootSchema.metadata = nullptr; + live->rootSchema.flags = 0; + live->rootSchema.n_children = ncols; + live->rootSchema.children = live->childSchemaPtrs.data(); + live->rootSchema.dictionary = nullptr; + live->rootSchema.release = AbiSchemaNoopRelease; + live->rootSchema.private_data = nullptr; + + // Root struct array. Struct layout has exactly one buffer slot (validity); + // we pass NULL to indicate no nulls at the row level. + live->rootBuffers = { nullptr }; + live->rootArray = {}; + live->rootArray.length = rootLength; + live->rootArray.null_count = 0; + live->rootArray.offset = 0; + live->rootArray.n_buffers = 1; + live->rootArray.n_children = ncols; + live->rootArray.buffers = live->rootBuffers.data(); + live->rootArray.children = live->childArrayPtrs.data(); + live->rootArray.dictionary = nullptr; + live->rootArray.release = AbiArrayNoopRelease; + live->rootArray.private_data = nullptr; + + chdb_state st = chdb_arrow_array_scan( + *conn, + tableName.c_str(), + reinterpret_cast(&live->rootSchema), + reinterpret_cast(&live->rootArray)); + if (st != CHDBSuccess) { + Napi::Error::New(env, "chdb_arrow_array_scan failed for table " + tableName).ThrowAsJavaScriptException(); + return env.Undefined(); + } + + { + std::lock_guard lock(g_live_arrow_mu); + LiveArrowKey key{ conn, tableName }; + // Replace any prior entry — Unregister would otherwise leak the old refs. + g_live_arrow_tables[key] = std::move(live); + } + return env.Undefined(); +} + +// nativeArrowUnregister(conn, tableName) — drops the registry entry. +Napi::Value ArrowUnregisterWrapper(const Napi::CallbackInfo & info) { + Napi::Env env = info.Env(); + if (info.Length() < 2 || !info[1].IsString()) { + Napi::TypeError::New(env, "Usage: connection, tableName").ThrowAsJavaScriptException(); + return env.Undefined(); + } + chdb_connection * conn = nullptr; + if (!resolveArrowConn(env, info[0], &conn)) return env.Undefined(); + std::string tableName = info[1].As().Utf8Value(); + + chdb_state st = chdb_arrow_unregister_table(*conn, tableName.c_str()); + if (st != CHDBSuccess) { + Napi::Error::New(env, "chdb_arrow_unregister_table failed for table " + tableName).ThrowAsJavaScriptException(); + return env.Undefined(); + } + // Drop the JS-buffer pins and the C-side struct storage. After chdb's + // unregister returns, the engine no longer reaches our buffers, so freeing + // them is safe. + { + std::lock_guard lock(g_live_arrow_mu); + g_live_arrow_tables.erase(LiveArrowKey{ conn, tableName }); + } + return env.Undefined(); +} + Napi::Object Init(Napi::Env env, Napi::Object exports) { // Export the functions exports.Set("Query", Napi::Function::New(env, QueryWrapper)); @@ -901,6 +1329,12 @@ Napi::Object Init(Napi::Env env, Napi::Object exports) { exports.Set("CloseConnection", Napi::Function::New(env, CloseConnectionWrapper)); exports.Set("QueryWithConnection", Napi::Function::New(env, QueryWithConnectionWrapper)); + // Arrow C Data Interface bindings. + exports.Set("ArrowRegisterArray", Napi::Function::New(env, ArrowRegisterArrayWrapper)); + exports.Set("ArrowRegisterStream", Napi::Function::New(env, ArrowRegisterStreamWrapper)); + exports.Set("ArrowRegisterColumns", Napi::Function::New(env, ArrowRegisterColumnsWrapper)); + exports.Set("ArrowUnregister", Napi::Function::New(env, ArrowUnregisterWrapper)); + // Most-reliable exit cleanup: close every live connection on env teardown, // ahead of (and complementary to) the std::atexit backstop. Both call the // idempotent hard_close_all, so running twice is harmless. diff --git a/package.json b/package.json index e863145..2c30fd5 100644 --- a/package.json +++ b/package.json @@ -4,6 +4,9 @@ "description": "chDB bindings for nodejs", "main": "index.js", "types": "index.d.ts", + "bin": { + "chdb-gen-types": "dist/layer3/codegen/cli.js" + }, "exports": { ".": { "types": "./index.d.ts", @@ -31,7 +34,7 @@ "build:platform": "bash ./scripts/build-platform-pkg.sh", "bench": "node bench/bench.js", "build": "node-gyp configure build --verbose && npm run fixloaderpath && npm run build:ts", - "build:ts": "tsc -p tsconfig.build.json", + "build:ts": "rm -rf dist && tsc -p tsconfig.build.json", "prepack": "npm run build:ts", "install": "node -e \"\"" }, diff --git a/src/errors.ts b/src/errors.ts index ee8e7e4..2f007c0 100644 --- a/src/errors.ts +++ b/src/errors.ts @@ -77,6 +77,21 @@ export class ChdbBindError extends ChdbError { readonly code = 'CHDB_BIND' } +/** + * A fluent-builder contract was violated. Most often this is a build-time fault — + * a SELECT with no source, an unsupported expression, an empty IN list, an + * invalid table/argument passed to a helper — raised while turning a chain into + * SQL, before any query runs. The builder also raises it for a few terminal-side + * contract violations that are the builder's own (not the engine's): e.g. + * `executeTakeFirstOrThrow` when the query returns no rows, or `registerArrowTable` + * rejecting malformed columns. Either way it is the builder's own error and never + * carries a ClickHouse code. Engine/runtime failures (syntax, connection, + * timeout, …) still surface as the existing classes above. + */ +export class ChdbCompileError extends ChdbError { + readonly code = 'CHDB_COMPILE' +} + /** Progress snapshot carried by streaming-insert callbacks and errors (payload-side ledger). */ export interface InsertProgress { /** Rows flushed to the engine so far (non-empty payload lines; exact for line-delimited formats). */ diff --git a/src/layer3/builder/delete.ts b/src/layer3/builder/delete.ts new file mode 100644 index 0000000..7957fc6 --- /dev/null +++ b/src/layer3/builder/delete.ts @@ -0,0 +1,43 @@ +/** + * DELETE chain: `deleteFrom(t).where(...).execute()`. Like UPDATE, this compiles + * to a ClickHouse mutation (`ALTER TABLE … DELETE WHERE …`), not an OLTP delete. + * A WHERE is required. + */ + +import type { ChdbResult } from '../../result' +import { compileQuery, type CompiledQuery } from '../compiler/compile' +import type { DeleteQueryNode } from '../compiler/nodes' +import { + buildPredicate, + combinePredicate, + type ExprInput, + type PredicateArgs, +} from './expression' +import { executeStatement, type ExecContext, type ExecuteOptions } from '../execute/terminal' + +export class DeleteQueryBuilder { + constructor( + private readonly ctx: ExecContext, + private readonly node: DeleteQueryNode, + ) { + Object.freeze(node) + } + + private derive(patch: Partial): DeleteQueryBuilder { + return new DeleteQueryBuilder(this.ctx, { ...this.node, ...patch }) + } + + where(lhs: ExprInput): DeleteQueryBuilder + where(lhs: ExprInput, op: string, rhs: unknown): DeleteQueryBuilder + where(...args: PredicateArgs): DeleteQueryBuilder { + return this.derive({ where: combinePredicate(this.node.where, buildPredicate(args), 'And') }) + } + + compile(): CompiledQuery { + return compileQuery(this.node) + } + + execute(opts?: ExecuteOptions): Promise { + return executeStatement(this.ctx, this.node, opts) + } +} diff --git a/src/layer3/builder/expression.ts b/src/layer3/builder/expression.ts new file mode 100644 index 0000000..fb0673c --- /dev/null +++ b/src/layer3/builder/expression.ts @@ -0,0 +1,122 @@ +/** + * Expression builders: the small set of helpers that turn user input into the + * node tree the compiler walks. + * + * - `sql` — a tagged template for any SQL the verbs don't cover. Interpolated + * values are BOUND (`{pN:Type}`), never spliced; use `sql.ref()` for an + * identifier and `sql.raw()` for trusted, author-controlled SQL text. + * - `ref(name)` — a column/table reference (quote-escaped). + * - `val(value, type?)` — an explicitly bound value (optionally typed). + * - `fn(name, args)` — a function call; string args are column references, + * `ChExpression` args are nested expressions, anything else is a bound value. + * + * `eb` bundles the same helpers for an ergonomic single import. + */ + +import { ChdbCompileError } from '../../errors' +import type { Expr } from '../compiler/nodes' + +/** A wrapped expression node, carrying `.as()` for aliasing. */ +export class ChExpression { + constructor(readonly node: Expr) {} + + /** `expr AS alias`. */ + as(alias: string): ChExpression { + return new ChExpression({ kind: 'Alias', node: this.node, alias }) + } +} + +/** Anything the builder verbs accept where an expression is expected. */ +export type ExprInput = string | ChExpression + +/** A column / table reference. */ +export function ref(name: string): ChExpression { + return new ChExpression({ kind: 'Reference', name }) +} + +/** An explicitly bound value, with an optional ClickHouse placeholder type. */ +export function val(value: unknown, chType?: string): ChExpression { + return new ChExpression({ kind: 'Value', value, chType }) +} + +/** A function call. String args are references; other primitives are bound. */ +export function fn(name: string, args: ReadonlyArray = []): ChExpression { + return new ChExpression({ + kind: 'Function', + name, + args: args.map((a) => { + if (a instanceof ChExpression) return a.node + if (typeof a === 'string') return { kind: 'Reference', name: a } + return { kind: 'Value', value: a } + }), + }) +} + +interface SqlTag { + (strings: TemplateStringsArray, ...values: unknown[]): ChExpression + /** A raw, author-controlled SQL fragment (NOT bound — never pass user input). */ + raw(text: string): ChExpression + /** A quote-escaped identifier. */ + ref(name: string): ChExpression + /** A bound value (optionally typed). */ + val(value: unknown, chType?: string): ChExpression +} + +const sqlTag = ((strings: TemplateStringsArray, ...values: unknown[]): ChExpression => { + const exprs: Expr[] = values.map((v) => + v instanceof ChExpression ? v.node : { kind: 'Value', value: v }, + ) + return new ChExpression({ kind: 'Raw', fragments: [...strings], values: exprs }) +}) as SqlTag + +sqlTag.raw = (text: string) => new ChExpression({ kind: 'Raw', fragments: [text], values: [] }) +sqlTag.ref = ref +sqlTag.val = val + +/** Tagged-template SQL escape hatch (see module docs). */ +export const sql: SqlTag = sqlTag + +/** Bundled expression helpers for a single import. */ +export const eb = { ref, val, fn, sql } + +/** Convert an expression-position input (string ⇒ reference, `*` ⇒ star). */ +export function toExpr(input: ExprInput): Expr { + if (input instanceof ChExpression) return input.node + if (typeof input !== 'string') { + throw new ChdbCompileError(`Expected a column name or expression, got ${typeof input}`) + } + if (input === '*') return { kind: 'Star' } + if (input.endsWith('.*')) return { kind: 'Star', table: input.slice(0, -2) } + const aliasMatch = /^(.+?)\s+as\s+(.+)$/i.exec(input) + if (aliasMatch) { + return { kind: 'Alias', node: { kind: 'Reference', name: aliasMatch[1]!.trim() }, alias: aliasMatch[2]!.trim() } + } + return { kind: 'Reference', name: input } +} + +/** Convert a value-position input: an expression wrapper stays an expression, + * anything else becomes a bound value. */ +export function toValue(input: unknown): Expr { + if (input instanceof ChExpression) return input.node + return { kind: 'Value', value: input } +} + +/** A where/having argument list: `(lhs, op, rhs)` or a single expression. */ +export type PredicateArgs = [ExprInput] | [ExprInput, string, unknown] + +/** Turn a predicate argument list into one expression node. */ +export function buildPredicate(args: PredicateArgs): Expr { + if (args.length === 1) return toExpr(args[0]) + return { kind: 'Binary', left: toExpr(args[0]), op: args[1], right: toValue(args[2]) } +} + +/** Combine an existing predicate with a new one under AND / OR, flattening. */ +export function combinePredicate( + existing: Expr | undefined, + next: Expr, + kind: 'And' | 'Or', +): Expr { + if (existing === undefined) return next + if (existing.kind === kind) return { kind, items: [...existing.items, next] } + return { kind, items: [existing, next] } +} diff --git a/src/layer3/builder/functions.ts b/src/layer3/builder/functions.ts new file mode 100644 index 0000000..a1920c9 --- /dev/null +++ b/src/layer3/builder/functions.ts @@ -0,0 +1,83 @@ +/** + * `chFn` — typed wrappers for the ClickHouse-specific functions an analyst + * reaches for most. The 190+ other functions stay reachable through the generic + * `fn(name, args)` (you supply the return type) or `sql\`...\``; these 8 are the + * common ones worth a named helper. + * + * String args are column references, `ChExpression` args are nested expressions, + * and any other value is bound — the same rule as `fn()`. Parametric aggregates + * (`name(params)(args)`) are assembled so their parameters bind too. + */ + +import { ChExpression, fn, val, type ExprInput } from './expression' +import { validateFunctionName } from '../compiler/identifier' +import type { Expr } from '../compiler/nodes' + +function toArg(input: ExprInput | unknown): Expr { + if (input instanceof ChExpression) return input.node + if (typeof input === 'string') return { kind: 'Reference', name: input } + return { kind: 'Value', value: input } +} + +/** + * Build a parametric aggregate `name(params)(args)` as a raw node whose params + * and args are still compiled (so values among them bind). Function name is + * validated eagerly. + */ +function parametric( + name: string, + params: ReadonlyArray, + args: ReadonlyArray, +): ChExpression { + validateFunctionName(name) + const values = [...params.map(toArg), ...args.map(toArg)] + const fragments: string[] = [`${name}(`] + for (let i = 1; i < params.length; i++) fragments.push(', ') + fragments.push(')(') + for (let i = 1; i < args.length; i++) fragments.push(', ') + fragments.push(')') + return new ChExpression({ kind: 'Raw', fragments, values }) +} + +export const chFn = { + /** `argMax(arg, val)` — the `arg` value at the row maximizing `val`. */ + argMax(arg: ExprInput, val: ExprInput): ChExpression { + return fn('argMax', [arg, val]) + }, + + /** `argMin(arg, val)` — the `arg` value at the row minimizing `val`. */ + argMin(arg: ExprInput, val: ExprInput): ChExpression { + return fn('argMin', [arg, val]) + }, + + /** `uniqExact(cols…)` — exact distinct count. */ + uniqExact(...columns: ExprInput[]): ChExpression { + return fn('uniqExact', columns) + }, + + /** `retention(conds…)` — retention flags across the given conditions. */ + retention(...conditions: ExprInput[]): ChExpression { + return fn('retention', conditions) + }, + + /** `topK(k)(column)` — the k most frequent values. */ + topK(k: number, column: ExprInput): ChExpression { + return parametric('topK', [k], [column]) + }, + + /** `quantileTDigest(level)(column)` — approximate quantile via t-digest. */ + quantileTDigest(level: number, column: ExprInput): ChExpression { + return parametric('quantileTDigest', [level], [column]) + }, + + /** `windowFunnel(window)(timestamp, conds…)` — funnel analysis. */ + windowFunnel(window: number, timestamp: ExprInput, ...conditions: ExprInput[]): ChExpression { + return parametric('windowFunnel', [window], [timestamp, ...conditions]) + }, + + /** `sequenceMatch(pattern)(timestamp, conds…)` — event-sequence matching. */ + sequenceMatch(pattern: string, timestamp: ExprInput, ...conditions: ExprInput[]): ChExpression { + // The pattern is a bound value, not a column reference, so wrap it in val(). + return parametric('sequenceMatch', [val(pattern)], [timestamp, ...conditions]) + }, +} diff --git a/src/layer3/builder/insert.ts b/src/layer3/builder/insert.ts new file mode 100644 index 0000000..033615a --- /dev/null +++ b/src/layer3/builder/insert.ts @@ -0,0 +1,87 @@ +/** + * INSERT chain: `insertInto(table).values(source).execute()`. The `source` is + * polymorphic (a single verb extended by its argument type, not several verbs): + * + * - an object/array-of-values row array → forwarded to Layer 1's `insert` + * (the multithreaded engine parser does the work; no SQL is built here); + * - a SelectQueryBuilder → compiled to `INSERT INTO … SELECT` and run. + * + * Arrow / TypedArray / stream sources are handled in a later step (the Arrow IPC + * input path); this is the row-array and subquery foundation. + */ + +import { ChdbCompileError } from '../../errors' +import type { ChdbResult } from '../../result' +import type { CompiledQuery } from '../compiler/compile' +import { compileQuery } from '../compiler/compile' +import type { InsertSelectNode } from '../compiler/nodes' +import { runtime, type RuntimeInsertParams, type RuntimeInsertSummary } from '../runtime' +import { executeStatement, type ExecContext, type ExecuteOptions } from '../execute/terminal' +import { SelectQueryBuilder } from './select' + +export type InsertRow = Record | ReadonlyArray +export type InsertColumns = ReadonlyArray | { except: ReadonlyArray } + +/** Terminal for a row-array insert (forwards to Layer 1 `insert`). */ +export class InsertValuesExecutable { + constructor( + private readonly ctx: ExecContext, + private readonly table: string, + private readonly rows: ReadonlyArray, + private readonly cols?: InsertColumns, + ) {} + + execute(): Promise { + const params: RuntimeInsertParams = { table: this.table, values: this.rows, columns: this.cols } + return this.ctx.session ? this.ctx.session.insert(params) : runtime().insert(params) + } +} + +/** Terminal for `INSERT INTO … SELECT` (compiles to SQL like the SELECT path). */ +export class InsertSelectExecutable { + constructor( + private readonly ctx: ExecContext, + private readonly node: InsertSelectNode, + ) {} + + /** Emit `{ sql, parameters }` without running anything. */ + compile(): CompiledQuery { + return compileQuery(this.node) + } + + execute(opts?: ExecuteOptions): Promise { + return executeStatement(this.ctx, this.node, opts) + } +} + +export class InsertQueryBuilder { + constructor( + private readonly ctx: ExecContext, + private readonly table: string, + private readonly cols?: ReadonlyArray, + ) {} + + /** Restrict the target columns: `INSERT INTO t (a, b) …`. */ + columns(columns: ReadonlyArray): InsertQueryBuilder { + return new InsertQueryBuilder(this.ctx, this.table, columns) + } + + /** Insert object/array rows (forwarded to Layer 1's engine-parsed insert). */ + values(rows: ReadonlyArray): InsertValuesExecutable + /** Insert the result of a SELECT (`INSERT INTO … SELECT`). */ + values(query: SelectQueryBuilder): InsertSelectExecutable + values(source: ReadonlyArray | SelectQueryBuilder): InsertValuesExecutable | InsertSelectExecutable { + if (source instanceof SelectQueryBuilder) { + return new InsertSelectExecutable(this.ctx, { + kind: 'InsertSelect', + table: this.table, + columns: this.cols, + select: source.toNode(), + }) + } + if (!Array.isArray(source)) { + throw new ChdbCompileError('values() expects a row array or a SELECT query builder') + } + return new InsertValuesExecutable(this.ctx, this.table, source, this.cols) + } +} diff --git a/src/layer3/builder/select.ts b/src/layer3/builder/select.ts new file mode 100644 index 0000000..d30540b --- /dev/null +++ b/src/layer3/builder/select.ts @@ -0,0 +1,285 @@ +/** + * SelectQueryBuilder — the immutable SELECT chain. Every method returns a NEW + * builder over a NEW frozen node, so `const q2 = q1.where(...)` derives from `q1` + * without mutating it and an unexecuted query can be passed around and composed. + * + * Verb names and clause order copy the Kysely v0.29.x shape (subject-first: + * `selectFrom().where().groupBy().orderBy().limit()`); the runtime is + * chDB's own, not Kysely's. + */ + +import { ChdbCompileError } from '../../errors' +import { compileQuery, type CompiledQuery } from '../compiler/compile' +import type { + Expr, + JoinKind, + OrderByItem, + SelectQueryNode, + SetOperator, +} from '../compiler/nodes' +import { ChExpression, toExpr, toValue, type ExprInput } from './expression' +import { + executeSelect, + type ExecContext, + type ExecuteOptions, +} from '../execute/terminal' + +/** Combine an existing predicate with a new one under AND / OR, flattening. */ +function combine(existing: Expr | undefined, next: Expr, kind: 'And' | 'Or'): Expr { + if (existing === undefined) return next + if (existing.kind === kind) return { kind, items: [...existing.items, next] } + return { kind, items: [existing, next] } +} + +/** One element of a where/having call: either `(lhs, op, rhs)` or a single expression. */ +type ColumnInput = ExprInput + +export class SelectQueryBuilder> { + // A phantom type marker so callers can recover the row type with + // `RowOf` — TypeScript's ReturnType picks .execute's last overload + // (the non-JSON fallback returning `unknown`), which loses the typed row. + declare readonly _row: O + + constructor( + private readonly ctx: ExecContext, + private readonly node: SelectQueryNode, + ) { + Object.freeze(node) + } + + private derive(patch: Partial): SelectQueryBuilder { + return new SelectQueryBuilder(this.ctx, { ...this.node, ...patch }) + } + + // ---- projection ----------------------------------------------------------- + + /** Add selections (accumulates across calls). Each is a column / expression. */ + select(selection: ColumnInput | ReadonlyArray): SelectQueryBuilder { + const incoming = (Array.isArray(selection) ? selection : [selection]).map(toExpr) + return this.derive({ selections: [...(this.node.selections ?? []), ...incoming] }) + } + + /** Select everything (`SELECT *`). */ + selectAll(): SelectQueryBuilder { + return this.derive({ selections: [{ kind: 'Star' }] }) + } + + distinct(): SelectQueryBuilder { + return this.derive({ distinct: true }) + } + + // ---- filtering ------------------------------------------------------------ + + private predicate(args: [ColumnInput] | [ColumnInput, string, unknown]): Expr { + if (args.length === 1) return toExpr(args[0]) + const [lhs, op, rhs] = args + return { kind: 'Binary', left: toExpr(lhs), op, right: toValue(rhs) } + } + + /** AND a predicate. `(col, op, value)` or a single expression. */ + where(lhs: ColumnInput): SelectQueryBuilder + where(lhs: ColumnInput, op: string, rhs: unknown): SelectQueryBuilder + where(...args: [ColumnInput] | [ColumnInput, string, unknown]): SelectQueryBuilder { + return this.derive({ where: combine(this.node.where, this.predicate(args), 'And') }) + } + + /** Alias for {@link where} (explicit AND). */ + andWhere(lhs: ColumnInput): SelectQueryBuilder + andWhere(lhs: ColumnInput, op: string, rhs: unknown): SelectQueryBuilder + andWhere(...args: [ColumnInput] | [ColumnInput, string, unknown]): SelectQueryBuilder { + return this.derive({ where: combine(this.node.where, this.predicate(args), 'And') }) + } + + /** OR a predicate into the WHERE clause. */ + orWhere(lhs: ColumnInput): SelectQueryBuilder + orWhere(lhs: ColumnInput, op: string, rhs: unknown): SelectQueryBuilder + orWhere(...args: [ColumnInput] | [ColumnInput, string, unknown]): SelectQueryBuilder { + return this.derive({ where: combine(this.node.where, this.predicate(args), 'Or') }) + } + + having(lhs: ColumnInput): SelectQueryBuilder + having(lhs: ColumnInput, op: string, rhs: unknown): SelectQueryBuilder + having(...args: [ColumnInput] | [ColumnInput, string, unknown]): SelectQueryBuilder { + return this.derive({ having: combine(this.node.having, this.predicate(args), 'And') }) + } + + // ---- grouping / ordering / paging ---------------------------------------- + + groupBy(columns: ColumnInput | ReadonlyArray): SelectQueryBuilder { + const incoming = (Array.isArray(columns) ? columns : [columns]).map(toExpr) + return this.derive({ groupBy: [...(this.node.groupBy ?? []), ...incoming] }) + } + + orderBy(column: ColumnInput, direction: 'asc' | 'desc' = 'asc'): SelectQueryBuilder { + const item: OrderByItem = { expr: toExpr(column), direction } + return this.derive({ orderBy: [...(this.node.orderBy ?? []), item] }) + } + + limit(count: number): SelectQueryBuilder { + return this.derive({ limit: count }) + } + + offset(count: number): SelectQueryBuilder { + return this.derive({ offset: count }) + } + + // ---- joins ---------------------------------------------------------------- + + private join( + joinType: JoinKind, + source: ExprInput, + leftKey?: string, + rightKey?: string, + ): SelectQueryBuilder { + const on = + leftKey !== undefined && rightKey !== undefined + ? ({ + kind: 'Binary', + left: { kind: 'Reference', name: leftKey }, + op: '=', + right: { kind: 'Reference', name: rightKey }, + } as Expr) + : undefined + return this.derive({ + joins: [...(this.node.joins ?? []), { kind: 'Join', joinType, source: toExpr(source), on }], + }) + } + + innerJoin(source: ExprInput, leftKey: string, rightKey: string): SelectQueryBuilder { + return this.join('Inner', source, leftKey, rightKey) + } + + leftJoin(source: ExprInput, leftKey: string, rightKey: string): SelectQueryBuilder { + return this.join('Left', source, leftKey, rightKey) + } + + fullJoin(source: ExprInput, leftKey: string, rightKey: string): SelectQueryBuilder { + return this.join('Full', source, leftKey, rightKey) + } + + crossJoin(source: ExprInput): SelectQueryBuilder { + return this.join('Cross', source) + } + + // ---- ClickHouse dialect sugar -------------------------------------------- + + /** `FROM t FINAL` — merge parts on read (ReplacingMergeTree, etc.). */ + final(): SelectQueryBuilder { + return this.derive({ final: true }) + } + + /** `SAMPLE rate` — approximate query over a fraction (or absolute count) of rows. */ + sample(rate: number): SelectQueryBuilder { + return this.derive({ sample: rate }) + } + + /** + * `PREWHERE` — ClickHouse reads these columns first and filters before the + * rest of the row is read. Same argument shapes as {@link where}; multiple + * calls are AND-combined. + */ + prewhere(lhs: ColumnInput): SelectQueryBuilder + prewhere(lhs: ColumnInput, op: string, rhs: unknown): SelectQueryBuilder + prewhere(...args: [ColumnInput] | [ColumnInput, string, unknown]): SelectQueryBuilder { + return this.derive({ prewhere: combine(this.node.prewhere, this.predicate(args), 'And') }) + } + + /** Append a trailing `SETTINGS k = v, …` clause (merges across calls). */ + settings(settings: Record): SelectQueryBuilder { + return this.derive({ settings: { ...this.node.settings, ...settings } }) + } + + /** + * `FORMAT x` — the SQL-level output format (changes how ClickHouse serializes + * the result). This is distinct from `.execute({ format })`, which picks the + * view over the bytes (`Row[]` / Arrow `Table` / raw). Use this only when you + * read the raw result yourself. + */ + format(name: string): SelectQueryBuilder { + return this.derive({ format: name }) + } + + /** + * `LIMIT n BY (cols)` — keep the first n rows per distinct value of the + * columns. Independent of (and combinable with) the trailing `LIMIT`. + */ + limitBy(count: number, columns: ColumnInput | ReadonlyArray): SelectQueryBuilder { + const cols = (Array.isArray(columns) ? columns : [columns]).map(toExpr) + return this.derive({ limitBy: { count, columns: cols } }) + } + + // ---- set operations ------------------------------------------------------- + + private setOp(operator: SetOperator, other: SelectQueryBuilder): SelectQueryBuilder { + return this.derive({ + setOps: [...(this.node.setOps ?? []), { operator, query: other.toNode() }], + }) + } + + union(other: SelectQueryBuilder): SelectQueryBuilder { + return this.setOp('union', other) + } + + unionAll(other: SelectQueryBuilder): SelectQueryBuilder { + return this.setOp('unionAll', other) + } + + intersect(other: SelectQueryBuilder): SelectQueryBuilder { + return this.setOp('intersect', other) + } + + except(other: SelectQueryBuilder): SelectQueryBuilder { + return this.setOp('except', other) + } + + // ---- composition ---------------------------------------------------------- + + /** The underlying node (for use as a subquery / set-operation operand). */ + toNode(): SelectQueryNode { + return this.node + } + + /** Wrap this query as an aliased subquery usable as a source or in a join. */ + as(alias: string): ChExpression { + return new ChExpression({ + kind: 'Alias', + node: { kind: 'Subquery', query: this.node }, + alias, + }) + } + + // ---- terminals ------------------------------------------------------------ + + /** Emit `{ sql, parameters }` without running anything (debug / audit / tools). */ + compile(): CompiledQuery { + return compileQuery(this.node) + } + + /** Run and return rows (default), an Arrow Table (`{format:'arrow'}`), or a raw result. */ + execute(opts?: ExecuteOptions & { format?: undefined | 'json' }): Promise + execute(opts: ExecuteOptions & { format: 'arrow' }): Promise + execute(opts: ExecuteOptions & { format: string }): Promise + execute(opts?: ExecuteOptions): Promise { + return executeSelect(this.ctx, this.node, opts) + } + + /** + * Run and return the first row, or `undefined` when there are none. Only the + * row (JSON) view makes sense here: `format:'arrow'`/raw return a Table or + * ChdbResult, not an array, so they're excluded at the type level rather than + * silently yielding a malformed `rows[0]`. + */ + async executeTakeFirst(opts?: Omit & { format?: 'json' }): Promise { + const rows = (await executeSelect(this.ctx, this.node, opts)) as O[] + return rows[0] + } + + /** Run and return the first row, throwing if there are none. */ + async executeTakeFirstOrThrow(opts?: Omit & { format?: 'json' }): Promise { + const first = await this.executeTakeFirst(opts) + if (first === undefined) { + throw new ChdbCompileError('executeTakeFirstOrThrow: the query returned no rows') + } + return first + } +} diff --git a/src/layer3/builder/table-functions.ts b/src/layer3/builder/table-functions.ts new file mode 100644 index 0000000..d87414a --- /dev/null +++ b/src/layer3/builder/table-functions.ts @@ -0,0 +1,196 @@ +/** + * `chTable` — typed helpers for the ClickHouse table functions used most often + * as federation / file sources. Each returns an expression you alias and pass to + * `selectFrom`: + * + * selectFrom(chTable.s3({ url, format: 'Parquet' }).as('t')).selectAll() + * + * Every argument is bound server-side (`{pN:Type}`) exactly like a value in a + * WHERE clause — a URL or path is data, never spliced into the SQL — and chDB + * accepts bound parameters in table-function argument position, so this keeps + * the zero-interpolation rule intact for sources too. The 8 most-used functions + * are here; the long tail stays reachable via `sql.raw(...)`. + */ + +import { ChExpression } from './expression' +import type { Expr } from '../compiler/nodes' +import { ChdbCompileError } from '../../errors' + +/** Build a table-function expression with bound arguments. */ +function tableFn(name: string, args: ReadonlyArray<{ value: unknown; type?: string }>): ChExpression { + const argNodes: Expr[] = args.map((a) => ({ kind: 'Value', value: a.value, chType: a.type })) + return new ChExpression({ kind: 'Function', name, args: argNodes }) +} + +/** Drop trailing undefined optional args (ClickHouse positional signatures). */ +function compact( + args: ReadonlyArray<{ value: unknown; type?: string } | undefined>, +): { value: unknown; type?: string }[] { + const out: { value: unknown; type?: string }[] = [] + for (const a of args) { + if (a === undefined) break + out.push(a) + } + return out +} + +const str = (value: string) => ({ value, type: 'String' }) + +/** + * Resolve a both-or-neither credential pair into bound args. Supplying exactly + * one half (e.g. `accessKeyId` without `secretAccessKey`) is almost always a + * config mistake that would otherwise be silently dropped, producing an + * unauthenticated query — so reject it instead. + */ +function credPair( + fn: string, + aName: string, + aValue: string | undefined, + bName: string, + bValue: string | undefined, +): { value: unknown; type?: string }[] { + if ((aValue === undefined) !== (bValue === undefined)) { + const have = aValue !== undefined ? aName : bName + const missing = aValue !== undefined ? bName : aName + throw new ChdbCompileError(`chTable.${fn}: ${have} was provided without ${missing}; supply both or neither`) + } + return aValue !== undefined && bValue !== undefined ? [str(aValue), str(bValue)] : [] +} + +export interface S3Options { + url: string + format?: string + structure?: string + accessKeyId?: string + secretAccessKey?: string +} + +export interface FileOptions { + path: string + format?: string + structure?: string +} + +export interface UrlOptions { + url: string + format?: string + structure?: string +} + +export interface PgOptions { + /** `host:port`. */ + host: string + database: string + table: string + user?: string + password?: string + schema?: string +} + +export interface MySqlOptions { + host: string + database: string + table: string + user?: string + password?: string +} + +export interface LakeOptions { + url: string + accessKeyId?: string + secretAccessKey?: string +} + +export const chTable = { + /** `numbers(count)` or `numbers(start, count)` — generated integer source. */ + numbers(countOrStart: number | bigint, count?: number | bigint): ChExpression { + const args = + count === undefined + ? [{ value: countOrStart, type: 'UInt64' }] + : [ + { value: countOrStart, type: 'UInt64' }, + { value: count, type: 'UInt64' }, + ] + return tableFn('numbers', args) + }, + + /** `s3(url[, access_key, secret], [format], [structure])`. */ + s3(opts: S3Options): ChExpression { + const creds = credPair('s3', 'accessKeyId', opts.accessKeyId, 'secretAccessKey', opts.secretAccessKey) + return tableFn( + 's3', + compact([ + str(opts.url), + ...creds, + opts.format !== undefined ? str(opts.format) : undefined, + opts.structure !== undefined ? str(opts.structure) : undefined, + ]), + ) + }, + + /** `file(path[, format][, structure])`. */ + file(opts: FileOptions): ChExpression { + return tableFn( + 'file', + compact([ + str(opts.path), + opts.format !== undefined ? str(opts.format) : undefined, + opts.structure !== undefined ? str(opts.structure) : undefined, + ]), + ) + }, + + /** `url(url[, format][, structure])`. */ + url(opts: UrlOptions): ChExpression { + return tableFn( + 'url', + compact([ + str(opts.url), + opts.format !== undefined ? str(opts.format) : undefined, + opts.structure !== undefined ? str(opts.structure) : undefined, + ]), + ) + }, + + /** `postgresql(host:port, database, table[, user, password][, schema])`. */ + postgresql(opts: PgOptions): ChExpression { + const auth = credPair('postgresql', 'user', opts.user, 'password', opts.password) + return tableFn( + 'postgresql', + compact([ + str(opts.host), + str(opts.database), + str(opts.table), + ...auth, + opts.schema !== undefined ? str(opts.schema) : undefined, + ]), + ) + }, + + /** `mysql(host:port, database, table, user, password)`. */ + mysql(opts: MySqlOptions): ChExpression { + const auth = credPair('mysql', 'user', opts.user, 'password', opts.password) + return tableFn('mysql', compact([str(opts.host), str(opts.database), str(opts.table), ...auth])) + }, + + /** `iceberg(url[, access_key, secret])`. */ + iceberg(opts: LakeOptions): ChExpression { + const creds = credPair('iceberg', 'accessKeyId', opts.accessKeyId, 'secretAccessKey', opts.secretAccessKey) + return tableFn('iceberg', compact([str(opts.url), ...creds])) + }, + + /** + * `arrowstream('')` — read a JS dataset registered via + * `registerArrowTable(name, columns)`. The argument is the table name; the + * engine reads it from the in-process Arrow registry with no IPC copy. + */ + arrowstream(name: string): ChExpression { + return tableFn('arrowstream', [{ value: name, type: 'String' }]) + }, + + /** `deltaLake(url[, access_key, secret])`. */ + deltaLake(opts: LakeOptions): ChExpression { + const creds = credPair('deltaLake', 'accessKeyId', opts.accessKeyId, 'secretAccessKey', opts.secretAccessKey) + return tableFn('deltaLake', compact([str(opts.url), ...creds])) + }, +} diff --git a/src/layer3/builder/update.ts b/src/layer3/builder/update.ts new file mode 100644 index 0000000..af83f0f --- /dev/null +++ b/src/layer3/builder/update.ts @@ -0,0 +1,58 @@ +/** + * UPDATE chain: `updateTable(t).set({...}).where(...).execute()`. ClickHouse has + * no row-level UPDATE statement, so this compiles to an `ALTER TABLE … UPDATE …` + * mutation — documented as mutation semantics (asynchronous, heavyweight), not + * an OLTP point update. A WHERE is required (the compiler refuses an unguarded + * whole-table mutation). + */ + +import type { ChdbResult } from '../../result' +import { compileQuery, type CompiledQuery } from '../compiler/compile' +import type { Expr, UpdateQueryNode } from '../compiler/nodes' +import { + buildPredicate, + combinePredicate, + toValue, + type ExprInput, + type PredicateArgs, +} from './expression' +import { executeStatement, type ExecContext, type ExecuteOptions } from '../execute/terminal' + +export class UpdateQueryBuilder { + constructor( + private readonly ctx: ExecContext, + private readonly node: UpdateQueryNode, + ) { + Object.freeze(node) + } + + private derive(patch: Partial): UpdateQueryBuilder { + return new UpdateQueryBuilder(this.ctx, { ...this.node, ...patch }) + } + + /** + * Column assignments. Values are bound (`{pN:Type}`) unless wrapped as an + * expression (e.g. `sql\`col + 1\``). Accumulates across calls. + */ + set(assignments: Record): UpdateQueryBuilder { + const incoming = Object.keys(assignments).map((column) => ({ + column, + value: toValue(assignments[column]) as Expr, + })) + return this.derive({ assignments: [...this.node.assignments, ...incoming] }) + } + + where(lhs: ExprInput): UpdateQueryBuilder + where(lhs: ExprInput, op: string, rhs: unknown): UpdateQueryBuilder + where(...args: PredicateArgs): UpdateQueryBuilder { + return this.derive({ where: combinePredicate(this.node.where, buildPredicate(args), 'And') }) + } + + compile(): CompiledQuery { + return compileQuery(this.node) + } + + execute(opts?: ExecuteOptions): Promise { + return executeStatement(this.ctx, this.node, opts) + } +} diff --git a/src/layer3/codegen/cli.ts b/src/layer3/codegen/cli.ts new file mode 100644 index 0000000..b4e810e --- /dev/null +++ b/src/layer3/codegen/cli.ts @@ -0,0 +1,226 @@ +#!/usr/bin/env node +/** + * `chdb gen-types` — introspect a data source and write a TypeScript Database + * interface (column types as ClickHouse string literals) ready to feed into + * `chdb.database()`. The CLI is a thin wrapper over `introspectDatabase` + * and `emitDatabase`; everything programmatic lives in those two modules. + * + * Source modes: + * --from-file DESCRIBE a local file via the file() table function + * --from-table DESCRIBE a local table (the default connection) + * --from-url Open via connect() and DESCRIBE + * + * Common options: + * --table Repeatable. Server sources (--from-url postgres://..., + * mysql://, clickhouse://) need at least one. Each + * becomes a property on the emitted interface. + * --format File format (auto-detected if omitted) + * --structure File structure (auto-detected if omitted) + * --name Emitted interface name (default: Database) + * --out Write to file (default: stdout) + * --label