Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 19 additions & 1 deletion docs/arrow.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ choose the name. Names are process-wide; prefer generated names unless you have
a reason to coordinate them. Close the registration after the last scan. If a
query or lazy stream is active, unregister and buffer release are deferred
until that scan finishes or is cancelled.
An Arrow `Table` with a non-empty schema and no batches is a valid zero-row
source; `insertArrow` likewise accepts it as a zero-row insert.

## Insert Arrow data

Expand Down Expand Up @@ -86,6 +88,11 @@ The native queue holds at most two batches not yet handed to
core; producing another waits for capacity. Core can retain delivered batches,
so this is not a bound on the entire query's memory. A cancelled producer
should cooperate with iterator `return()` to release its own resources.
Pass `{ session, signal }` to `registerArrowReader` or `insertArrowReader` to
cancel the native reader, including while waiting for its first batch. An
upstream iterator blocked inside `next()` cannot be forcibly interrupted;
the caller must arrange for it to cooperate with `return()` or its own signal.
An insert cancelled after it starts is not promised to roll back committed rows.
Use `queryAsync` or another async query API while a JS Arrow reader is
registered: synchronous `query`/`queryBind` on its connection are rejected
because blocking the event loop would prevent the producer from refilling
Expand All @@ -101,7 +108,9 @@ try {
const small = await session.queryArrow('SELECT toInt32(number) AS id FROM numbers(3)')
console.log(small.toArray().map(row => row.id)) // [0, 1, 2]

const reader = session.queryArrowStream('SELECT toInt32(number) AS id FROM numbers(1000)')
const controller = new AbortController()
const reader = session.queryArrowStream('SELECT toInt32(number) AS id FROM numbers(1000)',
{ signal: controller.signal })
try {
for await (const batch of reader) {
console.log(batch.numRows)
Expand All @@ -118,6 +127,11 @@ try {
Use `queryArrowBind` and `queryArrowStreamBind` for server-side
`{name:Type}` parameters, for example
`await session.queryArrowBind('SELECT {id:Int32} AS id', { id: 7 })`.
All four direct output methods accept `signal`. A buffered `queryArrow` settles
early when aborted, but its native computation may still run to completion;
the Session defers teardown until it finishes. A streamed reader cancels its
native handle after any in-flight fetch completes. Neither behavior promises
transaction rollback.
For TypeScript return types without making the base `chdb` import require
`apache-arrow`, use the helpers from `chdb/arrow`.
Only one iterator may consume a given Arrow result reader at a time; cancel
Expand All @@ -135,6 +149,10 @@ the JS event loop.
Timestamp, Decimal128/256, slices, null bitmaps, and a single-chunk
dictionary of UTF-8/Binary values with Int32/UInt32 indices. Nested input
and multi-chunk dictionaries are rejected rather than silently converted.
- Direct input accepts Arrow objects and batch iterables, not an Arrow IPC
byte stream from a file, `fetch()` response, or Node `Readable`. Decode IPC
to batches with `apache-arrow` before using `registerArrowReader`; the
existing Arrow IPC output path is separate.
- Supported output includes basic scalars, Timestamp, Decimal128/256, and
LowCardinality UTF-8/Binary dictionaries. Options include
`stringAsString`, `lowCardinalityAsDictionary`, and `unsupportedAsBinary`;
Expand Down
2 changes: 2 additions & 0 deletions index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -249,6 +249,8 @@ export interface ChdbArrowReader extends AsyncIterable<unknown> {
}

export interface DirectArrowOptions {
/** Abort the caller's wait; an already-running buffered query finishes natively before teardown. */
signal?: AbortSignal;
/** Degrade otherwise unmappable engine types to Arrow Binary. Default false. */
unsupportedAsBinary?: boolean;
/** Export ClickHouse String as Arrow Utf8 (default) or Binary. */
Expand Down
50 changes: 45 additions & 5 deletions index.js
Comment thread
macroscopeapp[bot] marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
Expand Up @@ -246,6 +246,28 @@ function withAbortTimeout(nativePromise, opts) {
});
}

// Direct Arrow returns an apache-arrow Table rather than a ChdbResult. Match
// queryAsync's early-settle abort without wrapping the typed value or releasing
// the connection while its native worker is still running.
function withArrowAbort(promise, signal) {
trackNative(promise);
if (!signal) return promise;
return new Promise((resolve, reject) => {
let settled = false;
const finish = (fn, value) => {
if (settled) return;
settled = true;
signal.removeEventListener('abort', onAbort);
fn(value);
};
const onAbort = () => finish(reject, new ChdbAbortError(
'Arrow query aborted (the underlying computation may still run to completion in the background)'));
if (signal.aborted) onAbort();
else signal.addEventListener('abort', onAbort, { once: true });
promise.then((value) => finish(resolve, value), (error) => finish(reject, error));
});
}

// Parameterized queries set+reset parameter state around each call
// (chdb-core's CApiQueryParameterGuard). With the bundled libchdb this set/reset
// is NOT isolated per connection under real concurrency: two parameterized
Expand Down Expand Up @@ -458,20 +480,27 @@ class ChdbQueryStream {
// Typed C Data stream. Each native fetch produces at most one RecordBatch;
// bytes are copied and C callbacks released before the JS batch is exposed.
class ChdbArrowReader {
constructor(handle) {
constructor(handle, signal) {
this._handle = handle;
this._closed = false;
this._inflight = null;
this._iterating = false;
this._signal = signal;
this._onAbort = signal ? () => this.cancel() : null;
if (signal) signal.addEventListener('abort', this._onAbort, { once: true });
this.schema = null; // available after the first fetch, even for zero rows
activeStreams.add(this);
}
get closed() { return this._closed; }
_removeAbortListener() {
if (this._signal) this._signal.removeEventListener('abort', this._onAbort);
}

async *[Symbol.asyncIterator]() {
if (this._iterating) throw new ChdbStreamError("Arrow reader already has an active iterator");
this._iterating = true;
try {
if (this._signal && this._signal.aborted) throw new ChdbAbortError('Arrow stream aborted');
while (!this._closed) {
let raw;
try {
Expand All @@ -480,27 +509,36 @@ class ChdbArrowReader {
} catch (e) {
const cancelling = this._closed;
this._closed = true;
this._removeAbortListener();
if (!cancelling) {
activeStreams.delete(this);
flushArrowIdle();
}
if (this._signal && this._signal.aborted) throw new ChdbAbortError('Arrow stream aborted', { cause: e });
throw asQueryError(e);
} finally {
this._inflight = null;
}
if (this._signal && this._signal.aborted) throw new ChdbAbortError('Arrow stream aborted');
if (this._closed) break;
const table = arrowTableFromDirect(raw);
this.schema = table.schema;
if (raw.done) {
const cancelling = this._closed;
this._closed = true;
this._removeAbortListener();
if (!cancelling) {
activeStreams.delete(this);
flushArrowIdle();
}
break;
}
for (const batch of table.batches) yield batch;
for (const batch of table.batches) {
yield batch;
// An abort while the caller holds a batch closes this reader. When
// iteration resumes, report truncation instead of silently ending.
if (this._signal && this._signal.aborted) throw new ChdbAbortError('Arrow stream aborted');
}
}
} finally {
this._iterating = false;
Expand All @@ -511,6 +549,7 @@ class ChdbArrowReader {
cancel() {
if (this._closed) return;
this._closed = true;
this._removeAbortListener();
const destroy = () => {
try { chdbNode.CancelArrowStream(this._handle); } catch (_) { /* best effort */ }
activeStreams.delete(this);
Expand Down Expand Up @@ -896,12 +935,12 @@ class Session {
queryArrow(query, opts = {}) {
if (!this.connection) return Promise.reject(new ChdbClosedError("No active connection available"));
if (!query) return Promise.reject(new ChdbStreamError("queryArrow requires a non-empty query"));
if (opts.signal && opts.signal.aborted) return Promise.reject(new ChdbAbortError('Arrow query aborted'));
try { _assertArrowCore(); } catch (e) { return Promise.reject(e); }
let native;
try { native = chdbNode.QueryArrowAsync(this.connection, query, opts); }
catch (e) { return Promise.reject(e); }
trackNative(native);
return native.then((raw) => arrowTableFromDirect(raw), (e) => { throw asQueryError(e); });
return withArrowAbort(native.then((raw) => arrowTableFromDirect(raw), (e) => { throw asQueryError(e); }), opts.signal);
}

queryArrowStream(query, opts = {}) {
Expand All @@ -923,14 +962,15 @@ class Session {
#startArrowStream(query, opts, bound) {
if (!this.connection) throw new ChdbClosedError("No active connection available");
if (!query) throw new ChdbStreamError("queryArrowStream requires a non-empty query");
if (opts.signal && opts.signal.aborted) throw new ChdbAbortError('Arrow stream aborted');
_assertArrowCore();
if (this._activeStream && !this._activeStream.closed) {
throw new ChdbStreamError("a stream is already active on this session; finish or cancel it first");
}
let handle;
try { handle = chdbNode.StartArrowStream(this.connection, query, opts, bound); }
catch (e) { throw asQueryError(e); }
const stream = new ChdbArrowReader(handle);
const stream = new ChdbArrowReader(handle, opts.signal);
this._activeStream = stream;
return stream;
}
Expand Down
24 changes: 23 additions & 1 deletion src/layer3/execute/arrow-input.ts
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,27 @@ export interface ArrowRecordBatchInput {
readonly numRows: number
readonly numCols: number
}
export interface ArrowTableInput { readonly batches: ReadonlyArray<ArrowRecordBatchInput> }
export interface ArrowTableInput {
/** Required for a zero-batch Table; existing nonempty structural inputs need not supply it. */
readonly schema?: ArrowRecordBatchInput['schema']
readonly batches: ReadonlyArray<ArrowRecordBatchInput>
}

/** A schema-bearing empty batch for Tables/readers with no data batches. */
export function emptyArrowBatch(schema: ArrowRecordBatchInput['schema'] | undefined): ArrowRecordBatchInput {
if (!schema?.fields?.length) throw new ChdbCompileError('empty Arrow source must provide a non-empty schema')
return { schema, numCols: schema.fields.length, numRows: 0,
data: { children: schema.fields.map((field) => {
const type = field.type as { toString(): string; indices?: { toString(): string } }
const dictionary = type.toString().startsWith('Dictionary<')
? { data: [{ offset: 0, length: 0, nullCount: 0,
values: new Uint8Array(0), valueOffsets: new Int32Array([0]) }] }
: undefined
return { offset: 0, length: 0, nullCount: 0,
values: type.indices?.toString() === 'Uint32' ? new Uint32Array(0) : new Int32Array(0),
valueOffsets: new Int32Array([0]), dictionary }
}) } }
}

interface ArrowDataInput {
readonly offset?: number
Expand Down Expand Up @@ -341,6 +361,8 @@ export function registerArrowTable(
if (Array.isArray(columns)) {
validate(name, columns)
encoded = columns.map(encodeColumn)
} else if ('batches' in columns && columns.batches.length === 0) {
encoded = encodeArrowBatch(emptyArrowBatch(columns.schema))
} else if ('batches' in columns && columns.batches.length > 1) {
encodedBatches = columns.batches.map(encodeArrowBatch)
encoded = encodedBatches[0]!
Expand Down
21 changes: 15 additions & 6 deletions src/layer3/execute/arrow-insert.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/** Bulk insert through one in-memory Arrow registration and an engine-side INSERT SELECT. */
import { ChdbCompileError } from '../../errors'
import { ChdbAbortError, ChdbCompileError } from '../../errors'
import { quoteIdentifier, quoteIdentifierSegment } from '../compiler/identifier'
import { registerArrowSource, type ArrowColumnInput, type ArrowRecordBatchInput, type ArrowTableInput } from './arrow-input'
import { registerArrowReader, type ArrowReaderSource } from './arrow-reader'
Expand Down Expand Up @@ -38,7 +38,8 @@ export async function insertArrow(
const batch = !Array.isArray(input) && 'batches' in input ? input.batches[0] : input
const columns = Array.isArray(batch)
? batch.map((column) => column.name)
: (batch as ArrowRecordBatchInput).schema?.fields.map((field) => field.name)
: (batch as ArrowRecordBatchInput | undefined)?.schema?.fields.map((field) => field.name)
?? (!Array.isArray(input) && 'batches' in input ? input.schema?.fields.map((field) => field.name) : undefined)
if (!columns?.length || new Set(columns).size !== columns.length) {
throw new ChdbCompileError('insertArrow: source needs unique column names')
}
Expand All @@ -64,18 +65,26 @@ export async function insertArrow(
export async function insertArrowReader(
table: string,
source: ArrowReaderSource,
opts: { session: ArrowInsertSession },
opts: { session: ArrowInsertSession; signal?: AbortSignal },
): Promise<ArrowInsertSummary> {
const destination = quoteIdentifier(table)
const reader = await registerArrowReader(source, opts)
try {
if (new Set(reader.fields).size !== reader.fields.length) {
throw new ChdbCompileError('insertArrowReader: source needs unique column names')
}
if (opts.signal?.aborted) throw new ChdbAbortError('Arrow insert aborted')
const fieldList = reader.fields.map(quoteIdentifierSegment).join(', ')
const result = await opts.session.queryAsync(
`INSERT INTO ${destination} (${fieldList}) SELECT ${fieldList} FROM arrowstream('${reader.name}')`,
)
let result: Awaited<ReturnType<ArrowInsertSession['queryAsync']>>
try {
result = await opts.session.queryAsync(
`INSERT INTO ${destination} (${fieldList}) SELECT ${fieldList} FROM arrowstream('${reader.name}')`,
)
} catch (error) {
if (opts.signal?.aborted) throw new ChdbAbortError('Arrow insert aborted', { cause: error })
throw error
}
if (opts.signal?.aborted) throw new ChdbAbortError('Arrow insert aborted')
await reader.producerDone
return { rowsSubmitted: reader.rowsSubmitted, rowsRead: result.rowsRead,
bytesRead: result.bytesRead, elapsed: result.elapsed }
Expand Down
50 changes: 32 additions & 18 deletions src/layer3/execute/arrow-reader.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
/** Bounded async Arrow reader -> core C Stream source. */
import { randomUUID } from 'node:crypto'
import { ChdbCompileError } from '../../errors'
import { ChdbAbortError, ChdbCompileError } from '../../errors'
import { runtime, type Runtime } from '../runtime'
import { encodeArrowBatch, type ArrowRecordBatchInput } from './arrow-input'
import { encodeArrowBatch, emptyArrowBatch, type ArrowRecordBatchInput } from './arrow-input'

export type ArrowReaderSource = (AsyncIterable<ArrowRecordBatchInput> | Iterable<ArrowRecordBatchInput>) & {
/** Required only when the reader yields no batches. */
Expand Down Expand Up @@ -39,9 +39,10 @@ const readerFinalizers = new FinalizationRegistry<{
*/
export async function registerArrowReader(
source: ArrowReaderSource,
opts: { session: { readonly connection: unknown } },
opts: { session: { readonly connection: unknown }; signal?: AbortSignal },
): Promise<ArrowReaderHandle> {
if (!opts?.session?.connection) throw new ChdbCompileError('registerArrowReader requires an active Session')
if (opts.signal?.aborted) throw new ChdbAbortError('Arrow reader registration aborted')
runtime()._assertArrowCore()
const conn = opts.session.connection
const asyncMethod = (source as AsyncIterable<ArrowRecordBatchInput>)[Symbol.asyncIterator]
Expand All @@ -51,28 +52,34 @@ export async function registerArrowReader(
}
const iterator: AsyncIterator<ArrowRecordBatchInput> | Iterator<ArrowRecordBatchInput> =
typeof asyncMethod === 'function' ? asyncMethod.call(source) : syncMethod.call(source)
const first = await iterator.next()
// Registration must not hold the caller hostage when a network-backed
// iterator has not produced its first batch. An uncooperative next() cannot
// be forcibly stopped; return() is requested without awaiting it.
const first = opts.signal ? await new Promise<IteratorResult<ArrowRecordBatchInput>>((resolve, reject) => {
const onAbort = () => {
try { void Promise.resolve(iterator.return?.()).catch(() => {}) } catch { /* best effort */ }
reject(new ChdbAbortError('Arrow reader registration aborted'))
}
if (opts.signal!.aborted) { onAbort(); return }
opts.signal!.addEventListener('abort', onAbort, { once: true })
Promise.resolve().then(() => iterator.next()).then(
(value) => { opts.signal!.removeEventListener('abort', onAbort); resolve(value) },
(error) => { opts.signal!.removeEventListener('abort', onAbort); reject(error) },
)
}) : await iterator.next()
if (opts.signal?.aborted) {
try { void Promise.resolve(iterator.return?.()).catch(() => {}) } catch { /* best effort */ }
throw new ChdbAbortError('Arrow reader registration aborted')
}
if (opts.session.connection !== conn) {
try { await iterator.return?.() } catch { /* source cleanup is best effort */ }
throw new ChdbCompileError('registerArrowReader: Session closed while awaiting the first batch')
}
let firstBatch: ArrowRecordBatchInput
if (first.done) {
const schema = (source as { schema?: ArrowRecordBatchInput['schema'] }).schema
if (!schema?.fields?.length) throw new ChdbCompileError('empty Arrow reader must provide a non-empty schema')
// Do not cross apache-arrow's independent ESM/CJS constructors: build a
// zero-row structural batch from the public schema shape instead.
firstBatch = { schema, numCols: schema.fields.length, numRows: 0,
data: { children: schema.fields.map((field) => {
const type = field.type as { toString(): string; indices?: { toString(): string } }
const dictionary = type.toString().startsWith('Dictionary<')
? { data: [{ offset: 0, length: 0, nullCount: 0,
values: new Uint8Array(0), valueOffsets: new Int32Array([0]) }] }
: undefined
return { offset: 0, length: 0, nullCount: 0,
values: type.indices?.toString() === 'Uint32' ? new Uint32Array(0) : new Int32Array(0),
valueOffsets: new Int32Array([0]), dictionary }
}) } }
// Apache Arrow has separate ESM/CJS constructors; use its public schema shape.
firstBatch = emptyArrowBatch(schema)
} else firstBatch = first.value

const rt = runtime()
Expand All @@ -87,6 +94,7 @@ export async function registerArrowReader(
throw e
}
let closing = false
let onAbort: (() => void) | undefined
let rowsSubmitted = first.done ? 0 : firstBatch.numRows
const producerDone = (async () => {
try {
Expand Down Expand Up @@ -115,6 +123,7 @@ export async function registerArrowReader(
async close() {
if (!closing) {
closing = true
if (onAbort) opts.signal!.removeEventListener('abort', onAbort)
rt._arrowCancelReader(native)
rt._arrowUnregister(conn, name, token)
readerFinalizers.unregister(result)
Expand All @@ -127,5 +136,10 @@ export async function registerArrowReader(
},
}
readerFinalizers.register(result, { rt, native, conn, name, token }, result)
if (opts.signal) {
onAbort = () => { void result.close() }
if (opts.signal.aborted) onAbort()
else opts.signal.addEventListener('abort', onAbort, { once: true })
}
return result
}
Loading
Loading