Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/chdb-node-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ on:

# update_libchdb.sh pins a published chdb-core libchdb across all four target
# platforms, so the full v2 + v3 suite is expected green here. The pin is
# currently the v26.11.0-rc.2 release candidate. v26.7.3 was the first stable
# currently the v26.11.1 stable release. v26.7.3 was the first stable
# release to export the Durable V1 ABI (chdb_backup_database_n /
# chdb_restore_database_n / chdb_classify_query_n); v26.7.2-rc.2 was the
# candidate that introduced them.
Expand Down
18 changes: 12 additions & 6 deletions .github/workflows/prebuild-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -206,17 +206,20 @@ jobs:
VER="${GITHUB_REF_NAME#v}"
work="$(mktemp -d)"
cp -R "$GITHUB_WORKSPACE/test/pack" "$work/test"
cp "$GITHUB_WORKSPACE/scripts/smoke-arrow-e2e.mjs" "$work/smoke-arrow-e2e.mjs"
cd "$work"
npm init -y >/dev/null
# The version may lag in the registry right after publish; retry.
ok=
for i in 1 2 3 4 5 6; do
if npm install "chdb@$VER" vitest@2 >/tmp/i.log 2>&1; then ok=1; break; fi
echo "install attempt $i failed (registry propagation?); retry in 15s"; sleep 15
for i in $(seq 1 20); do
if npm install --prefer-online "chdb@$VER" vitest@2 apache-arrow@21.1.0 >/tmp/i.log 2>&1 &&
node -e "const p=require('chdb/package.json');const n='@chdb/lib-'+process.platform+'-'+process.arch+(process.platform==='linux'?'-gnu':'');if(require(n+'/package.json').version!==p.optionalDependencies[n])process.exit(1)" >>/tmp/i.log 2>&1; then ok=1; break; fi
echo "install attempt $i failed (registry propagation?); retry in 20s"; sleep 20
done
[ -n "$ok" ] || { echo "could not install chdb@$VER"; tail -20 /tmp/i.log; exit 1; }
node -e "console.log('verifying installed chdb', require('chdb/package.json').version, 'on', process.version, process.platform, process.arch)"
npx vitest run test
node smoke-arrow-e2e.mjs

verify-old-distro:
# Every leg of `verify` runs on a GitHub runner image, and all four of them
Expand All @@ -243,15 +246,18 @@ jobs:
echo "distro: $(. /etc/os-release; echo "$PRETTY_NAME"), glibc $(ldd --version | head -1 | grep -oE '[0-9]+\.[0-9]+$')"
work="$(mktemp -d)"
cp -R "$GITHUB_WORKSPACE/test/pack" "$work/test"
cp "$GITHUB_WORKSPACE/scripts/smoke-arrow-e2e.mjs" "$work/smoke-arrow-e2e.mjs"
cd "$work"
npm init -y >/dev/null
ok=
for i in 1 2 3 4 5 6; do
if npm install "chdb@$VER" vitest@2 >/tmp/i.log 2>&1; then ok=1; break; fi
echo "install attempt $i failed (registry propagation?); retry in 15s"; sleep 15
for i in $(seq 1 20); do
if npm install --prefer-online "chdb@$VER" vitest@2 apache-arrow@21.1.0 >/tmp/i.log 2>&1 &&
node -e "const p=require('chdb/package.json');const n='@chdb/lib-'+process.platform+'-'+process.arch+(process.platform==='linux'?'-gnu':'');if(require(n+'/package.json').version!==p.optionalDependencies[n])process.exit(1)" >>/tmp/i.log 2>&1; then ok=1; break; fi
echo "install attempt $i failed (registry propagation?); retry in 20s"; sleep 20
done
[ -n "$ok" ] || { echo "could not install chdb@$VER"; tail -20 /tmp/i.log; exit 1; }
npx vitest run test
node smoke-arrow-e2e.mjs

promote:
needs: [verify, verify-old-distro]
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ somewhere later.
| Streaming results (`AsyncIterable`) | ✅ |
| Arrow IPC **output** (`format: 'arrow'` + `toArrow()`) | ✅ |
| AbortSignal / timeout | ✅ (single-shot is honest: rejects early; native runs to completion) |
| Direct Arrow **scan**, reader, typed results, and bulk insert | ✅ in source (matching rc.3 native packages pending) |
| Direct Arrow **scan**, reader, typed results, and bulk insert | ✅ with the matching native package |
| Arrow zero-copy (M2, `{ zeroCopy: true }`) | ⏳ follow-up |
| chDB ↔ `@clickhouse/client` integration (`chdb/connection`, **experimental**) | ✅ |
| Durable V1 control plane (`chdb/durable`, **experimental**) | ✅ pure TS, engine injected |
Expand All @@ -155,8 +155,8 @@ somewhere later.

See the [Arrow guide](docs/arrow.md) for examples, lifecycle rules, and the
difference between direct C Data/C Stream and the existing Arrow IPC path.
The direct APIs require chdb-core v26.11.0-rc.3; they are not in the currently
published npm platform packages.
The direct APIs require chdb-core v26.11.0-rc.3 or newer. This release pins
v26.11.1 and packages its native library for each supported platform.

### chDB ↔ `@clickhouse/client` integration (`chdb/connection`, experimental)

Expand Down
4 changes: 2 additions & 2 deletions docs/arrow.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

The direct Arrow APIs let you query in-memory Arrow data, read typed results,
and insert batches without Arrow IPC. They require chdb-core v26.11.0-rc.3
or newer. The matching npm platform packages are not published yet.
or newer; this release pins v26.11.1 in its matching npm platform packages.

Arrow Inter-Process Communication (IPC) encodes data as a byte stream or file. The direct APIs here exchange in-memory Arrow buffers and batches with the native engine instead; they may still copy data while converting it. A `RecordBatch` is a typed group of columnar rows, and a reader supplies successive batches.

Once the matching packages are released, install `chdb` and its optional peer:
Install `chdb` and its optional peer:

```sh
npm install chdb apache-arrow
Expand Down
8 changes: 4 additions & 4 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion test/v3/layer3/arrow-worker-isolate.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,12 @@ describe('Arrow source in a worker isolate', () => {
const {Worker}=require('node:worker_threads'),m=require('./index.js');
(async()=>{const s=new m.Session();
try{const worker=new Worker(${JSON.stringify(workerCode)},{eval:true,workerData:{path:s.path}});
const exited=new Promise((resolve,reject)=>worker.once('exit',code=>code===0?resolve():reject(Error('exit '+code))));
const value=await new Promise((resolve,reject)=>{
worker.once('message',resolve);worker.once('error',reject)
});
if(value!=='ok')throw Error(value);
await new Promise((resolve,reject)=>worker.once('exit',code=>code===0?resolve():reject(Error('exit '+code))));
await exited;
const result=await s.queryAsync('SELECT 7',{format:'CSV'});
if(result.text().trim()!=='7')throw Error('main Session was closed');
console.log('concurrent isolate OK');
Expand Down
12 changes: 4 additions & 8 deletions update_libchdb.sh
Original file line number Diff line number Diff line change
Expand Up @@ -16,14 +16,10 @@ set -e
# count and target-database proof. chdb/durable's engine adapter cannot be built
# without them. v26.7.3 is the first stable release that exports them.
#
# The pin is currently a release candidate, and chdb_version() on it returns the
# full "26.11.0-rc.3", suffix and all. Durable records that string in
# head.engine.version and raises min_reader to it on every write. A pre-release
# sorts below the release it leads to, so an object written against this RC
# stays readable by the eventual stable, but every earlier engine (26.11.0-rc.2,
# 26.9.x and before) is refused it with engine_incompatible from then on.
# The Arrow A1–A6 ABI and chdb-core#323's registration ownership fixes are
# present in rc.3. See docs/arrow-rc3-conformance.md for validation scope.
# The pin is a stable release containing the direct Arrow ABI and registration
# ownership fixes first validated against rc.3. Durable records the engine
# version in head.engine.version and refuses earlier incompatible engines.
# See docs/arrow-rc3-conformance.md for the earlier candidate's validation scope.
#
# Keep the pin on its own line and literal: the release check greps for it when
# it proposes a bump.
Expand Down
Loading