Repository navigation
Description
Summary
The test workflow is red on main.
Run: https://github.com/chdb-io/chdb-node/actions/runs/28039984270
Triggering commit: 09454213 — merge of #51 ("feat(l1): multi-connection model + async-stress timeout stabilization").
One matrix leg, test (macos-14, 20.x), failed. All 13 other legs (ubuntu-latest, ubuntu-24.04-arm, macos-15-intel, and macos-14 with Node 18.x / 22.x) passed.
Failure signature
test/v3/async-stress.test.ts → lifecycle race: close / registry mutation during an in-flight query › closing a session while its async query is in flight does not crash (100x) → Test timed out in 120000ms.
The abandoned in-flight native op then poisoned the single in-process engine, so 48 later v3 tests cascade-failed with:
Code: 236. DB::Exception: The server is shutting down due to a fatal error. (ABORTED)
across async.test.ts, arrow.test.ts, insert*.test.ts, querybind.test.ts, registry.test.ts, serialize.test.ts, smoke.test.ts, stream.test.ts, robustness.test.ts, e2e.test.ts, lifecycle.test.ts. The test file's own header comment predicts exactly this cascade.
Why this is a real regression, not a slow-runner flake
In the pre-merge PR run (run 27993099218, 2026-06-23T00:16Z) the same lifecycle race … (100x) case passed in ~3.2 s on macos-14/20.x (and 2.7–6.5 s on every other leg). On post-merge main the same case hung the full 120 s on macos-14/20.x — a >37× blow-up, i.e. a hang/deadlock, not a marginal slowdown.
It is timing-sensitive / probabilistic: macos-14 with Node 18.x and 22.x both passed in this same run; only the 20.x leg hung. The test deliberately widens the race window (await sleep(i % 5) between worker dispatch and s.close()), so the deadlock surfaces intermittently.
Root-cause hypothesis
s.close() during an in-flight queryAsync can leave the query promise unsettled: the close / registry-teardown path (index.js + lib/chdb_node.cpp) races the async worker dispatch, and on a lost-wakeup ordering the awaited await p never resolves or rejects → 120 s timeout. Because plain queryAsync is not tracked for drain, the orphaned native op survives into the next test and aborts the shared in-process engine (code 236), cascading into every later file. #51 reworked the multi-connection model and this exact close/in-flight path; its "timeout stabilization" did not fully close the window on macos-14/20.x.
Reproduction
- CI: re-run the
testworkflow onmain@09454213; the macos-14/20.x leg reproduces intermittently. - Local (focused):
npm ci && npm run build && npx vitest run test/v3/async-stress.test.ts— watch the "lifecycle race … (100x)" case.
Note: this is a macOS-arm64 + Node-20, timing-sensitive deadlock; it does not reproduce deterministically on Linux x86_64, so I have not validated a fix locally and am intentionally not submitting a speculative patch into the native concurrency path. Filing for the maintainer who is actively iterating on this code (#51).
Suggested directions
- Ensure
Session.close()cannot orphan an in-flight async op: either track plainqueryAsyncfor drain (as the storm cases already are) so close awaits/cancels it, or make close idempotently settle the pending promise with a typed error. - Robustness: a single hung/abandoned op should not leave the shared engine in a fatal
ABORTEDstate that poisons all later tests — consider isolating engine teardown so one failure doesn't cascade.
Dedup
No open or closed chdb-node issue matches this signature (searched shutting down fatal error, lifecycle race async, and the full open/closed issue list). #17 is an unrelated 2024 shutdown-design question.
Filed automatically by chibugai's CI monitor (Responsibility 1: failure on main). cc @auxten @wudidapaopao
Activity
chibugai commentedon Jun 24, 2026
Seen again on
mainafter the #52 merge — the hang is not tied to the single09454213commit.08ffdb2b(merge of feat(l2): pluggable Connection — chdb/connection for @clickhouse/client #52, "feat(l2): pluggable Connection");mainhas advanced past the09454213(feat(l1): multi-connection model + async-stress timeout stabilization #51 merge) in the original report.test (macos-14, 20.x), identical signature:test/v3/async-stress.test.ts→ "lifecycle race … closing a session while its async query is in flight does not crash (100x)" →Test timed out in 120000ms, then theCode: 236. DB::Exception: The server is shutting down due to a fatal error. (ABORTED)cascade throughserialize/smoke/stream/lifecycle.Still macOS-arm64 + Node-20 only (all other legs green). Confirms the close/in-flight race reproduces on newer
main.fix(ci): stop close()-mid-flight engine abort (#53) and absorb Result…
Merge pull request #56 from ShawnChen-Sirius/fix/ci-issues-53-54
chibugai commentedon Jun 26, 2026
Seen again on
main— the async-stress lifecycle-race hang recurred.main,771a5c85)AGENTS.md+llms-full.txt, no code/test changes) — confirms this is an environmental/timing flake independent of any code change.test (macos-15-intel, 20.x)failed; the other 13 matrix jobs (incl.macos-14/20.x) passed.test/v3/async-stress.test.ts› lifecycle race… › "opening a new session while a default-conn query is in flight is safe (60x)" hung ~150s (150683ms) and left sessionchdb-node-oo9h6Dactive.1500 sequential async queries…failed with "a session is active; close it before using standalone query()", and all 6test/v3/lifecycle.test.tscases failed with "only one active data directory per process" (8 failed total).Not reopening — recording the recurrence only. Note it migrated from the
macos-14runner (original report) tomacos-15-intel, consistent with a runner-speed-dependent timing race in the samelifecycle raceblock.chibugai commentedon Jul 1, 2026
Seen again on
main— same "only one active data directory per process" session-leak cascade, now ond64dda32(merge of #61, first push-CI run on that commit).test (macos-15-intel, 20.x)only — 13/14 jobs green.…/chdb-node-wXcKJS) is left open, so every subsequentnew Session()(index.js:530) throwschdb: only one active data directory per process. This run it surfaces intest/v3/lifecycle.test.ts(cleanup safety gates (#30),signal handlers (D3)), ending inError: Worker exited unexpectedly.09454213→08ffdb2b→771a5c85→d64dda32, still isolated to the macos-15-intel/20.x runner. The failing tests are pre-existing (Cleanup should rmSync only ifisTemp== true#30 / D3), not the new integration suites added in Add Vercel AI SDK and Mastra tool adapters (chdb/ai-sdk, chdb/mastra) #61.Run: https://github.com/chdb-io/chdb-node/actions/runs/28504117679/job/84488190024
Not reopening — informational only, since this was triaged/closed as a known env-timing flake.