Skip to content

test CI: async-stress lifecycle-race 120s hang on main (macos-14/20.x) since 09454213 #53

Closed

Description

@chibugai

Summary

The test workflow is red on main.
Run: https://github.com/chdb-io/chdb-node/actions/runs/28039984270
Triggering commit: 09454213 — merge of #51 ("feat(l1): multi-connection model + async-stress timeout stabilization").

One matrix leg, test (macos-14, 20.x), failed. All 13 other legs (ubuntu-latest, ubuntu-24.04-arm, macos-15-intel, and macos-14 with Node 18.x / 22.x) passed.

Failure signature

test/v3/async-stress.test.ts → lifecycle race: close / registry mutation during an in-flight query › closing a session while its async query is in flight does not crash (100x) → Test timed out in 120000ms.

The abandoned in-flight native op then poisoned the single in-process engine, so 48 later v3 tests cascade-failed with:

Code: 236. DB::Exception: The server is shutting down due to a fatal error. (ABORTED)

across async.test.ts, arrow.test.ts, insert*.test.ts, querybind.test.ts, registry.test.ts, serialize.test.ts, smoke.test.ts, stream.test.ts, robustness.test.ts, e2e.test.ts, lifecycle.test.ts. The test file's own header comment predicts exactly this cascade.

Why this is a real regression, not a slow-runner flake

In the pre-merge PR run (run 27993099218, 2026-06-23T00:16Z) the same lifecycle race … (100x) case passed in ~3.2 s on macos-14/20.x (and 2.7–6.5 s on every other leg). On post-merge main the same case hung the full 120 s on macos-14/20.x — a >37× blow-up, i.e. a hang/deadlock, not a marginal slowdown.

It is timing-sensitive / probabilistic: macos-14 with Node 18.x and 22.x both passed in this same run; only the 20.x leg hung. The test deliberately widens the race window (await sleep(i % 5) between worker dispatch and s.close()), so the deadlock surfaces intermittently.

Root-cause hypothesis

s.close() during an in-flight queryAsync can leave the query promise unsettled: the close / registry-teardown path (index.js + lib/chdb_node.cpp) races the async worker dispatch, and on a lost-wakeup ordering the awaited await p never resolves or rejects → 120 s timeout. Because plain queryAsync is not tracked for drain, the orphaned native op survives into the next test and aborts the shared in-process engine (code 236), cascading into every later file. #51 reworked the multi-connection model and this exact close/in-flight path; its "timeout stabilization" did not fully close the window on macos-14/20.x.

Reproduction

  • CI: re-run the test workflow on main @ 09454213; the macos-14/20.x leg reproduces intermittently.
  • Local (focused): npm ci && npm run build && npx vitest run test/v3/async-stress.test.ts — watch the "lifecycle race … (100x)" case.

Note: this is a macOS-arm64 + Node-20, timing-sensitive deadlock; it does not reproduce deterministically on Linux x86_64, so I have not validated a fix locally and am intentionally not submitting a speculative patch into the native concurrency path. Filing for the maintainer who is actively iterating on this code (#51).

Suggested directions

  • Ensure Session.close() cannot orphan an in-flight async op: either track plain queryAsync for drain (as the storm cases already are) so close awaits/cancels it, or make close idempotently settle the pending promise with a typed error.
  • Robustness: a single hung/abandoned op should not leave the shared engine in a fatal ABORTED state that poisons all later tests — consider isolating engine teardown so one failure doesn't cascade.

Dedup

No open or closed chdb-node issue matches this signature (searched shutting down fatal error, lifecycle race async, and the full open/closed issue list). #17 is an unrelated 2024 shutdown-design question.


Filed automatically by chibugai's CI monitor (Responsibility 1: failure on main). cc @auxten @wudidapaopao

Activity

  1. chibugai commented on Jun 24, 2026

    @chibugai
    ContributorAuthor

    Seen again on main after the #52 merge — the hang is not tied to the single 09454213 commit.

    Still macOS-arm64 + Node-20 only (all other legs green). Confirms the close/in-flight race reproduces on newer main.

  2. added a commit that references this issue on Jun 24, 2026
  3. added a commit that references this issue on Jun 24, 2026
  4. chibugai commented on Jun 26, 2026

    @chibugai
    ContributorAuthor

    Seen again on main — the async-stress lifecycle-race hang recurred.

    • Run: https://github.com/chdb-io/chdb-node/actions/runs/28222953048 (push to main, 771a5c85)
    • Triggering merge is docs-only (Ship AGENTS.md and llms-full.txt for agent discovery #60 ships AGENTS.md + llms-full.txt, no code/test changes) — confirms this is an environmental/timing flake independent of any code change.
    • Only test (macos-15-intel, 20.x) failed; the other 13 matrix jobs (incl. macos-14/20.x) passed.
    • Originating test: test/v3/async-stress.test.ts › lifecycle race… › "opening a new session while a default-conn query is in flight is safe (60x)" hung ~150s (150683ms) and left session chdb-node-oo9h6D active.
    • That leaked session then cascaded: 1500 sequential async queries… failed with "a session is active; close it before using standalone query()", and all 6 test/v3/lifecycle.test.ts cases failed with "only one active data directory per process" (8 failed total).

    Not reopening — recording the recurrence only. Note it migrated from the macos-14 runner (original report) to macos-15-intel, consistent with a runner-speed-dependent timing race in the same lifecycle race block.

  5. chibugai commented on Jul 1, 2026

    @chibugai
    ContributorAuthor

    Seen again on main — same "only one active data directory per process" session-leak cascade, now on d64dda32 (merge of #61, first push-CI run on that commit).

    • Job: test (macos-15-intel, 20.x) only — 13/14 jobs green.
    • An earlier session (…/chdb-node-wXcKJS) is left open, so every subsequent new Session() (index.js:530) throws chdb: only one active data directory per process. This run it surfaces in test/v3/lifecycle.test.ts (cleanup safety gates (#30), signal handlers (D3)), ending in Error: Worker exited unexpectedly.
    • Confirms the flake is not commit-specific — recurs across 09454213 → 08ffdb2b → 771a5c85 → d64dda32, still isolated to the macos-15-intel/20.x runner. The failing tests are pre-existing (Cleanup should rmSync only if isTemp== true #30 / D3), not the new integration suites added in Add Vercel AI SDK and Mastra tool adapters (chdb/ai-sdk, chdb/mastra) #61.

    Run: https://github.com/chdb-io/chdb-node/actions/runs/28504117679/job/84488190024

    Not reopening — informational only, since this was triaged/closed as a known env-timing flake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions