Skip to content

Pool in-memory connections and make Python and DataStore execution thread-safe - #808

Open
auxten wants to merge 1 commit into
mainfrom
cursor/unified-session-and-datastore-concurrency-f9e8
Open

auxten wants to merge 1 commit into
mainfrom
cursor/unified-session-and-datastore-concurrency-f9e8

Conversation

@auxten

@auxten auxten commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

Summary

Harden concurrency, session lifecycle, and in-memory connection reuse across both the chdb Python wrapper and DataStore:

  1. Zero-boot import chdb & free-threaded RTLD_DEEPBIND (chdb/__init__.py):

    • Stop booting and tearing down EmbeddedServer (SELECT version()) during import chdb; resolve chdb.engine_version lazily via PEP 562 __getattr__ (or _chdb.engine_version() when bound by chdb-core) while keeping chdb.__version__, chdb.chdb_version, chdb.core_version, and chdb.__chdb_core_version__ immediately available.
    • Iterate importlib.machinery.EXTENSION_SUFFIXES in _preload_chdb_with_deepbind() so free-threaded Linux Python wheels (_chdb.cpython-313t-*.so / 314t) receive RTLD_DEEPBIND preload.
  2. Path-aware :memory: connection pool & deterministic stream cleanup (chdb/_pool.py, chdb/__init__.py, chdb/arrow.py, chdb/agents/tool.py):

    • Replace module-level g_conn_lock serialization and g_arrow_stream_active rejection in chdb.query(), chdb.query_arrow(), and chdb.query_arrow_batches() with a thread-safe :memory: connection pool and condition-backed path coordinator. Concurrent :memory: queries, transferable batch streams, and re-entrant Python UDFs borrow independent handles; any non-READ_ONLY query marks the :memory: generation dirty and drains pooled handles (including surviving borrowers on release) until total :memory: count reaches zero.
    • Coordinate transient and user-owned connections across :memory:, persistent paths, and custom startup options so concurrent query(..., path=...) / query_arrow(..., path=...) calls across different paths wait for incompatible transient handles on other threads to finish closing while preserving immediate fail-fast behavior for same-thread / user-held / stream-held path conflicts.
    • Bind ChDBTool.dataframe_query DataFrames into an isolated call-local function frame (types.FunctionType) instead of mutating module globals(), serialize per-Connection stream startup/registration/fetch under _ConnTracker (preserving active streams when preflight validation fails before native execution), and instrument StreamingResult and ArrowBatchStream with re-entrant-safe weakref.finalize / __del__ cleanup so abandoned iterators cancel their native stream and release PythonTableCache DataFrame references on GC without deadlocks.
  3. Thread-safe DataStore executor, connection locking, and streaming (datastore/executor.py, datastore/connection.py, datastore/table_functions.py):

    • Give Executor (including _global_executor) dedicated per-thread :memory: Connection slots via threading.local(), using a thread-local-only exit sentinel drained on both fast and slow slot lookups so departed worker threads reclaim their connections automatically and reset_executor() races retry cleanly against the replacement global executor.
    • Guard datastore.Connection operations with a per-connection threading.RLock, remove the process-global gc.disable() / gc.enable() race in _query_df_streaming(), synchronize _PYTHON_DATAFRAME_REGISTRY in datastore/table_functions.py, and re-enable/expand multithreaded tests in datastore/tests/test_concurrency.py.

Type of change

  • Bug fix

Linked issues

Part of #807
Refs #197
Refs #391

How to verify

On Linux x86_64 with Python 3.12.3 and chdb-core 26.11.1:

# Lint checks
ruff check datastore --statistics
ruff check --select E,F,W --ignore E501 chdb/__init__.py chdb/_pool.py chdb/arrow.py chdb/agents/tool.py tests/test_session_pool_concurrency.py tests/test_native_arrow.py

# Top-level chdb wrapper, durable, agents, Arrow, and session/pool concurrency tests
python -m pytest tests/ -v --tb=short
# 159 passed, 6 skipped, 33 subtests passed

# DataStore concurrency, streaming, settings, and multi-datasource suites on pandas 2.3.3 (<3.0) and pandas 3.1.1 (>=3.0)
cd datastore && python -m pytest tests/test_concurrency.py tests/test_streaming_df.py tests/test_chdb_settings.py tests/test_multi_datasource.py -v --tb=short -m "not clickhouse_server"
# pandas 2.3.3: 70 passed
# pandas 3.1.1: 55 passed (concurrency + streaming + settings)

Checklist

  • Tests added or updated for the behavior change (regression test for bug fixes)
  • ruff check datastore --statistics passes locally
  • If the change touches DataStore semantics: ran the relevant pytest module on both pandas<3.0 and pandas>=3.0 (Python ≥ 3.11)
  • User-facing API or behavior changes have updated docs (docs/, docstrings, examples/)
  • No new dependencies added to requirements.txt without prior discussion

Notes for reviewers

  • Module-level chdb.query(), chdb.query_arrow(), and chdb.query_arrow_batches() now run concurrently with active module-level Arrow batch streams by borrowing independent :memory: connections; single-session ArrowSession instances continue to reject overlapping queries while a stream on that same session is open.
  • chdb.state.sqlitelike and chdb.session.state are instrumented in-place via an import loader hook in chdb/_pool.py so chdb does not duplicate chdb-core's subpackage files on disk (chdb and chdb-core both install chdb/__init__.py #679) and type(conn._conn) is chdb._chdb.connect remains untouched.
Open in Web Open in Cursor 

Note

Pool in-memory connections and make Python and DataStore execution thread-safe

  • Adds MemoryConnectionPool in chdb/_pool.py. Module-level chdb.query, query_arrow, and query_arrow_batches on unadorned memory paths now borrow pooled connections instead of serializing behind the global connection lock, so concurrent queries and open batch streams work together. Other paths use transient connections marked dirty for mutating SQL.
  • Makes Executor use per-thread in-memory connections with cleanup of exited threads' slots, and serializes Connection operations (execute, query_arrow, query_df, DataFrame binding) with an instance lock in datastore/connection.py and datastore/executor.py.
  • ChDBTool.dataframe_query no longer mutates module globals; it builds a per-call globals mapping and serializes calls with an instance lock in chdb/agents/tool.py.
  • ArrowSession tracks its active stream by weak reference, so abandoned streams are collected; engine_version resolves lazily through the pool instead of an eager query.
  • Behavioral Change: module-level queries no longer fail while an Arrow batch stream is active; g_arrow_stream_active and the global-lock serialization in chdb.query/chdb.arrow are removed (see chdb/init.py and docs/python-arrow-direct.md).

Macroscope summarized 9bea469.

@CLAassistant

CLAassistant commented Nov 19, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Comment thread chdb/_pool.py Resolved
Comment thread chdb/_pool.py Resolved
Comment thread chdb/_pool.py Resolved
Comment thread chdb/_pool.py Resolved
Comment thread chdb/__init__.py Outdated
conn.close()
_memory_pool.release(conn, gen, sql_class=sql_class, had_error=had_error)

_memory_pool.prepare_for_connection(conn_str)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High chdb/__init__.py:260

Concurrent query() calls using different persistent paths fail with cannot connect with different path instead of both completing. Removing g_conn_lock lets the second call reach _chdb.connect() while the first path is still open, and _memory_pool.prepare_for_connection() only drains idle handles; retain path-aware serialization that waits for incompatible active connections to close.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @chdb/__init__.py around line 260:

Concurrent `query()` calls using different persistent paths fail with `cannot connect with different path` instead of both completing. Removing `g_conn_lock` lets the second call reach `_chdb.connect()` while the first path is still open, and `_memory_pool.prepare_for_connection()` only drains idle handles; retain path-aware serialization that waits for incompatible active connections to close.

Comment thread datastore/executor.py Resolved
Comment thread datastore/executor.py Resolved
Comment thread chdb/_pool.py Resolved
Comment thread datastore/executor.py Resolved
Comment thread chdb/_pool.py Resolved
Comment thread chdb/_pool.py Resolved
@cursor
cursor Bot force-pushed the cursor/unified-session-and-datastore-concurrency-f9e8 branch from 2e6a1aa to 7ec1bd2 Compare November 19, 2026 14:26
Comment thread chdb/arrow.py Outdated
if _native_connection is _DEFAULT_NATIVE_CONNECTION:
if not settings and _is_memory_path(path):
conn, gen, owner_tid = _memory_pool.borrow()
sql_class = _classify_sql(conn, sql)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High chdb/arrow.py:357

A Ctrl-C during _classify_sql() leaks the borrowed connection and can leave subsequent queries from another thread waiting indefinitely. Classification runs before the try/finally that calls _memory_pool.release(), and KeyboardInterrupt escapes; initialize sql_class conservatively and move classification inside the protected region so the lease is always released.

Also found in 1 other location(s)

chdb/__init__.py:240

_classify_sql() runs after borrowing but before entering the try/finally that releases the handle. If classification is interrupted by Ctrl-C (KeyboardInterrupt), its except Exception in chdb/_pool.py:73 does not catch that exception, so _memory_pool.release() never runs. The pool permanently retains the phantom _in_use / _in_use_by_thread reservation; subsequent persistent-path queries from another thread wait indefinitely in _wait_and_drain_for_target() for that reservation to disappear. Move classification inside the protected region, with a conservative initial SQL class, so interruption cannot leak the lease.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @chdb/arrow.py around line 357:

A Ctrl-C during `_classify_sql()` leaks the borrowed connection and can leave subsequent queries from another thread waiting indefinitely. Classification runs before the `try/finally` that calls `_memory_pool.release()`, and `KeyboardInterrupt` escapes; initialize `sql_class` conservatively and move classification inside the protected region so the lease is always released.

Also found in 1 other location(s):
- chdb/__init__.py:240 -- `_classify_sql()` runs after borrowing but before entering the `try/finally` that releases the handle. If classification is interrupted by Ctrl-C (`KeyboardInterrupt`), its `except Exception` in `chdb/_pool.py:73` does not catch that exception, so `_memory_pool.release()` never runs. The pool permanently retains the phantom `_in_use` / `_in_use_by_thread` reservation; subsequent persistent-path queries from another thread wait indefinitely in `_wait_and_drain_for_target()` for that reservation to disappear. Move classification inside the protected region, with a conservative initial SQL class, so interruption cannot leak the lease.

…read-safe

- Defer engine_version resolution in chdb/__init__.py so import chdb does not boot EmbeddedServer, and preload free-threaded extension suffixes with RTLD_DEEPBIND on Linux.
- Replace g_conn_lock and g_arrow_stream_active with a path-aware :memory: connection pool, retiring dirty catalog generations across concurrent borrowers and coordinating transient vs user-held paths.
- Isolate ChDBTool.dataframe_query bindings per call without mutating module globals, serialize Connection stream registration after preflight validation, and clean up abandoned StreamingResult and ArrowBatchStream iterators safely under GC.
- Isolate DataStore Executor :memory: connections per thread with thread-exit slot reclamation, per-Connection RLock, synchronized _PYTHON_DATAFRAME_REGISTRY, and streaming without process-wide gc.disable()/gc.enable().
@cursor
cursor Bot force-pushed the cursor/unified-session-and-datastore-concurrency-f9e8 branch from 7ec1bd2 to 9bea469 Compare November 19, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants