Repository navigation
Conversation
| conn.close() | ||
| _memory_pool.release(conn, gen, sql_class=sql_class, had_error=had_error) | ||
|
|
||
| _memory_pool.prepare_for_connection(conn_str) |
There was a problem hiding this comment.
🟠 High chdb/__init__.py:260
Concurrent query() calls using different persistent paths fail with cannot connect with different path instead of both completing. Removing g_conn_lock lets the second call reach _chdb.connect() while the first path is still open, and _memory_pool.prepare_for_connection() only drains idle handles; retain path-aware serialization that waits for incompatible active connections to close.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @chdb/__init__.py around line 260:
Concurrent `query()` calls using different persistent paths fail with `cannot connect with different path` instead of both completing. Removing `g_conn_lock` lets the second call reach `_chdb.connect()` while the first path is still open, and `_memory_pool.prepare_for_connection()` only drains idle handles; retain path-aware serialization that waits for incompatible active connections to close.
2e6a1aa to
7ec1bd2
Compare
| if _native_connection is _DEFAULT_NATIVE_CONNECTION: | ||
| if not settings and _is_memory_path(path): | ||
| conn, gen, owner_tid = _memory_pool.borrow() | ||
| sql_class = _classify_sql(conn, sql) |
There was a problem hiding this comment.
🟠 High chdb/arrow.py:357
A Ctrl-C during _classify_sql() leaks the borrowed connection and can leave subsequent queries from another thread waiting indefinitely. Classification runs before the try/finally that calls _memory_pool.release(), and KeyboardInterrupt escapes; initialize sql_class conservatively and move classification inside the protected region so the lease is always released.
Also found in 1 other location(s)
chdb/__init__.py:240
_classify_sql()runs after borrowing but before entering thetry/finallythat releases the handle. If classification is interrupted by Ctrl-C (KeyboardInterrupt), itsexcept Exceptioninchdb/_pool.py:73does not catch that exception, so_memory_pool.release()never runs. The pool permanently retains the phantom_in_use/_in_use_by_threadreservation; subsequent persistent-path queries from another thread wait indefinitely in_wait_and_drain_for_target()for that reservation to disappear. Move classification inside the protected region, with a conservative initial SQL class, so interruption cannot leak the lease.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @chdb/arrow.py around line 357:
A Ctrl-C during `_classify_sql()` leaks the borrowed connection and can leave subsequent queries from another thread waiting indefinitely. Classification runs before the `try/finally` that calls `_memory_pool.release()`, and `KeyboardInterrupt` escapes; initialize `sql_class` conservatively and move classification inside the protected region so the lease is always released.
Also found in 1 other location(s):
- chdb/__init__.py:240 -- `_classify_sql()` runs after borrowing but before entering the `try/finally` that releases the handle. If classification is interrupted by Ctrl-C (`KeyboardInterrupt`), its `except Exception` in `chdb/_pool.py:73` does not catch that exception, so `_memory_pool.release()` never runs. The pool permanently retains the phantom `_in_use` / `_in_use_by_thread` reservation; subsequent persistent-path queries from another thread wait indefinitely in `_wait_and_drain_for_target()` for that reservation to disappear. Move classification inside the protected region, with a conservative initial SQL class, so interruption cannot leak the lease.
…read-safe - Defer engine_version resolution in chdb/__init__.py so import chdb does not boot EmbeddedServer, and preload free-threaded extension suffixes with RTLD_DEEPBIND on Linux. - Replace g_conn_lock and g_arrow_stream_active with a path-aware :memory: connection pool, retiring dirty catalog generations across concurrent borrowers and coordinating transient vs user-held paths. - Isolate ChDBTool.dataframe_query bindings per call without mutating module globals, serialize Connection stream registration after preflight validation, and clean up abandoned StreamingResult and ArrowBatchStream iterators safely under GC. - Isolate DataStore Executor :memory: connections per thread with thread-exit slot reclamation, per-Connection RLock, synchronized _PYTHON_DATAFRAME_REGISTRY, and streaming without process-wide gc.disable()/gc.enable().
7ec1bd2 to
9bea469
Compare
Summary
Harden concurrency, session lifecycle, and in-memory connection reuse across both the
chdbPython wrapper andDataStore:Zero-boot
import chdb& free-threadedRTLD_DEEPBIND(chdb/__init__.py):EmbeddedServer(SELECT version()) duringimport chdb; resolvechdb.engine_versionlazily via PEP 562__getattr__(or_chdb.engine_version()when bound bychdb-core) while keepingchdb.__version__,chdb.chdb_version,chdb.core_version, andchdb.__chdb_core_version__immediately available.importlib.machinery.EXTENSION_SUFFIXESin_preload_chdb_with_deepbind()so free-threaded Linux Python wheels (_chdb.cpython-313t-*.so/314t) receiveRTLD_DEEPBINDpreload.Path-aware
:memory:connection pool & deterministic stream cleanup (chdb/_pool.py,chdb/__init__.py,chdb/arrow.py,chdb/agents/tool.py):g_conn_lockserialization andg_arrow_stream_activerejection inchdb.query(),chdb.query_arrow(), andchdb.query_arrow_batches()with a thread-safe:memory:connection pool and condition-backed path coordinator. Concurrent:memory:queries, transferable batch streams, and re-entrant Python UDFs borrow independent handles; any non-READ_ONLYquery marks the:memory:generation dirty and drains pooled handles (including surviving borrowers on release) until total:memory:count reaches zero.:memory:, persistent paths, and custom startup options so concurrentquery(..., path=...)/query_arrow(..., path=...)calls across different paths wait for incompatible transient handles on other threads to finish closing while preserving immediate fail-fast behavior for same-thread / user-held / stream-held path conflicts.ChDBTool.dataframe_queryDataFrames into an isolated call-local function frame (types.FunctionType) instead of mutating moduleglobals(), serialize per-Connectionstream startup/registration/fetch under_ConnTracker(preserving active streams when preflight validation fails before native execution), and instrumentStreamingResultandArrowBatchStreamwith re-entrant-safeweakref.finalize/__del__cleanup so abandoned iterators cancel their native stream and releasePythonTableCacheDataFrame references on GC without deadlocks.Thread-safe
DataStoreexecutor, connection locking, and streaming (datastore/executor.py,datastore/connection.py,datastore/table_functions.py):Executor(including_global_executor) dedicated per-thread:memory:Connectionslots viathreading.local(), using a thread-local-only exit sentinel drained on both fast and slow slot lookups so departed worker threads reclaim their connections automatically andreset_executor()races retry cleanly against the replacement global executor.datastore.Connectionoperations with a per-connectionthreading.RLock, remove the process-globalgc.disable()/gc.enable()race in_query_df_streaming(), synchronize_PYTHON_DATAFRAME_REGISTRYindatastore/table_functions.py, and re-enable/expand multithreaded tests indatastore/tests/test_concurrency.py.Type of change
Linked issues
Part of #807
Refs #197
Refs #391
How to verify
On Linux x86_64 with Python 3.12.3 and
chdb-core 26.11.1:Checklist
ruff check datastore --statisticspasses locallypandas<3.0andpandas>=3.0(Python ≥ 3.11)docs/, docstrings,examples/)requirements.txtwithout prior discussionNotes for reviewers
chdb.query(),chdb.query_arrow(), andchdb.query_arrow_batches()now run concurrently with active module-level Arrow batch streams by borrowing independent:memory:connections; single-sessionArrowSessioninstances continue to reject overlapping queries while a stream on that same session is open.chdb.state.sqlitelikeandchdb.session.stateare instrumented in-place via an import loader hook inchdb/_pool.pysochdbdoes not duplicatechdb-core's subpackage files on disk (chdb and chdb-core both install chdb/__init__.py #679) andtype(conn._conn) is chdb._chdb.connectremains untouched.Note
Pool in-memory connections and make Python and DataStore execution thread-safe
MemoryConnectionPoolin chdb/_pool.py. Module-levelchdb.query,query_arrow, andquery_arrow_batcheson unadorned memory paths now borrow pooled connections instead of serializing behind the global connection lock, so concurrent queries and open batch streams work together. Other paths use transient connections marked dirty for mutating SQL.Executoruse per-thread in-memory connections with cleanup of exited threads' slots, and serializesConnectionoperations (execute,query_arrow,query_df, DataFrame binding) with an instance lock in datastore/connection.py and datastore/executor.py.ChDBTool.dataframe_queryno longer mutates module globals; it builds a per-call globals mapping and serializes calls with an instance lock in chdb/agents/tool.py.ArrowSessiontracks its active stream by weak reference, so abandoned streams are collected;engine_versionresolves lazily through the pool instead of an eager query.g_arrow_stream_activeand the global-lock serialization inchdb.query/chdb.arroware removed (see chdb/init.py and docs/python-arrow-direct.md).Macroscope summarized 9bea469.