connection: tolerate clean startup close

[dkropachev]

Summary

Fixes #942.

Version 3.29.11 started raising ConnectionShutdown from Connection.factory() when the server closed the socket during startup without setting last_error. That made a clean server-side close look like an immediate connection-factory failure instead of leaving it to the owning pool/control connection path.

Scylla maintenance mode has the same socket shape for regular CQL: it can accept the TCP connection, receive the first CQL startup frame, and close it because regular CQL is unavailable while the maintenance socket remains usable. This restores the pre-3.29.11 factory behavior for that clean startup close while keeping actual startup errors and timeouts unchanged.

Reproducer

Added test_factory_returns_maintenance_mode_clean_startup_close, which starts a real local TCP listener that behaves like maintenance-mode regular CQL:

1. accept a CQL socket
2. read the driver's initial OPTIONS frame
3. close the socket cleanly during startup

With the 3.29.11 elif conn.is_closed factory check temporarily restored, this test fails with:

cassandra.connection.ConnectionShutdown: Connection to 127.0.0.1:<port> was closed by server

With this PR, the same test passes and the closed connection is returned to the owning path.

Driver surface

Touches connection startup behavior only. No protocol serialization, query execution, metadata, or reactor code changed.

Compatibility / risk

Low but behavior-visible: clean server-side close during startup no longer raises directly from Connection.factory(). Callers receive the closed connection and existing pool/control-connection handling decides host state and reconnection cadence, matching behavior before 3.29.11.

Tests

• uv run pytest -rf tests/unit/test_connection.py tests/unit/test_host_connection_pool.py
• TZ=UTC uv run pytest -rf tests/unit

Note: full unit suite without TZ=UTC still has an unrelated timezone-sensitive failure in tests/unit/test_types.py::DateRangeDeserializationTests::test_deserialize_date_range_year in this environment.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bf1a0079-45bc-4320-8179-a7edbae76f55

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Restores pre-3.29.11 handling of clean server-side closes during connection startup.

Changes:

• Returns cleanly closed startup connections to their owner.
• Adds regression coverage for this behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
cassandra/connection.py	Removes immediate ConnectionShutdown for clean startup closes.
tests/unit/test_connection.py	Tests returning a cleanly closed connection.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cassandra/connection.py:990

• Returning a closed connection is treated as a successful reconnect by existing owners. _ReconnectionHandler.run() invokes on_reconnection() and its callback for any returned value (cassandra/pool.py:281-305), and _HostReconnectionHandler then marks the host up (cassandra/pool.py:353-361); initial pool construction likewise installs the returned connection without checking is_closed (cassandra/pool.py:429-431). A clean startup close can therefore falsely mark a maintenance-mode host up and build a pool around a closed socket instead of preserving reconnection cadence. The owning paths need explicit closed-result handling (with coverage), or the factory needs a distinct failure/result contract.

        else:
            return conn

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

cassandra/connection.py:975

• The opening sentence (“returns a connection once startup has completed”) conflicts with the documented behavior immediately below (direct callers may receive a closed connection when startup did not complete). Consider rewording the first sentence to reflect that the method may return a closed connection when the server cleanly closes during the handshake.

        A factory function which returns a connection once startup has
        completed, or raises an exception otherwise.

        Direct callers may receive a closed connection when the server accepts
        the socket and then cleanly closes it during the startup handshake.
        Callers that own pool startup or reconnection probes pass ``host_conn``
        or ``_raise_on_startup_close`` and receive :exc:`ConnectionShutdown`
        for that clean startup close instead.

cassandra/connection.py:998

• The exception is constructed using the input endpoint rather than the connection’s actual endpoint (conn.endpoint). Using conn.endpoint (both for message formatting and for the exception’s endpoint attribute) is typically more accurate if the connection normalizes/rewrites endpoints (e.g., via an endpoint factory or translation) and also avoids the slightly confusing % (endpoint,) tuple formatting.

        elif conn.is_closed and (host_conn is not None or raise_on_startup_close):
            raise ConnectionShutdown(
                "Connection to %s was closed during the startup handshake" % (endpoint,),
                endpoint)

tests/unit/test_connection.py:480

• server.first_frame[4] can raise IndexError if the server receives fewer than 5 bytes (e.g., if the client closes early or a timeout occurs). Adding an explicit assert len(server.first_frame) >= 5 (or parsing/validating the 9-byte header length) would make failures clearer and avoid masking the underlying cause.

            assert conn.is_closed
            assert server.received_frame.wait(2)
            assert server.error is None
            assert server.first_frame[4] == 0x05  # OPTIONS

tests/unit/test_connection.py:502

• This test asserts on the private/internal attribute _pending_connections, which can make the test brittle to refactors of internal connection tracking. If possible, assert the externally observable behavior you care about (e.g., that the factory raises and that the returned/created connection is closed) without coupling to _pending_connections’ internal representation.

            host_conn = Mock()
            host_conn.is_shutdown = False
            host_conn._pending_connections = []

            with pytest.raises(ConnectionShutdown) as exc_info:
                MaintenanceModeConnection.factory(
                    DefaultEndPoint('127.0.0.1', server.port),
                    timeout=2,
                    host_conn=host_conn)

            assert "closed during the startup handshake" in str(exc_info.value)
            assert server.received_frame.wait(2)
            assert server.error is None
            assert server.first_frame[4] == 0x05  # OPTIONS
            assert len(host_conn._pending_connections) == 1
            assert host_conn._pending_connections[0].is_closed

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

cassandra/connection.py:975

• The first sentence says the factory “returns a connection once startup has completed,” but the docstring later states direct callers may receive a closed connection during the startup handshake. Please revise the opening description to reflect that the method can return a closed (non-serviceable) connection in the clean-startup-close case, so the contract is not self-contradictory.

        A factory function which returns a connection once startup has
        completed, or raises an exception otherwise.

        Direct callers may receive a closed connection when the server accepts
        the socket and then cleanly closes it during the startup handshake.
        Callers that own pool startup or reconnection probes pass ``host_conn``
        or ``_raise_on_startup_close`` and receive :exc:`ConnectionShutdown`
        for that clean startup close instead.

cassandra/connection.py:966

• _raise_on_startup_close is currently a “hidden” control flag extracted from **kwargs, which makes the factory behavior easier to call incorrectly (typos silently change behavior) and harder to discover for implementers of custom Connection subclasses. Consider making it an explicit (preferably keyword-only) parameter of factory() (defaulting to False) and documenting it alongside host_conn.

    def factory(cls, endpoint, timeout, host_conn = None, *args, **kwargs):

cassandra/connection.py:977

• _raise_on_startup_close is currently a “hidden” control flag extracted from **kwargs, which makes the factory behavior easier to call incorrectly (typos silently change behavior) and harder to discover for implementers of custom Connection subclasses. Consider making it an explicit (preferably keyword-only) parameter of factory() (defaulting to False) and documenting it alongside host_conn.

        raise_on_startup_close = kwargs.pop('_raise_on_startup_close', False)

tests/unit/test_connection.py:429

• close() uses a timed join(2) on a daemon thread and does not verify the thread actually stopped. This can leak background activity across tests and introduce intermittency under slow CI. Prefer a deterministic shutdown (e.g., signal + unblock accept/recv, then join() without a timeout, or assert not self.thread.is_alive() after joining) so failures are visible and cleanup is reliable.

            def close(self):
                self._sock.close()
                self.thread.join(2)

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cassandra/connection.py:998

• This clean-close exception is still raised before reaching this branch for AsyncoreConnection: its close() assigns a generated ConnectionShutdown to last_error while startup is pending (cassandra/io/asyncorereactor.py:392-397), and factory() raises any last_error at line 987. Asyncore is the documented default when libev is unavailable (cassandra/cluster.py:953-956), so those users do not get the restored clean-close behavior described here. The synthetic connection in the new test leaves last_error unset and therefore misses this reactor-specific path; distinguish a clean-close-generated shutdown from a real startup error and cover Asyncore as well.

        elif conn.is_closed and (host_conn is not None or raise_on_startup_close):
            raise ConnectionShutdown(
                "Connection to %s was closed during the startup handshake" % (endpoint,),
                endpoint)