Skip to content

Sockets: fix the start/close races that hang the test runner - #1849

Merged
Mysticial merged 3 commits into
mainfrom
sockets-hang-fix
Nov 6, 2026
Merged

Mysticial merged 3 commits into
mainfrom
sockets-hang-fix

Conversation

@pa-devbot

Copy link
Copy Markdown
Collaborator

What

Follow-up to #1832 / the master-hang investigation (watchdog PR #1848). While chasing the intermittent stuck CI runners, I found three independent defects that can each hang a socket test forever; a hung test is what stalls the parallel runner until the 8 h job timeout.

1) ServerSocket_Qt::start() left m_starting set on the rejected path

m_starting was set before the "am I allowed to start" check. make_server() keeps the QTcpServer even when listen() fails, so every retry of the test's start-retry loop was rejected with m_starting still true — and the destructor deadlocks in close(), which waits on !m_starting. This matches the Clang CI hang in Sockets (Qt) - Completion lifetime under event-thread stall (test thread parked in close()'s first cv wait, server already listening from an earlier attempt).

Fix: only set the flag when a start actually begins, and on a failed listen destroy the server so the object is left exactly as it was (the POSIX/WinSocket backends already behave that way: a failed listen leaves no listener behind).

2) start() destroyed a still-running accept task (POSIX + WinSocket)

start() stored the new accept task in the single m_accept_thread slot; the assignment destroys the previous AsyncTask, whose destructor blocks until that task finishes. When the caller's wait timed out while the pool was starved, the old task was still running — and a successful old task had entered the accept loop, which only exits via close(). The caller was stuck in start() and could never reach close() → permanent hang (the MSVC CI hang in Sockets (WinSocket) - Send/close race).

Fix: start() appends to a vector of tasks; close() joins them all. Each accept task captures a generation number; a task that notices it was replaced stays silent (its listen result would corrupt the attempt that superseded it) and exits. The poll-loop exit check includes the generation too.

3) The test harness's start_server() retry loop read stale state

The loop waited on listen_finished > 0 — a counter that is never reset between attempts. After a transient bind failure, the next attempt's wait returned immediately with the previous attempt's result. Either the stale error was misread as this attempt's failure, or — worse — the server was already LISTENING on the previous attempt's port, the new start() was rejected, and the loop handed back a port the server was not actually listening on. The client then got "Connection refused" and the test spun through a full 60 s wait on a server that would never see the connection.

Fix: each attempt records the counter baseline and waits for it to advance past that baseline.

Verification

  • Local (aarch64, Linux, POSIX + Qt backends): 24 back-to-back Sockets-suite rounds under CPU load (tests pinned to 4 cores, 6 busy spinners on the others), 25/25 tests each round. Before the fixes the same loop flaked every 2–7 rounds with exactly these symptoms (stale-error misread, wrong-port "Connection refused", 60 s on_connect waits).
  • Full suite under the same load: 20,330 passed, 0 watchdog lines, clean exit (the 5 failing OCR tests fail on this sandbox without the PaddleOCR model — pre-existing, unrelated).
  • Windows (WinSocket) and the Qt paths: same code shapes; will be covered by the CI Windows Clang/MSVC jobs.

pa-devbot and others added 3 commits November 5, 2026 19:02
Three independent defects could hang a socket test forever; on CI they
show up as a stuck runner (the parallel UnitTestRunner never finishes,
the job sits until the 8 h timeout).

1) ServerSocket_Qt::start() set m_starting = true before checking
   whether the call was actually accepted. make_server() keeps the
   QTcpServer even when listen() fails, so every retry of the
   start-server loop was rejected with m_starting still set. The
   destructor then deadlocked in close(), which waits for
   !m_starting. Only set the flag when a start actually begins, and on
   a failed listen destroy the server so the object is left exactly as
   it was (matching the POSIX/WinSocket backends, where a failed listen
   leaves no listener behind).

2) ServerSocket_POSIX / ServerSocket_WinSocket: start() stored the new
   accept task in the single m_accept_thread slot. The assignment
   destroys the previous AsyncTask, and its destructor blocks until
   that task has finished. When the caller's wait timed out while the
   pool was starved, the old task was still running: a successful old
   task had entered the accept loop and would only exit via close(),
   which the caller could not reach because it was stuck in start() -
   a permanent hang. start() now appends to a vector of tasks and
   close() joins them all. Each accept task captures a generation
   number; a task that notices it was replaced stays silent (its listen
   result would corrupt the attempt that superseded it) and exits.

3) The tests' start_server() retry loop waited on
   "listen_finished > 0", a counter that is never reset between
   attempts. After a transient bind failure the wait returned
   immediately on the next attempt with the previous attempt's result:
   either the stale error was misread as this attempt's failure, or -
   worse - the server was already LISTENING on the previous attempt's
   port, the new start() was rejected, and the loop handed back a port
   the server was not actually listening on. The client then got
   "Connection refused" and the test spun for a full 60 s wait on a
   server that would never see the connection. Each attempt now waits
   for the counter to advance past its own baseline.
MSVC's flow analysis does not track that "listening" is read only when
"destructing" is false (short-circuit on the second condition), and
treats the warning as an error on CI. Initialize both locals; the
destructing==true path never uses "listening".
@Mysticial
Mysticial merged commit 73b7423 into main Nov 6, 2026
7 checks passed
@Mysticial
Mysticial deleted the sockets-hang-fix branch November 6, 2026 03:40
@pa-devbot pa-devbot mentioned this pull request Nov 6, 2026
5 of 7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants