Repository navigation
Conversation
If no test COMPLETES for 60 s (PA_HANG_WATCHDOG overrides in seconds; 0 disables), the command-line test runner now dumps its own state (pending/running/completed counts, per-running-test elapsed time, thread pool queue/busy counts, file logger queue depth) and a stack trace of every thread in the process straight to the stderr file descriptor, then exits with code 124. This turns a hung CI runner - which would otherwise sit until the 8 h job timeout - into a failed job with a full diagnostic dump. - HangWatchdog: dedicated watchdog thread (not on the test pools, so it fires even when every test thread is blocked). POSIX dumps capture each thread's backtrace() by sending it SIGUSR2 (Linux signals every /proc tid; macOS signals a registry of program-created threads); Windows suspends each thread and walks it with StackWalk64 + SymFromAddr. - ParallelUnitTestRunner: diagnostics_state() snapshot and note_test_completed() per finished test. - ThreadPool: queued_tasks()/busy_threads() accessors; FileLogger: pending_count()/max_queue_size(). CI note (separate change, needs workflow-scope push): to make the dump readable in CI logs, keep the Windows PDBs, add -g on ubuntu-clang and -rdynamic on Linux/macOS, and upload the Linux/macOS builds for offline addr2line. Without those, frames resolve to module+offset only. [DEBUG: do not merge] -- part of the CI-hang investigation; see qt-socket-close-send-race-only for the suspected root-cause fix.
…gistry THREADLIST32/Thread32First are not available in the CI SDK/MSVC configuration (and SymFunction is not in the public dbghelp header, and STACKFRAME64 has no Machine field on x64), so enumerate the dump's threads from the program's own registry instead (register_thread() now records the thread id on Windows, as it does on macOS): the runner, pool, event-thread and logger threads - i.e., the ones that matter for a hang - are all program-created and registered. Library-created threads are not captured, noted in the dump as on macOS. The StackWalk64 callback is now a local stub that just continues.
The CI SDK's dbghelp.h has the 9-parameter StackWalk64 (no frame-walk callback, stack frame before context) and lacks the PSTACKFRAME64 / PSYMBOL_INFO pointer typedefs, so drop the callback entirely and stop using the pointer types (SYMBOL_INFO / STACKFRAME64 are used by value).
…x StackWalk64 args Linux/macOS dumps now resolve each frame through dladdr() (demangled, with the raw address as a fallback for offline symbolization). To make dladdr able to name frames inside the test executables, link them with -rdynamic on Linux so the full symbol table is exported to .dynsym. Each thread header now shows its scheduling policy/priority/nice (pidfd_open + sched_getattr, called as raw syscalls), which will show whether the pool threads actually got their SCHED_RR policy on a CI runner or fell back to SCHED_OTHER. On Windows, StackWalk64 was receiving SymGetModuleBase64 in the TranslateAddress slot; with the callback in the right slot the walk can find module bases and produce frames. When it still produces none, the dump now reports GetLastError() and the Rip.
…slot The previous commit moved it into the GetFunctionTable slot and used '#' for the slot comments, which MSVC rejects. Restore the 9-arg form that the last Windows build verified, with // comments.
…lk64 has no PDB CI Windows builds ship without PDBs, so StackWalk64 returns no frames (error 0) for every thread. When that happens, scan the top 24 QWORDs of the suspended thread's stack and report each value that points into executable memory as '<module> + <offset>'. No function names are possible without symbols, but the module pattern (SerialPrograms.exe, Qt6Network.dll, ntdll.dll, ...) still identifies where each thread is parked.
The CI build defines no _WIN32_WINNT, so the SDK hides the Vista-era psapi declarations (EnumProcessModulesEx/GetModuleInfo/MODULEINFO) and QWORD is not visible. EnumProcessModules (all versions) plus SymGetModuleInfo64 (already used) covers the same ground, and UINT64 replaces QWORD.
IMAGEHLP_MODULE64 has no SizeOfImage field on this SDK, so drop the size check: modules never overlap, so the module with the largest base at or below the address is the one containing it. Also fix a use of the loop variable outside its scope (use the matched module instead).
Scan 128 stack slots (cap 20 reported) instead of 24/10: MSVC x64 stacks have non-executable garbage between real return addresses, so the caller frames of a parked thread sit well past the first 24 slots. Also report each thread's Rip attributed to a module, which catches threads parked in user code (spin loops) that the stack scan alone would not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Same hang watchdog as the earlier #1840, rebased onto current main
(
e2ae30190). If no test has completed within 60 s (overridable viaPA_HANG_WATCHDOG, 0 disables), it dumps the runner/pool/FileLoggerstate plus every thread's stack trace to raw stderr and exits 124.
Why
Main's CI keeps hanging near the end of the suite (6h silence, then
manual cancel), with a different small set of tests stuck each time:
e2ae301Windows Default:Sockets (Qt) - Send/close race,ListenerSet - remove() of a running listener,ListenerSet - try_remove(),BlackBorderDetector - Dark_SV_Crystal_False.pngSockets (Qt) - Large transfer,Sockets (WinSocket) - close() cancels an in-flight send, AudioFile, PokemonHome readersSockets (POSIX) - Client/server echo round tripEvery stuck set includes a Sockets test, and the two ListenerSet tests
hang on 3 of the 4 runs. The watchdog will name the exact parked frame
on the next hang instead of leaving a 6h silence.
Status
Cherry-picked verbatim from the reviewed #1840 commits; builds clean
against main.
[DO NOT MERGE]while we root-cause; the watchdogitself is the mergeable content once this is done. (CI symbolization
flags - PDBs/-g/-rdynamic/build uploads - still need a
workflow-scope push; macOS symbolizes already.)