Skip to content

[DO NOT MERGE] Test runner: hang watchdog (rebased for the master hang) - #1848

Open
pa-devbot wants to merge 9 commits into
mainfrom
hang-watchdog-master-debug
Open

pa-devbot wants to merge 9 commits into
mainfrom
hang-watchdog-master-debug

Conversation

@pa-devbot

@pa-devbot pa-devbot commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

What

Same hang watchdog as the earlier #1840, rebased onto current main
(e2ae30190). If no test has completed within 60 s (overridable via
PA_HANG_WATCHDOG, 0 disables), it dumps the runner/pool/FileLogger
state plus every thread's stack trace to raw stderr and exits 124.

Why

Main's CI keeps hanging near the end of the suite (6h silence, then
manual cancel), with a different small set of tests stuck each time:

  • master e2ae301 Windows Default: Sockets (Qt) - Send/close race, ListenerSet - remove() of a running listener,
    ListenerSet - try_remove(), BlackBorderDetector - Dark_SV_Crystal_False.png
  • Add gamepad support #1844 Windows x2: ListenerSet pair again, plus Sockets (Qt) - Large transfer, Sockets (WinSocket) - close() cancels an in-flight send, AudioFile, PokemonHome readers
  • Max Lair: test the current boss inference (no behavior change) #1846 Ubuntu clang: 3 Audio tests, MMOSpriteMatcher,
    Sockets (POSIX) - Client/server echo round trip

Every stuck set includes a Sockets test, and the two ListenerSet tests
hang on 3 of the 4 runs. The watchdog will name the exact parked frame
on the next hang instead of leaving a 6h silence.

Status

Cherry-picked verbatim from the reviewed #1840 commits; builds clean
against main. [DO NOT MERGE] while we root-cause; the watchdog
itself is the mergeable content once this is done. (CI symbolization
flags - PDBs/-g/-rdynamic/build uploads - still need a
workflow-scope push; macOS symbolizes already.)

If no test COMPLETES for 60 s (PA_HANG_WATCHDOG overrides in seconds;
0 disables), the command-line test runner now dumps its own state
(pending/running/completed counts, per-running-test elapsed time, thread
pool queue/busy counts, file logger queue depth) and a stack trace of
every thread in the process straight to the stderr file descriptor, then
exits with code 124. This turns a hung CI runner - which would otherwise
sit until the 8 h job timeout - into a failed job with a full diagnostic
dump.

- HangWatchdog: dedicated watchdog thread (not on the test pools, so it
  fires even when every test thread is blocked). POSIX dumps capture
  each thread's backtrace() by sending it SIGUSR2 (Linux signals every
  /proc tid; macOS signals a registry of program-created threads);
  Windows suspends each thread and walks it with StackWalk64 + SymFromAddr.
- ParallelUnitTestRunner: diagnostics_state() snapshot and
  note_test_completed() per finished test.
- ThreadPool: queued_tasks()/busy_threads() accessors; FileLogger:
  pending_count()/max_queue_size().

CI note (separate change, needs workflow-scope push): to make the dump
readable in CI logs, keep the Windows PDBs, add -g on ubuntu-clang and
-rdynamic on Linux/macOS, and upload the Linux/macOS builds for offline
addr2line. Without those, frames resolve to module+offset only.

[DEBUG: do not merge] -- part of the CI-hang investigation; see
qt-socket-close-send-race-only for the suspected root-cause fix.
…gistry

THREADLIST32/Thread32First are not available in the CI SDK/MSVC
configuration (and SymFunction is not in the public dbghelp header, and
STACKFRAME64 has no Machine field on x64), so enumerate the dump's
threads from the program's own registry instead (register_thread() now
records the thread id on Windows, as it does on macOS): the runner,
pool, event-thread and logger threads - i.e., the ones that matter for a
hang - are all program-created and registered. Library-created threads
are not captured, noted in the dump as on macOS.

The StackWalk64 callback is now a local stub that just continues.
The CI SDK's dbghelp.h has the 9-parameter StackWalk64 (no frame-walk
callback, stack frame before context) and lacks the PSTACKFRAME64 /
PSYMBOL_INFO pointer typedefs, so drop the callback entirely and stop
using the pointer types (SYMBOL_INFO / STACKFRAME64 are used by value).
…x StackWalk64 args

Linux/macOS dumps now resolve each frame through dladdr() (demangled, with
the raw address as a fallback for offline symbolization). To make dladdr
able to name frames inside the test executables, link them with -rdynamic
on Linux so the full symbol table is exported to .dynsym.

Each thread header now shows its scheduling policy/priority/nice
(pidfd_open + sched_getattr, called as raw syscalls), which will show
whether the pool threads actually got their SCHED_RR policy on a CI
runner or fell back to SCHED_OTHER.

On Windows, StackWalk64 was receiving SymGetModuleBase64 in the
TranslateAddress slot; with the callback in the right slot the walk can
find module bases and produce frames. When it still produces none, the
dump now reports GetLastError() and the Rip.
…slot

The previous commit moved it into the GetFunctionTable slot and used '#'
for the slot comments, which MSVC rejects. Restore the 9-arg form that
the last Windows build verified, with // comments.
…lk64 has no PDB

CI Windows builds ship without PDBs, so StackWalk64 returns no frames
(error 0) for every thread. When that happens, scan the top 24 QWORDs of
the suspended thread's stack and report each value that points into
executable memory as '<module> + <offset>'. No function names are
possible without symbols, but the module pattern (SerialPrograms.exe,
Qt6Network.dll, ntdll.dll, ...) still identifies where each thread is
parked.
The CI build defines no _WIN32_WINNT, so the SDK hides the Vista-era
psapi declarations (EnumProcessModulesEx/GetModuleInfo/MODULEINFO) and
QWORD is not visible. EnumProcessModules (all versions) plus
SymGetModuleInfo64 (already used) covers the same ground, and UINT64
replaces QWORD.
IMAGEHLP_MODULE64 has no SizeOfImage field on this SDK, so drop the size
check: modules never overlap, so the module with the largest base at or
below the address is the one containing it. Also fix a use of the loop
variable outside its scope (use the matched module instead).
Scan 128 stack slots (cap 20 reported) instead of 24/10: MSVC x64
stacks have non-executable garbage between real return addresses, so
the caller frames of a parked thread sit well past the first 24 slots.
Also report each thread's Rip attributed to a module, which catches
threads parked in user code (spin loops) that the stack scan alone
would not.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant