A gc shared library loads without starting the collector's markers under the loader lock

[ASDAlexander77]

Every JIT run against a -mm=gc default library has hung at 0% CPU since #516, and the release workflow hung with it: run 37377707693 (v0.0-pre-alpha90) sat in "Test Compiler with Default Library" until it was cancelled.

Cause

#516 calls __tslang_gc_enter first in every exported function of a gc shared library, and its first call ran GC_enable_threads. An exported function can run while the library is being loaded. The default library's exported Number.static_constructor is a global constructor, and top-level code can also call an export. On Windows that happens inside DllMain, under the loader lock. GC_allow_register_threads starts the collector's parallel marker threads and waits for them to come up. A new thread can't start until the loader lock is released, so LoadLibrary never returned.

Stacks of the hung tslang --emit=jit (ProcDump + WinDbg):

main:  LoadLibraryExW(TypeScriptDefaultLib.dll) -> LdrpCallInitRoutine -> _initterm
       -> Number.static_constructor -> __tslang_gc_enter -> gc!GC_allow_register_threads
       -> gc!GC_start_mark_threads -> WaitForSingleObject
15 threads (the markers): LdrpInitializeThread -> LdrpDrainWorkQueue

GC_MARKERS=1 (no marker threads) makes the same run finish. A plain C++ DllMain that calls GC_allow_register_threads hangs the same way. --emit=exe links the static default library and runs no DllMain, so it was unaffected. The age of the bundled gc.dll is not involved: CI builds bdwgc with its default enable_parallel_mark=ON.

Fix

__tslang_gc_enter now enables threads only for a thread it has to register. The thread a library loads on is the one GC_init registered, or one its host registered, so loading no longer starts the markers. For that thread this is what every release before #516 did. The program's own injected GC_enable_threads still starts the markers, outside the loader lock.

Test

test-compile-foreign-thread-gc gets two more libraries. In one, an exported class has a static constructor; in the other, top-level code calls an export. Both allocate into a global, so --opt can't fold the call away: a first attempt whose top level called work(1) was folded to a constant and never hung. Both libraries hung before this change. A regression costs 2 × 240 s before the test fails.

Local (Windows, Release): full ctest 3895/3895. A default library built from DefaultLib main against this runtime passes test-jit-/test-compile-gc-defaultlib-collector, and tests.ps1 release compile 162/162 and release JIT 162/162. The debug pass, Linux and Android were not run locally.

Known gaps (follow-up)

1. A gc shared library's own async pool is no longer hooked when only registered threads call it. The thread hooks are per module, so the library's copy of the scheduler gets them only from its own __tslang_gc_enter. Its pool threads would then run coroutines unregistered, with the allocator unlocked. That matches every release before A gc shared library registers the threads its host calls it on #516, but A gc shared library registers the threads its host calls it on #516 had covered it.
2. For a foreign host, the markers now start from the first unregistered thread. GC_allow_register_threads starts them before it sets GC_need_to_lock, so a registered thread allocating at that moment races it. The window is once per process and narrow (A gc shared library registers the threads its host calls it on #516 had the same window when a foreign thread called first).

Both want threads enabled from a registered thread once the load has finished (a flag set when loading ends, or loader-lock detection), and that needs its own design.

🤖 Generated with Claude Code