-mm=gc shared library: a foreign thread's GC_allow_register_threads races a registered thread's allocation #523

For a -mm=gc shared library called by a host that is not a tslang program, the collector's threads are now enabled by the first unregistered thread that calls in. GC_allow_register_threads does its work before the allocator starts locking, so a registered thread allocating at that moment races it.

Background

Since #521, __tslang_gc_enter (lib/AsyncGCThreadsCommon.inc) runs GC_enable_threads only for a thread it has to register. It must not run on the thread the library loads on: there it was inside DllMain and deadlocked. In bdwgc 8.2.12, GC_allow_register_threads (win32_threads.c; pthread_support.c is alike) does this:

GC_start_mark_threads();   /* LOCK() is still a no-op: GC_need_to_lock is FALSE */
set_need_to_lock();

GC_start_mark_threads takes the allocation lock, which does nothing yet. It then changes collector state (GC_markers_m1, the mark-thread events, the main local mark stack via GET_MEM) and waits for the markers. Only then does it switch locking on.

Effect

A host whose loading thread (registered by GC_init) keeps calling into the library while its first worker thread arrives: for example an Android app's main thread plus JNI worker threads, or a C++ host with a thread pool. The registered thread's GC_malloc runs unlocked at the same time as the marker start-up on the foreign thread, and the collector state can end up corrupted. The window is narrow and happens once per process.

#516 had the same window whenever a foreign thread happened to be the first caller. #521 makes the foreign thread's call the only path that enables threads for a foreign host.

Not covered by tests

test-compile-foreign-thread-gc finishes all calls on the loading thread (run(0)) before it starts the pool, then waits in join(), so the two never overlap.

Direction

Same as the async-pool gap: enable threads from a registered thread at a point known to be after the load finished. The thread that enables them should be the one that is already allocating, not a newcomer, and locking should be on before any second thread enters. Alternatively, on Windows, call GC_enable_threads on the first call that is not under the loader lock.

Related: #516, #521, and the async-pool gap (filed beside this).