Skip to content

Make loadLibrary()/unloadLibrary() safe against concurrent load and unload - #234

Merged
clalancette merged 1 commit into
ros:rollingfrom
otamachan:otamachan/thread-safe-plugin-load-unload
Sep 2, 2026
Merged

Make loadLibrary()/unloadLibrary() safe against concurrent load and unload#234
clalancette merged 1 commit into
ros:rollingfrom
otamachan:otamachan/thread-safe-plugin-load-unload

Conversation

@otamachan

@otamachan otamachan commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Problem

loadLibrary() and unloadLibrary() can run concurrently on the same library:

  • loadLibrary() locks only its own dlopen() and the final push_back onto the loaded
    library vector. The isLibraryLoadedByAnybody() test at the top is a check-then-act.
  • unloadLibrary() takes no lock at all while it erase()s from that same vector and drops
    the last reference to the shared library.

Two threads therefore mutate one std::vector concurrently, and a dlopen() can overlap a
dlclose(). That shows up either as

MultiLibraryClassLoader: Could not create class of type <T>

when the metaobjects go away between the check and the bind, or as a SIGSEGV in
unloadLibrary() -> rcpputils::SharedLibrary::unload_library() while another thread is
inside _dl_map_object_from_fd() for the same plugin.

ros2/rosbag2#2350 is hitting this. We hit it in production on Jazzy, where each
rosbag2_cpp::Writer owns a StorageFactory and so loads and unloads the mcap storage
plugin once per bag.

Change

Hold one recursive mutex across the whole of loadLibrary() and unloadLibrary(), take
getLoadedLibraryVectorMutex() around the erase, and guard the already-loaded branch with
getPluginBaseToFactoryMapMapMutex(). Lock order is loader mutex -> vector / factory map
everywhere, and the loader mutex is recursive so the plugin registration that runs from
inside dlopen() on the same thread is fine.

The last two hunks restore #40, which fixed #39 on indigo-devel in 2016 and reached
melodic-devel and noetic-devel but never the ROS 2 branches — those still carry the 2014
locking from 96ed2e4. #40 only touched the load side, so locking unloadLibrary() is needed
on top of that forward-port.

Verification

Two threads doing nothing but open/close an MCAP bag through rosbag2_cpp::Writer on Jazzy.
Only libclass_loader.so differs between rows.

class_loader 2 threads x 200 open/close, 5 runs
distro 2.7.0 5/5 SIGSEGV
self-built from this branch's parent 5/5 SIGSEGV
with this change 0/5

Soak on the patched build: 8 threads x 400, 3 runs, clean. Over 200 fresh processes each
doing one concurrent open per thread: 12/200 SIGSEGV before, 0/200 after. Single-threaded
behaviour is unchanged.

…nload

loadLibrary() locked only its own dlopen() and the final push_back onto the
loaded library vector, and unloadLibrary() took no lock at all while it erased
from that same vector and dropped the last reference to the shared library.
Two threads loading and unloading the same plugin therefore race: the loader
either fails to find the class and throws

  MultiLibraryClassLoader: Could not create class of type <T>

because the metaobjects were destroyed between the isLibraryLoadedByAnybody()
check and binding the new owner, or it segfaults inside the dlclose() path.

Hold one recursive mutex across the whole of both functions so a dlopen() can
never overlap a dlclose() of the same library, take the loaded library vector
mutex around the erase in unloadLibrary(), and guard the already-loaded branch
of loadLibrary() with the factory map mutex.

The last two hunks restore what ros#40 did on indigo-devel in 2016; that fix
reached melodic-devel and noetic-devel but was never forward-ported to the
ROS 2 branches, which still carry the 2014 locking. See ros#39 and
ros2/rosbag2#2350.

Signed-off-by: Tamaki Nishino <otamachan@gmail.com>

@clalancette clalancette left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems reasonable to me. I'll run CI on it next.

@clalancette

clalancette commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

CI:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Linux-rhel Build Status
  • Windows Build Status

@clalancette
clalancette merged commit 221ae79 into ros:rolling Sep 2, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Intermittent Failure to Load Library Under High Thread Contention

2 participants