Keep the image alive while the C detector reads it

[yakir12]

Summary

convert(image_u8_t, image) builds a temporary row-major copy of the image (image'[:]) and returns an image_u8_t holding only a raw pointer into it. Nothing roots that copy. If a garbage collection runs on another thread while apriltag_detector_detect is reading it, the copy is freed under the detector. Tags then go missing on good frames, or the process segfaults when the memory was returned to the OS.

Detection now goes through a small helper that holds the copy for the whole call:

 (detector::AprilTagDetector)(image)        # and threadcalldetect, detectAndPose
-  image8 = convert(image_u8_t, image)      # pointer into an unrooted temporary
-  detections = apriltag_detector_detect(detector.td, image8)
+  detections = with_image_u8(image) do image8
+      apriltag_detector_detect(detector.td, image8)
+  end

+with_image_u8(f, image)
+  imbuf = row-major copy of image
+  GC.@preserve imbuf f(image_u8_t(cols, rows, cols, pointer(imbuf)))

convert(::Type{image_u8_t}, …) is unchanged, for compatibility, with a comment that its pointer can dangle.

CI now runs the tests with JULIA_NUM_THREADS: 4. On one thread nothing can collect during the C call, so the new test would pass whatever the code did.

Evidence

Found in Fromage.jl, which detects tags in drone footage with several videos processed in parallel. The detections run under a global lock, but a lock cannot stop the GC. 16 runs lost their first registrations this way.

• A standalone script (--threads=8): 30 detections of a synthetic 3-tag scene while 4 tasks allocate and call GC.gc(false).
  • Before: the process segfaults (signal 11), at --check-bounds=yes and auto alike.
  • After: 0/30 missed in each of four configurations (scene ×1 and ×2, 4 and 8 churning tasks).
• On a 1080p drone frame, in a long-running session:
  • Before: 37/40 detections missed tags under the same churn, 0/40 without it.
  • After: 0/40.
• The new testset, "Detection survives garbage collection on other threads" (JULIA_NUM_THREADS=8, Pkg.test() on Julia 1.13.1): 300 detections while 8 tasks churn.
  • Before: it failed in one of two runs, and in that run every detection found 0 of 3 tags. Loading the test's image libraries makes the race less likely to show than in a plain script, so the test catches it only part of the time without the fix.
  • After: 68 passed.

Merge Danger

Door: two-way.

Blast Radius: detection

Every detection now makes one copy of the image, as before, and keeps it until the C call returns. Results are unchanged wherever the race did not strike. with_image_u8 accepts any AbstractMatrix of UInt8, N0f8 or Gray{N0f8}; the detector methods' signatures are unchanged.

🤖 Generated with Claude Code