[BUG] Test DDS domains map to UDP ports inside the kernel ephemeral range

[bburda]

Summary

Integration tests get a DDS domain from ROS2MedkitTestDomain.cmake. CycloneDDS turns a domain into a UDP discovery port with 7400 + 250 * domain. Most of the domains we hand out land inside the port range the Linux kernel uses for ephemeral ports, so any process on the machine can be given one of our ports and the whole domain stops working.

When that happens the node dies at startup:

ddsi_udp_create_conn: failed to bind to ANY:53150: address in use
[ERROR] [rmw_cyclonedds_cpp]: rmw_create_node: failed to create domain, error Error
Fatal exception in main: failed to initialize rcl node: rcl node's rmw handle is invalid

The test then fails with Exception: Launch stopped before the active tests finished, which says nothing about the real cause. Every case in the file fails at once, because the fixture never came up.

The overlap

The kernel ephemeral range on our runners is 32768-60999. That maps back to domains 102 to 214.

package	domain range	domains inside the ephemeral range
ros2_medkit_integration_tests	130-219	85 of 90
ros2_medkit_linux_introspection	130-139	10 of 10
ros2_medkit_opcua	220-229	0 of 10
ros2_medkit_gateway	30-89	0 of 60

Only domains <= 101 and >= 215 are safe. That is 120 domains in total, and we currently allocate 220.

How to reproduce

Hold the port with a plain socket, then try to create a node on the matching domain:

import socket, rclpy
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(("", 53150))            # 7400 + 250 * 183
rclpy.init(args=["--ros-args"])  # with ROS_DOMAIN_ID=183, RMW=rmw_cyclonedds_cpp
rclpy.create_node("probe")     # fails with the message above

Release the port and the same call works.

Why it was not caught before

Earlier work on test domains fixed collisions between our own tests - two tests sharing a domain and seeing each other's nodes. This is a different problem with the same symptom. The port is taken by a process outside our test run, so nothing in our allocation can see it coming. It is rare per run (I found it in 1 of 13 recent Humble jobs), which is why it reads as a flaky test.

Proposed solution

Move the allocation into the safe band, or stop the kernel from handing out our ports. Two options:

1. Re-carve ROS2MedkitTestDomain.cmake so every package draws from <= 101 or >= 215. 120 domains is fewer than the 220 we allocate today, so packages would need smaller pools. ctest runs serially inside a package, so a pool per package rather than a domain per test is enough.
2. Reserve the ports in CI with net.ipv4.ip_local_reserved_ports on the container. Smaller change, but it only helps CI, not a developer machine, and it needs checking against Fast-DDS unicast ports too.

Option 1 fixes it everywhere.

Additional context

Adding a test file to test/features/ shifts the domain of every test that sorts after it, because the registration uses file(GLOB ...) and assigns domains in order. So a test that has been stable for months can start failing after an unrelated test is added, with no change to the test itself.