Skip to content

Keep a departed Writer's samples readable by a Reliable DataReader - #475

Open
bobdingAI wants to merge 1 commit into
Atostek:masterfrom
bobdingAI:forget-writer-keeps-samples
Open

bobdingAI wants to merge 1 commit into
Atostek:masterfrom
bobdingAI:forget-writer-keeps-samples

Conversation

@bobdingAI

Copy link
Copy Markdown

Problem

When a reliable Writer goes away, Reader::remove_writer_proxy calls TopicCache::forget_reliable_writer, which removes the Writer's received_reliably_before marker. With no marker, reliable_before() falls back to SequenceNumber::default(), which is 1. So get_changes_in_range_reliable hides every sample from that Writer that the application has not taken yet:

  • samples received in full, which were already deliverable;
  • samples withheld behind a missing one, which the comment on forget_reliable_writer says are no longer awaited, so they should be handed over.

As a result, a sample can reach a matched reliable Reader, be acknowledged, and never be delivered. This happens whenever the Writer leaves before the application takes the sample, for example a ROS 2 node that publishes once and exits. We hit it in dora (dora-rs/dora#4213), which reads through ros2-client.

Measured on one host with Humble and Fast-DDS. Each trial starts a fresh dora subscriber and a fresh rclpy publisher that publishes one message as soon as it sees the subscription, then exits. The application starts taking a few tens of ms after the subscription is created:

RustDDS Fast-DDS publisher CycloneDDS publisher
master 0/8 0/8
this PR 21/24 8/8

A native rclpy subscriber under the same conditions gets 16/16 with Fast-DDS.

In each of the 3 trials that remain lost, the Reader had not matched the Writer when the DATA arrived: no ACKNACK from it to that Writer precedes the DATA. That is the discovery race, a separate issue. In all 21 received Fast-DDS trials, the Reader's first ACKNACK came 0.8–5 ms before the DATA.

In a packet capture of a trial lost on master, the DATA reaches the Reader about 5 ms after the Reader's preemptive ACKNACK, so the Writer is matched. The Writer is disposed 8 ms later, and take never returns the sample.

Change

forget_reliable_writer now moves the marker past the last sample of that Writer still in the cache, so all of them can be taken, including withheld ones.

  • The marker is dropped together with the Writer's last sample, so departed Writers leave nothing behind. It is also dropped at once when no sample is present.
  • A Writer that is matched again (mark_reliably_received_before) is no longer treated as departed.

Tests

Three new dds_cache unit tests. The first two fail on master:

  • received_samples_stay_readable_after_their_writer_is_forgotten: sample 1 is readable before forget_reliable_writer and must stay readable after it.
  • withheld_samples_are_handed_over_when_their_writer_is_forgotten: samples 2 and 3, withheld behind a missing 1, are handed over.
  • a_forgotten_writer_leaves_no_marker_once_its_samples_are_gone: once garbage collection removes the departed Writer's last sample, its marker is gone too.

cargo test --lib passes. network::udp_sender::tests::unicast_socket_rejects_ipv6_destination binds a fixed port (11401) and occasionally fails when the tests run in parallel. That is pre-existing, and it passes alone.

🤖 Generated with Claude Code

TopicCache::forget_reliable_writer removed the Writer's
received_reliably_before marker. reliable_before() then falls back to
SequenceNumber::default() (1), so get_changes_in_range_reliable hid every
sample from that Writer that the application had not taken yet: samples
received in full, and samples withheld behind a missing one, which the
comment says are no longer awaited.

This loses data whenever a Writer leaves soon after writing, before the
application takes the sample: e.g. a ROS 2 node that publishes once and
exits. The sample has reached a matched Reader and is acknowledged, but is
never delivered.

Now the marker moves past the last sample still present, so all of them can
be taken. The marker is dropped with the Writer's last sample, or at once
when none is present, and a Writer that is matched again is treated as
present.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant