Skip to content

Do not wait for acknowledgments from best-effort readers in the heartbeat tick - #472

Open
johannesschrimpf wants to merge 2 commits into
Atostek:masterfrom
johannesschrimpf:fix/heartbeat-best-effort-reader
Open

johannesschrimpf wants to merge 2 commits into
Atostek:masterfrom
johannesschrimpf:fix/heartbeat-best-effort-reader

Conversation

@johannesschrimpf

@johannesschrimpf johannesschrimpf commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Two commits. The first makes the heartbeat tick look at reliable readers only. The second gives one built-in reader, which a remote participant can declare best-effort, a proxy that says so.

Found while working on #471, which fixes a HEARTBEAT storm with a different cause (a reader proxy that nothing removes). The two PRs are independent: they touch different files, and either can be merged first.

Problem

1. Any best-effort reader keeps the fast HEARTBEAT going

A reliable Writer that is matched with a best-effort reader sends a non-final HEARTBEAT to multicast every HEARTBEAT_PERIOD_FAST (100 ms) for as long as that reader is matched. It does so when every reliable reader has acknowledged everything, and when the best-effort reader is the only reader.

Writer::handle_heartbeat_tick() (src/rtps/writer.rs) decides two things by checking that every matched reader proxy has acknowledged the last change: whether to send the periodic HEARTBEAT, and whether the next tick comes after the fast or the slow period (the adaptive rate from commit 792792f). A best-effort reader never sends an ACKNACK, so its proxy stays at all_acked_before == 0 and the check never holds.

Reliable writers with best-effort readers are common in ROS 2, for example a topic published with the default QoS and read with the sensor-data profile. Every live reliable reader answers each of these HEARTBEATs with an ACKNACK, so the cost is on the whole domain.

2. A ParticipantMessage reader declared best-effort is kept as reliable

  • A remote participant can declare in its SPDP data that its BuiltinParticipantMessageReader is best-effort (builtinEndpointQos).
  • DPEventLoop::update_participant() (src/rtps/dp_event_loop.rs) has a special case for that. It sets the reader's QoS to best-effort, so that the local BuiltinParticipantMessageWriter does not expect ACKNACKs.
  • But the reader proxy was built before the special case, from the QoS of the built-in endpoint list, which is reliable. The changed QoS only went into the QoS compatibility check.
  • The Writer goes by the proxy. With the first commit alone, this reader still counted as a reliable reader that is behind, so the BuiltinParticipantMessageWriter kept the fast HEARTBEAT for as long as that participant was there. refresh_acked_frontier() counted it in the same way.

Neither case occurs in our system by itself: no writer has the first pairing today, and no participant in our captures sends PID_BUILTIN_ENDPOINT_QOS. We did check the first one on our robot (i.MX6) by starting a best-effort subscriber by hand, ros2 topic echo --qos-reliability best_effort, on topics that a RustDDS node publishes reliably:

topic published at about HEARTBEATs/s from its writer, before with this change
10 Hz 19.9 9.9
100 Hz 110.1 100.0

What remains is the HEARTBEAT that the Writer sends along with each DATA. Each figure is one 15 s measurement, taken with this change on our 0.14.3-based branch.

What the specification says (DDSI-RTPS 2.5)

  • A reliable Writer may be matched with a best-effort Reader, and the Writer's behavior towards each matched Reader depends on both reliability levels (8.4.4).
  • The periodic HEARTBEAT is owed to reliable readers: "A Writer must periodically inform each matching reliable Reader" (8.4.2.2).
  • "A best-effort Reader is completely passive as it only receives data and does not send messages itself" (8.4.2.3). The best-effort Reader behaviors (8.4.11.1 StatelessReader, 8.4.12.1 StatefulReader) have no HEARTBEAT transition. RustDDS's own Reader ignores HEARTBEATs when it is best-effort (Reader::handle_heartbeat_msg).
  • The BuiltinParticipantMessageWriter is reliable, but the matching reader may be best-effort. If builtinEndpointQos is in the SPDP data, "the BuiltinParticipantMessageWriter shall treat the BuiltinParticipantMessageReader as indicated by the flags" (8.4.13.3). The second commit is about that rule.
  • For the first commit, the extra HEARTBEATs break no rule on the wire. They are traffic that the specification does not ask for.

Comparison with Fast DDS (v2.14.7, v3.6.2)

StatefulWriter marks a change acknowledged for a best-effort reader as soon as it is delivered (acked_changes_set() when !is_reliable()). send_periodic_heartbeat() asks each reader proxy has_unacknowledged(), so a best-effort reader never keeps the periodic heartbeat going.

Fix

  • First commit: handle_heartbeat_tick() looks at reliable reader proxies only, with the same filter refresh_acked_frontier() already uses (rp.qos().is_reliable()).
    • A manual liveliness assertion (is_manual_assertion) keeps considering every matched reader, so what it sends is unchanged. Whether a liveliness HEARTBEAT should go out when every reader is caught up is a separate question, and this PR does not touch it.
  • Second commit: update_participant() builds the built-in reader proxy after the special case, with the QoS that it sets. The lines moved; nothing else changed.

Not changed: the unicast HEARTBEAT a reliable Writer sends to a newly matched reader also goes to a best-effort one. That is one datagram per match.

Tests

All fail without the change they belong to.

First commit:

  • best_effort_reader_does_not_keep_the_heartbeat_going: a reliable Writer with samples and only a best-effort reader. The tick returns false.
  • reliable_reader_behind_keeps_the_heartbeat_going_beside_a_best_effort_one: the tick returns true while the reliable reader has acknowledged nothing, and false once it has acknowledged everything.
  • manual_liveliness_assertion_still_considers_best_effort_readers: the manual path behaves as before.

Second commit:

  • best_effort_participant_message_reader_does_not_keep_the_heartbeat_going: the event loop learns about a participant that declares the reader best-effort. The local BuiltinParticipantMessageWriter is matched with it, and its tick returns false. A participant that does not declare it is the control: the tick returns true until that participant is lost.

Applicability

  • Made and tested on the 0.14.3 line.
  • The library changes apply to master unchanged.
  • On master the first commit's tests go before acknack_schedules_repair_after_nack_response_delay and use the fixture's two extra arguments (payload_len, reader_locator).
  • On master the second commit's test brings its own test_event_loop fixture, because master's event loop tests have none, and the WriterIngredients there have no doorbell fields.

Validation (macOS)

  • cargo test -- --test-threads=1, with and without --features security: all pass.
    • master (01bc03a) + first commit: 714 / 724 lib tests, plus all integration and doc tests.
    • master + both commits: 715 / 725 lib tests, plus all integration and doc tests.
    • 0.14.3 + both commits: 708 / 718 lib tests, plus all integration and doc tests.
  • cargo clippy --all-targets -- -D warnings, with and without --features security: clean at each commit on master, and on 0.14.3.
  • cargo +nightly fmt --check: clean at each commit on master, and on 0.14.3.
  • cargo check --all-targets --target armv7-unknown-linux-gnueabihf: clean at each commit on master, and on 0.14.3.
  • The second commit's test was seen failing without the library change on both lines.

Note on #473

One line depends on the merge order with #473. That PR adds a field to WriterIngredients, and the test of the second commit here builds that struct (src/rtps/dp_event_loop.rs). The two merge without a conflict, but the tests of the combined tree do not compile (E0063) until that fixture has matched_counts: Default::default(),. Whichever of the two is merged second needs the line; I can add it here on a rebase if #473 goes first.

Writer::handle_heartbeat_tick() decides whether a reliable Writer sends
its periodic HEARTBEAT, and whether the next one comes after the fast or
the slow period, by checking that every matched reader proxy has
acknowledged the last change. A best-effort reader never sends an
ACKNACK, so its proxy stays at all_acked_before == 0 and the check never
holds. A reliable Writer with a best-effort reader matched therefore sent
a non-final HEARTBEAT to multicast every HEARTBEAT_PERIOD_FAST (100 ms)
for as long as that reader was matched, also when every reliable reader
had acknowledged everything, and when the best-effort reader was the only
one. Reliable writers with best-effort readers are common in ROS 2, e.g.
a sensor topic read with the sensor-data QoS profile.

RTPS 2.5 lets a reliable Writer be matched with a best-effort Reader
(8.4.4). The best-effort Reader behaviors (8.4.11.1, 8.4.12.1) have no
HEARTBEAT transition, so such a reader does nothing with these
HEARTBEATs; RustDDS's own Reader ignores them too. Fast DDS marks a
change acknowledged for a best-effort reader as soon as it is sent
(StatefulWriter: acked_changes_set() when !is_reliable()), so its
periodic heartbeat does not see such a reader as unacknowledged.

The tick now looks at reliable reader proxies only, as
refresh_acked_frontier() already does. A manual liveliness assertion
(is_manual_assertion) keeps considering every reader, so what it sends is
unchanged.

Tests, all failing without the change:
- best_effort_reader_does_not_keep_the_heartbeat_going
- reliable_reader_behind_keeps_the_heartbeat_going_beside_a_best_effort_one
- manual_liveliness_assertion_still_considers_best_effort_readers

Assisted-by: Claude Fable 5.2
A remote participant can declare in its SPDP data that its
BuiltinParticipantMessageReader is best-effort (builtinEndpointQos, RTPS
2.5 8.4.13.3). DPEventLoop::update_participant() has a special case for
that: it sets the reader's QoS to best-effort, so that the local
BuiltinParticipantMessageWriter knows not to expect ACKNACKs from it.

But the reader proxy was built before the special case, from the QoS of
the built-in endpoint list, which is reliable. The changed QoS only went
into the QoS compatibility check of update_reader_proxy(). The proxy
that the writer keeps said reliable, and the writer goes by its proxies:
- refresh_acked_frontier() counted the reader as a reliable reader that
  has acknowledged nothing.
- handle_heartbeat_tick() looks at reliable reader proxies only, since
  the previous commit. It took this reader for one that is behind, so
  the periodic HEARTBEAT kept going, at HEARTBEAT_PERIOD_FAST, for as
  long as the participant was there.

Build the proxy after the special case, with the QoS that it sets.

Test, failing without the change:
- best_effort_participant_message_reader_does_not_keep_the_heartbeat_going:
  the event loop learns about a participant that declares the reader
  best-effort. The local BuiltinParticipantMessageWriter is matched with
  it, and its HEARTBEAT tick finds no reader to wait for. A participant
  that does not declare it is the control: its reader keeps the
  HEARTBEAT going until it is lost.
The test builds a DPEventLoop without UDP listeners. test_event_loop()
is new for that: the event loop tests had no such fixture.

Found by code review, not in a capture: no participant in our captures
sends PID_BUILTIN_ENDPOINT_QOS.

Validation on macOS:
- cargo test -- --test-threads=1, with and without --features security
- cargo clippy --all-targets -- -D warnings, with and without
  --features security
- cargo +nightly fmt --check
- cargo check --all-targets --target armv7-unknown-linux-gnueabihf

Assisted-by: Claude Fable 5.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant