Repository navigation
Do not wait for acknowledgments from best-effort readers in the heartbeat tick - #472
Open
johannesschrimpf wants to merge 2 commits into
Open
johannesschrimpf wants to merge 2 commits into
johannesschrimpf wants to merge 2 commits into
Conversation
Writer::handle_heartbeat_tick() decides whether a reliable Writer sends its periodic HEARTBEAT, and whether the next one comes after the fast or the slow period, by checking that every matched reader proxy has acknowledged the last change. A best-effort reader never sends an ACKNACK, so its proxy stays at all_acked_before == 0 and the check never holds. A reliable Writer with a best-effort reader matched therefore sent a non-final HEARTBEAT to multicast every HEARTBEAT_PERIOD_FAST (100 ms) for as long as that reader was matched, also when every reliable reader had acknowledged everything, and when the best-effort reader was the only one. Reliable writers with best-effort readers are common in ROS 2, e.g. a sensor topic read with the sensor-data QoS profile. RTPS 2.5 lets a reliable Writer be matched with a best-effort Reader (8.4.4). The best-effort Reader behaviors (8.4.11.1, 8.4.12.1) have no HEARTBEAT transition, so such a reader does nothing with these HEARTBEATs; RustDDS's own Reader ignores them too. Fast DDS marks a change acknowledged for a best-effort reader as soon as it is sent (StatefulWriter: acked_changes_set() when !is_reliable()), so its periodic heartbeat does not see such a reader as unacknowledged. The tick now looks at reliable reader proxies only, as refresh_acked_frontier() already does. A manual liveliness assertion (is_manual_assertion) keeps considering every reader, so what it sends is unchanged. Tests, all failing without the change: - best_effort_reader_does_not_keep_the_heartbeat_going - reliable_reader_behind_keeps_the_heartbeat_going_beside_a_best_effort_one - manual_liveliness_assertion_still_considers_best_effort_readers Assisted-by: Claude Fable 5.2
A remote participant can declare in its SPDP data that its BuiltinParticipantMessageReader is best-effort (builtinEndpointQos, RTPS 2.5 8.4.13.3). DPEventLoop::update_participant() has a special case for that: it sets the reader's QoS to best-effort, so that the local BuiltinParticipantMessageWriter knows not to expect ACKNACKs from it. But the reader proxy was built before the special case, from the QoS of the built-in endpoint list, which is reliable. The changed QoS only went into the QoS compatibility check of update_reader_proxy(). The proxy that the writer keeps said reliable, and the writer goes by its proxies: - refresh_acked_frontier() counted the reader as a reliable reader that has acknowledged nothing. - handle_heartbeat_tick() looks at reliable reader proxies only, since the previous commit. It took this reader for one that is behind, so the periodic HEARTBEAT kept going, at HEARTBEAT_PERIOD_FAST, for as long as the participant was there. Build the proxy after the special case, with the QoS that it sets. Test, failing without the change: - best_effort_participant_message_reader_does_not_keep_the_heartbeat_going: the event loop learns about a participant that declares the reader best-effort. The local BuiltinParticipantMessageWriter is matched with it, and its HEARTBEAT tick finds no reader to wait for. A participant that does not declare it is the control: its reader keeps the HEARTBEAT going until it is lost. The test builds a DPEventLoop without UDP listeners. test_event_loop() is new for that: the event loop tests had no such fixture. Found by code review, not in a capture: no participant in our captures sends PID_BUILTIN_ENDPOINT_QOS. Validation on macOS: - cargo test -- --test-threads=1, with and without --features security - cargo clippy --all-targets -- -D warnings, with and without --features security - cargo +nightly fmt --check - cargo check --all-targets --target armv7-unknown-linux-gnueabihf Assisted-by: Claude Fable 5.2
This was referenced Nov 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two commits. The first makes the heartbeat tick look at reliable readers only. The second gives one built-in reader, which a remote participant can declare best-effort, a proxy that says so.
Found while working on #471, which fixes a HEARTBEAT storm with a different cause (a reader proxy that nothing removes). The two PRs are independent: they touch different files, and either can be merged first.
Problem
1. Any best-effort reader keeps the fast HEARTBEAT going
A reliable Writer that is matched with a best-effort reader sends a non-final HEARTBEAT to multicast every
HEARTBEAT_PERIOD_FAST(100 ms) for as long as that reader is matched. It does so when every reliable reader has acknowledged everything, and when the best-effort reader is the only reader.Writer::handle_heartbeat_tick()(src/rtps/writer.rs) decides two things by checking that every matched reader proxy has acknowledged the last change: whether to send the periodic HEARTBEAT, and whether the next tick comes after the fast or the slow period (the adaptive rate from commit 792792f). A best-effort reader never sends an ACKNACK, so its proxy stays atall_acked_before == 0and the check never holds.Reliable writers with best-effort readers are common in ROS 2, for example a topic published with the default QoS and read with the sensor-data profile. Every live reliable reader answers each of these HEARTBEATs with an ACKNACK, so the cost is on the whole domain.
2. A ParticipantMessage reader declared best-effort is kept as reliable
builtinEndpointQos).DPEventLoop::update_participant()(src/rtps/dp_event_loop.rs) has a special case for that. It sets the reader's QoS to best-effort, so that the local BuiltinParticipantMessageWriter does not expect ACKNACKs.refresh_acked_frontier()counted it in the same way.Neither case occurs in our system by itself: no writer has the first pairing today, and no participant in our captures sends
PID_BUILTIN_ENDPOINT_QOS. We did check the first one on our robot (i.MX6) by starting a best-effort subscriber by hand,ros2 topic echo --qos-reliability best_effort, on topics that a RustDDS node publishes reliably:What remains is the HEARTBEAT that the Writer sends along with each DATA. Each figure is one 15 s measurement, taken with this change on our 0.14.3-based branch.
What the specification says (DDSI-RTPS 2.5)
Reader::handle_heartbeat_msg).builtinEndpointQosis in the SPDP data, "the BuiltinParticipantMessageWriter shall treat the BuiltinParticipantMessageReader as indicated by the flags" (8.4.13.3). The second commit is about that rule.Comparison with Fast DDS (v2.14.7, v3.6.2)
StatefulWritermarks a change acknowledged for a best-effort reader as soon as it is delivered (acked_changes_set()when!is_reliable()).send_periodic_heartbeat()asks each reader proxyhas_unacknowledged(), so a best-effort reader never keeps the periodic heartbeat going.Fix
handle_heartbeat_tick()looks at reliable reader proxies only, with the same filterrefresh_acked_frontier()already uses (rp.qos().is_reliable()).is_manual_assertion) keeps considering every matched reader, so what it sends is unchanged. Whether a liveliness HEARTBEAT should go out when every reader is caught up is a separate question, and this PR does not touch it.update_participant()builds the built-in reader proxy after the special case, with the QoS that it sets. The lines moved; nothing else changed.Not changed: the unicast HEARTBEAT a reliable Writer sends to a newly matched reader also goes to a best-effort one. That is one datagram per match.
Tests
All fail without the change they belong to.
First commit:
best_effort_reader_does_not_keep_the_heartbeat_going: a reliable Writer with samples and only a best-effort reader. The tick returns false.reliable_reader_behind_keeps_the_heartbeat_going_beside_a_best_effort_one: the tick returns true while the reliable reader has acknowledged nothing, and false once it has acknowledged everything.manual_liveliness_assertion_still_considers_best_effort_readers: the manual path behaves as before.Second commit:
best_effort_participant_message_reader_does_not_keep_the_heartbeat_going: the event loop learns about a participant that declares the reader best-effort. The local BuiltinParticipantMessageWriter is matched with it, and its tick returns false. A participant that does not declare it is the control: the tick returns true until that participant is lost.Applicability
acknack_schedules_repair_after_nack_response_delayand use the fixture's two extra arguments (payload_len,reader_locator).test_event_loopfixture, because master's event loop tests have none, and theWriterIngredientsthere have no doorbell fields.Validation (macOS)
cargo test -- --test-threads=1, with and without--features security: all pass.cargo clippy --all-targets -- -D warnings, with and without--features security: clean at each commit on master, and on 0.14.3.cargo +nightly fmt --check: clean at each commit on master, and on 0.14.3.cargo check --all-targets --target armv7-unknown-linux-gnueabihf: clean at each commit on master, and on 0.14.3.Note on #473
One line depends on the merge order with #473. That PR adds a field to
WriterIngredients, and the test of the second commit here builds that struct (src/rtps/dp_event_loop.rs). The two merge without a conflict, but the tests of the combined tree do not compile (E0063) until that fixture hasmatched_counts: Default::default(),. Whichever of the two is merged second needs the line; I can add it here on a rebase if #473 goes first.