Repository navigation
Match remote endpoints only while their participant is known, and again when it returns - #471
Open
johannesschrimpf wants to merge 5 commits into
Open
johannesschrimpf wants to merge 5 commits into
johannesschrimpf wants to merge 5 commits into
Conversation
When a participant times out, DiscoveryDB moves its readers and writers to the "attic", because SEDP does not deliver them a second time: the SEDP DataReaders have read those samples already. Commit 203fd01 added this, to re-install the endpoints "without discovery messages" when the participant is rediscovered. But update_participant() only moved them back into the DB. No ReaderUpdated or WriterUpdated followed, so the local writers and readers, which had dropped their proxies at the timeout, were not matched with them again. A participant that returned, e.g. after a network outage longer than its lease, was known again, but the endpoints it had before stayed unmatched. restore_endpoints_from_attic() now enters the endpoints of a participant again and returns them. process_discovered_participant_data() calls it under the same DiscoveryDB lock as update_participant(), and sends ReaderUpdated and WriterUpdated for them after ParticipantUpdated. update_participant() no longer moves them back on its own. An endpoint restored from the attic is now matched, so two cases that the attic did not handle are fixed too: - An endpoint that is disposed over SEDP while its participant is timed out is removed from the attic. - A participant that is disposed after it timed out takes the endpoints it has in the attic with it. With security enabled, a participant is entered into the DB by SecureDiscovery while it is being authenticated. Its endpoints are entered again there, as before, and are not matched. Matching them once the participant is authenticated is left as a TODO. Tests: - endpoints_of_a_timed_out_participant_are_matched_again_when_it_returns: a participant with a reader and a writer is discovered, times out and is discovered again. If the endpoints are restored without being announced, as before, the local writer and the local reader are matched with them once each. Now they are matched twice. - discdb_endpoints_of_a_timed_out_participant_return_with_it - discdb_endpoint_disposed_while_its_participant_is_timed_out_does_not_return - discdb_dispose_of_a_timed_out_participant_drops_its_endpoints Validation on macOS: - cargo test -- --test-threads=1, with and without --features security - cargo clippy --all-targets -- -D warnings, with and without --features security - cargo +nightly fmt --check - cargo check --all-targets --target armv7-unknown-linux-gnueabihf Assisted-by: Claude Fable 5.2
johannesschrimpf
marked this pull request as draft
November 19, 2026 16:13
DiscoveryDB::update_subscription() and update_publication() entered a remote reader or writer even when its participant was not in the DB, and Discovery then sent ReaderUpdated or WriterUpdated, so local endpoints matched it. The SEDP readers deliver a sample only once its WriterProxy has received it, so the participant was known when the sample arrived. But Discovery can take the sample after it has processed the participant's SPDP dispose, or its lease expiry: the two arrive on separate poll events, and a rediscovery of another participant also takes every pending SEDP sample. The endpoint is then matched after its participant was removed, and nothing removes it again. remove_participant() and participant_cleanup() go by the participant table, which no longer has the participant, and the endpoint's own SEDP dispose is not delivered either, since the WriterProxy of the remote SEDP writer is gone. For a reliable local writer, such a reader is a ReaderProxy that never acknowledges. wait_for_acknowledgments() never succeeds. Since commit 792792f ("Dynamic heartbeat rate to avoid data loss at interop test."), handle_heartbeat_tick() also keeps the writer on HEARTBEAT_PERIOD_FAST: a non-final HEARTBEAT to multicast every 100 ms, which every live reader answers with an ACKNACK, for the life of the process. We saw this at ROS startup on about one start in ten, on the rt/ros_discovery_info writer of a RustDDS node. There, rclcpp's component container runs a first Fast DDS participant for under a second while the RustDDS nodes come up. RTPS 2.5 ties discovered endpoints to discovered participants: SEDP is configured from SPDP (8.5.4.1, 8.5.5.1), and when a participant goes away, the local endpoints that communicated with its endpoints must be reconfigured (8.5.5.2). Fast DDS keeps remote endpoint data inside the remote ParticipantProxyData and drops SEDP data of a participant it does not know (EDPBaseSUBListener::add_reader_from_change, "From UNKNOWN RTPSParticipant, removing"). Keep an endpoint of a participant that is not known in the attic instead, with the endpoints of participants that timed out. The previous commit made the way back from there work: the endpoint is matched if the participant is discovered (again), and dropped with the participant or when SEDP disposes it. update_subscription() and update_publication() now return None for such an endpoint, and Discovery sends no notification for it. A plain drop would lose the endpoint for good if its participant comes back after a lease timeout, since the sample has already been taken from the DataReader. Tests: - sedp_reader_of_removed_participant_is_not_matched: a participant is discovered and disposed, and then Discovery processes its reliable reader's SEDP sample. Without the fix, the local writer matches the reader, wait_for_acknowledgments() fails, and the writer sends about 10 HEARTBEATs/s after its live reader has acknowledged everything. With it, the reader is not matched, all samples are acknowledged, and no HEARTBEAT is sent. When the participant is discovered again, its reader is matched. - discdb_endpoints_wait_for_their_participant: the DiscoveryDB side. - rediscovery_keeps_pending_sedp_samples_of_other_participants and discdb_external_endpoint_snapshots matched endpoints of participants the DB did not know. They now enter the participant first. Validation on macOS: - cargo test --tests -- --test-threads=1, with and without --features security - cargo clippy --all-targets -- -D warnings, with and without --features security - cargo +nightly fmt --check - cargo check --all-targets --target armv7-unknown-linux-gnueabihf Assisted-by: Claude Fable 5.2
A remote reader or writer that announces no unicast, or no multicast, locators is reached at the default locators of its participant. update_subscription() and update_publication() fill those in, and the DiscoveryDB stores the result. When the participant timed out, that record went to the attic as it was. If the participant returned with other default locators, e.g. because its address had changed, its endpoints were entered again with the old ones, as if they had announced them: update_subscription() fills in default locators only where the list is empty. A local writer matched again with such a reader would send to the old address, get no acknowledgment, and stay on the fast HEARTBEAT period. The DiscoveryDB now remembers which locator lists of a remote endpoint are the default locators of its participant, and leaves those out when the endpoint goes to the attic. The attic then has the locators that the endpoint announced, as it already has for an endpoint whose participant was not known when its SEDP data was processed. When the participant returns, update_subscription() and update_publication() fill in the default locators it announces then. This is about a participant that returns after a timeout. A participant that announces other default locators while it is known still does not change the locators of the endpoints it has, as before. Tests: - discdb_endpoints_of_a_returning_participant_use_its_new_default_locators: a reader without locators and a writer with a unicast locator of its own. Their participant times out and returns with other default locators. Without the fix, both have the old default locators. With it, they have the new ones, and the writer keeps its own unicast locator. Validation on macOS: - cargo test --tests -- --test-threads=1, with and without --features security - cargo clippy --all-targets -- -D warnings, with and without --features security - cargo +nightly fmt --check Assisted-by: Claude Fable 5.2 Assisted-by: Astra GPT-6.2
When a participant that had timed out was discovered again,
process_discovered_participant_data() entered its readers and writers
from the attic and announced them to the local endpoints directly. With
DDS Security, that used what Discovery had accepted before the
participant was lost. But the participant is validated anew when it
returns, and the result can differ: remove_participant() drops its
authentication status, and its permissions handle is unregistered.
In particular, if the governance document allows unauthenticated
participants, a participant that was authenticated can return as one
that cannot be authenticated, e.g. with SPDP data that has no identity
token. Its readers and writers on a topic with read or write access
control were then matched again with the local endpoints, although
Discovery does not accept such endpoints from an unauthenticated
participant when SEDP announces them (DDS Security 1.1, 8.8.7.1). Before
the attic restore announced the endpoints, they were only entered into
the DiscoveryDB, so only local endpoints created later could match them.
Discovery now processes the endpoints from the attic as the SEDP samples
that they were, with the code that sedp_receive_subscription() and
sedp_receive_publication() use, which moves into
process_subscription_sample() and process_publication_sample(). So they
pass check_nonsecure_subscription_read() or
check_nonsecure_publication_read() for what their participant is now,
and one that is rejected is neither entered nor matched.
DiscoveryDB::take_endpoints_from_attic() replaces
restore_endpoints_from_attic(): it hands the endpoints out without
entering them.
A participant that is authenticated again is entered into the
DiscoveryDB by SecureDiscovery when the authentication starts.
SecureDiscovery no longer enters its endpoints from the attic at that
point, unchecked. They stay in the attic until the participant is
authenticated and its DCPSParticipantsSecure data arrives, and are then
processed as above.
Limitations:
- Endpoints from the attic are checked as DCPSSubscription and
DCPSPublication data. An endpoint on a topic with discovery protection
came from DCPSSubscriptionsSecure or DCPSPublicationsSecure and is
rejected by that check, so it is not matched again when its
participant returns. Before, such endpoints were entered unchecked and
not announced.
- Only the unauthenticated case has a test. The case of a participant
that is authenticated again was not run.
Tests:
- endpoints_of_a_participant_that_returns_unauthenticated_are_checked_again:
a participant with the built-in security plugins, and a governance
document that allows unauthenticated participants and has a topic with
read and write access control. A remote participant has a reader and a
writer on that topic in the attic and returns with SPDP data without
an identity token. Without the fix, the local writer and the local
reader are matched with them. With it, they are not matched and not
entered.
- The check rejects them at the lookup of the security attributes of
the topic, for which it needs the permissions handle of the remote
participant: an unauthenticated participant has none. So the test
shows that endpoints from the attic get the check, not which of its
rules rejects them.
- examples/security_configuration_files/allow_unauthenticated/ has the
governance and permissions documents of that test, signed with the
example permissions CA. Both scripts that sign the example documents
sign these too; generate-test-certificates-and-signatures.sh was not
run.
- The DiscoveryDB tests enter the endpoints from the attic themselves
now, as Discovery does.
Validation on macOS:
- cargo test --tests -- --test-threads=1, with and without
--features security
- cargo clippy --all-targets -- -D warnings, with and without
--features security
- cargo +nightly fmt --check
- cargo check --all-targets --target armv7-unknown-linux-gnueabihf
Assisted-by: Claude Fable 5.2
Assisted-by: Astra GPT-6.2
With DDS Security, the checks for a dispose on the SEDP topics look the
reader or writer up to find its topic, and reject the dispose if it is
not known. They looked among the entered endpoints only. A dispose of an
endpoint that is in the attic was therefore rejected, the endpoint
stayed in the attic, and it was entered and matched when its participant
was processed, although it no longer exists. A local reliable writer
matched with such a reader gets no acknowledgment from it.
The previous commit made that more likely: the endpoints of a
participant that is authenticated again now stay in the attic until its
DCPSParticipantsSecure data is processed, and the participant can delete
an endpoint in that time. Without security, a dispose is not checked,
and remove_topic_reader() and remove_topic_writer() drop an endpoint
from the attic too.
The four checks now use get_disposable_topic_reader() and
get_disposable_topic_writer(), which also find an endpoint in the attic.
Limitation: the checks still reject a dispose while its participant is
not known or is being authenticated, as before. An endpoint that is
disposed then, e.g. by an SEDP sample that was pending when the
participant timed out, returns from the attic with its participant.
Tests:
- endpoint_disposed_while_its_participant_is_authenticated_again_does_not_return:
two participants with the built-in security plugins authenticate each
other. The test puts the DiscoveryDB of one into the state after the
other has timed out and has been authenticated again, with its
endpoints in the attic. The other deletes its reader, and then
announces itself. Without the fix, the deleted reader is entered
again. With it, only the writer is. Then the same with a deleted
writer.
- A RustDDS participant disposes an endpoint on DCPSSubscription and
on DCPSSubscriptionsSecure (or the publication topics). One of the
two checks finding the endpoint is enough for the test to pass, so
it does not tell the two apart.
- This is also the first test in which a participant that is
authenticated again gets endpoints from the attic checked and
entered. The timeout and the second authentication are not real:
the test writes that state into the DiscoveryDB.
Validation on macOS:
- cargo test --tests -- --test-threads=1, with and without
--features security
- cargo clippy --all-targets -- -D warnings, with and without
--features security
- cargo +nightly fmt --check
- cargo check --all-targets --target armv7-unknown-linux-gnueabihf
Assisted-by: Claude Fable 5.2
Assisted-by: Astra GPT-6.2
johannesschrimpf
force-pushed
the
fix/orphan-reader-proxy
branch
from
November 19, 2026 17:38
b54791d to
cd07b9f
Compare
johannesschrimpf
marked this pull request as ready for review
November 19, 2026 17:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five commits. The first makes the existing "attic" announce the endpoints it restores, so that a participant that returns after a timeout is matched again. The second keeps an endpoint whose participant is not known in the attic too, instead of matching it. The second is the fix for the HEARTBEAT storm below, and it builds on the first.
The third to fifth commits fix problems that a review found in what the first commit matches again: an endpoint kept the default locators that its participant had before the timeout; with DDS Security, endpoints were matched without being checked again; and the dispose of an endpoint in the attic was rejected.
Problem
1. A reader matched after its participant was removed is never removed
A reliable writer can end up with a ReaderProxy for a remote reader whose participant is already gone. Nothing ever removes that proxy. The proxy never acknowledges, so:
HEARTBEAT_PERIOD_FASTfor the life of the process (the adaptive heartbeat rate from commit 792792f), sending a non-final HEARTBEAT to multicast every 100 ms;wait_for_acknowledgments()never succeeds, and the reliable send window is held back.We saw this at ROS 2 startup on an embedded robot (i.MX6, plain localhost):
rt/ros_discovery_infowriter of one RustDDS node stayed at 9.85 HEARTBEATs/s, multicast on two interfaces, while all 15 live readers acknowledged everything.ros_discovery_inforeader is the likely orphan.2. The endpoints of a participant that timed out are not matched again when it returns
DiscoveryDBmoves its readers and writers to the attic (commit 203fd01), because SEDP does not deliver them a second time.update_participant()moved them back into the DB, but noReaderUpdatedorWriterUpdatedfollowed.Cause
1. The orphan reader
DiscoveryDB::update_subscription()(src/discovery/discovery_db.rs) entered a remote reader even when its participant was not in the DB ("This is normal, since we might not know about the participant yet").Discovery::sedp_receive_subscription()then sentReaderUpdated, and local writers added a proxy. Writers were handled the same way byupdate_publication().remove_participant()/participant_cleanup()go by the participant table, which no longer has the participant, so noParticipantLostfollows;Reader::participant_lost()removed the WriterProxy of the remote SEDP writer.A participant that just goes silent, without a dispose, is cleaned up correctly at lease expiry. Only the "removed first, then the pending sample" order creates a permanent proxy.
2. The silent attic
update_participant(). Nothing told the event loop.Reproduction
sedp_reader_of_removed_participant_is_not_matched(second commit). It uses the existingadd_pending_sedp_samplehelper:wait_for_acknowledgmentsThe test also checks that the reader is matched when its participant is discovered again.
endpoints_of_a_timed_out_participant_are_matched_again_when_it_returns(first commit):What the specification says (DDSI-RTPS 2.5)
Comparison with Fast DDS (v2.14.7, v3.6.2)
EDPBaseSUBListener::add_reader_from_change()drops endpoint data whose participant is not in the PDP database ("From UNKNOWN RTPSParticipant, removing").PDP::addReaderProxyData()returns nullptr, and remote endpoints are stored inside the remoteParticipantProxyData.remove_remote_participant), unpairs every endpoint of that participant.StatefulReader::acceptMsgFrom()refuses DATA from unmatched writers on SEDP.Fix
update_subscription()/update_publication()returnOption. Discovery sends no notification forNone.take_endpoints_from_attic()hands out the endpoints of a participant.process_discovered_participant_data()calls it under the same DiscoveryDB lock asupdate_participant(), sendsParticipantUpdated, and then processes each endpoint as the SEDP sample that it was (process_subscription_sample(),process_publication_sample()). So it is checked (with security), entered and announced as new SEDP data is.update_participant()no longer moves endpoints out of the attic on its own.Behaviour changes
SecureDiscoveryremembers the endpoints it has sent keys to and never forgets them (user_data_endpoints_with_keys_already_sent_to), sostart_key_exchange_with_remote_endpoint()returns early. Before, such endpoints were not matched again at all.Tests
endpoints_of_a_timed_out_participant_are_matched_again_when_it_returns, and three DiscoveryDB tests of the attic:discdb_endpoints_of_a_timed_out_participant_return_with_it,discdb_endpoint_disposed_while_its_participant_is_timed_out_does_not_return,discdb_dispose_of_a_timed_out_participant_drops_its_endpoints.sedp_reader_of_removed_participant_is_not_matched,discdb_endpoints_wait_for_their_participant. Two existing tests (rediscovery_keeps_pending_sedp_samples_of_other_participants,discdb_external_endpoint_snapshots) matched endpoints of participants the DB did not know. They now enter the participant first.discdb_endpoints_of_a_returning_participant_use_its_new_default_locators.endpoints_of_a_participant_that_returns_unauthenticated_are_checked_againandendpoint_disposed_while_its_participant_is_authenticated_again_does_not_return.examples/security_configuration_files/allow_unauthenticated/, signed with the example permissions CA.Not in this PR, possible follow-ups:
handle_heartbeat_tick()also counts best-effort reader proxies, which never acknowledge.Applicability
mio_channel::sync_channel. That is the only difference. The other three commits are not ported there.Validation (macOS)
cargo test -- --test-threads=1, with and without--features security: all pass at each commit.cargo clippy --all-targets -- -D warnings, with and without--features security: clean at each commit.cargo +nightly fmt --check: clean at each commit.cargo check --all-targets --target armv7-unknown-linux-gnueabihf: clean at each commit.