Repository navigation
SysbotBase: bound the shutdown and heartbeat sends at 1 second - #1877
Merged
Merged
Conversation
A peer that stops reading can stall the application's shutdown: the destructor sends detachController before closing the socket, and write_data() used an unlimited send timeout. If the peer has filled the send path (or stalled it in any other way), destruction blocks for the whole TCP retransmission timeout - or forever, if the peer never errors the connection. write_data() now takes a send deadline (default: unlimited, so the command and setup paths are unchanged) and passes it to the socket's send(). Two paths use a 1-second deadline: - The destructor's detachController send. The detach is best-effort; the socket close that follows is what matters. - The heartbeat's ping/getVersion sends. These hold m_send_lock while the destructor's send waits for it, so a heartbeat send stalled on a non-reading peer must not hold the shutdown path; 1 s matches the heartbeat's own 1 s cadence. A timed-out ping is simply dropped (m_active_pings is capped at 60 entries). The new regression test drives a real TcpSysbotBase_Connection against a ServerSocket_Qt that accepts and then blocks its event thread on the first received byte (a peer that stops reading). It verifies that a write_data() with the 1-second deadline returns within the shared 60 s budget, and that the destructor completes within the budget even while the peer is still stalling - the pre-fix behavior was an unbounded hang in both cases.
Collaborator
|
Putting on hold until a new atmosphere is released for FW 23.5.0. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the audit finding "Non-reading server stalls shutdown": the destructor sends
detachControllerbefore closing the socket, andwrite_data()used an unlimited send timeout. A peer that stops reading can stall the application's shutdown for the whole TCP retransmission timeout (or forever, if the peer never errors the connection).Changes
write_data()now takes a send deadline (default: unlimited — the command and setup paths are unchanged) and passes it through to the socket'ssend(), which was given a timeout parameter by the earlier send-deadline work.Two paths pass a 1-second deadline:
detachControllersendping/getVersionsendsm_send_lockwhile the destructor's send waits for it — a heartbeat send stalled on a non-reading peer must not hold the shutdown path; 1 s matches the heartbeat's own 1 s cadence, and a timed-out ping is simply dropped (m_active_pingsis capped at 60)Testing
New regression test
NintendoSwitch::SysbotBase - shutdown send is bounded against a stalled peer:ServerSocket_Qtthat accepts the connection and then blocks its event thread on the first received byte (a peer that stops reading),TcpSysbotBase_Connectionagainst it,write_data(payload, 1s)returns within the shared 60 s budget,The test skips (doesn't fail) on starved machines via the same
timing_unreliable()probe the socket tests use; the 60 s budget only has to prove boundedness (pre-fix = infinite hang), matching the socket deadline tests' convention. 16 MiB payload reported viam_memory.Full suite: 20,448 passed / 0 failed (5 skips = the usual audio-file environment skips).