Skip to content

feat(message_bus): read replica sockets ahead, bypassing large frames - #4224

Open
hubcio wants to merge 1 commit into
masterfrom
feat/msg-bus-bufreadwrite
Open

hubcio wants to merge 1 commit into
masterfrom
feat/msg-bus-bufreadwrite

Conversation

@hubcio

@hubcio hubcio commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

The link shard that owns a peer's replica socket burns a full
core under sustained replicated writes. Nothing batches its reads:
read_message issues one io_uring read for the 256-byte header and
a second for the body. Bytes already sitting in the kernel receive
queue wait for the next call.

A read-ahead buffer sized by message_bus.replica_read_buffer_size
lets one socket read serve a whole burst. It adds no latency, because
the reader never waits for the buffer to fill. One fill delivers
at most one buffer, so a body above that size crosses the buffer in
buffer-sized pieces and pays one extra pass over the payload and one
read per piece. A read that large goes straight into the frame's own
buffer instead.

Zero keeps the unbuffered path, so the A/B baseline is a config change
and not a separate build. The plaintext writer already coalesces into
one writev, so it is left alone.

replica_socket_reads_total and replica_inbound_frames_total carry the
evidence on the real workload: their ratio is the batching factor,
and only link shards bump them. A read counts once it completes,
so a link torn down mid-read cannot inflate the ratio.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Sep 18, 2026
@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.32432% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.83%. Comparing base (c115986) to head (f980b26).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
core/message_bus/src/framing.rs 97.77% 0 Missing and 1 partial ⚠️
core/message_bus/src/transports/tcp.rs 99.31% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master    #4224       +/-   ##
=============================================
- Coverage     87.50%   73.83%   -13.67%     
- Complexity     1575     1576        +1     
=============================================
  Files          1280     1278        -2     
  Lines        223241   203774    -19467     
  Branches     186604   167137    -19467     
=============================================
- Hits         195337   150448    -44889     
- Misses        23193    48668    +25475     
+ Partials       4711     4658       -53     
Components Coverage Δ
Rust Core 71.77% <99.32%> (-16.80%) ⬇️
Java SDK 68.70% <ø> (+0.01%) ⬆️
C# SDK 77.57% <ø> (+0.11%) ⬆️
Python SDK 90.97% <ø> (ø)
PHP SDK 85.67% <ø> (ø)
Node SDK 96.43% <ø> (-0.02%) ⬇️
Go SDK 70.11% <ø> (-0.06%) ⬇️
Files with missing lines Coverage Δ
core/configs/src/server_config/defaults.rs 100.00% <100.00%> (ø)
core/configs/src/server_config/displays.rs 100.00% <ø> (ø)
core/configs/src/server_config/message_bus.rs 98.85% <100.00%> (+0.47%) ⬆️
core/message_bus/src/config.rs 77.64% <100.00%> (+0.53%) ⬆️
core/message_bus/src/installer/replica.rs 88.56% <100.00%> (+0.20%) ⬆️
core/message_bus/src/lib.rs 97.52% <100.00%> (-0.06%) ⬇️
core/shard/src/lib.rs 85.51% <100.00%> (-0.03%) ⬇️
core/shard/src/metrics.rs 91.62% <100.00%> (+0.20%) ⬆️
core/message_bus/src/framing.rs 92.96% <97.77%> (+2.60%) ⬆️
core/message_bus/src/transports/tcp.rs 97.91% <99.31%> (-0.74%) ⬇️

... and 296 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hubcio
hubcio force-pushed the feat/msg-bus-bufreadwrite branch from 777cb9d to 1a555e5 Compare September 18, 2026 12:07
The link shard that owns a peer's replica socket burns a full
core under sustained replicated writes. Nothing batches its reads:
read_message issues one io_uring read for the 256-byte header and
a second for the body. Bytes already sitting in the kernel receive
queue wait for the next call.

A read-ahead buffer sized by message_bus.replica_read_buffer_size
lets one socket read serve a whole burst. It adds no latency, because
the reader never waits for the buffer to fill. One fill delivers
at most one buffer, so a body above that size crosses the buffer in
buffer-sized pieces and pays one extra pass over the payload and one
read per piece. A read that large goes straight into the frame's own
buffer instead.

Zero keeps the unbuffered path, so the A/B baseline is a config change
and not a separate build. The plaintext writer already coalesces into
one writev, so it is left alone.

replica_socket_reads_total and replica_inbound_frames_total carry the
evidence on the real workload: their ratio is the batching factor,
and only link shards bump them. A read counts once it completes,
so a link torn down mid-read cannot inflate the ratio.
@hubcio
hubcio force-pushed the feat/msg-bus-bufreadwrite branch from 1a555e5 to f980b26 Compare September 18, 2026 12:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant