Conversation
…liveries The broker handled every inbound message inline on the read loop. Each handleMessage can run a synchronous worker relay bounded by rpcWorkerCallTimeout (5s), so one slow proxy/cluster/settings call blocked every other client request behind it. Dispatch through a small fixed pool instead. Cross-request ordering is preserved where it matters by the per-handler mutexes (workloadEmitMu for workload apply→fan→emit, per-state mutexes for subscription bookkeeping), and JSON-RPC does not promise cross-request response ordering — each response carries its own id, and the codec's write mutex keeps concurrent responses from interleaving. The relay also delivered to each subscriber with a blocking send, so a subscriber that stopped reading could stall the pump for everyone. Deliver now coalesces under the subscriber's own lock and drops the oldest pending message when a subscriber falls behind, so one stuck client cannot back-pressure the shared stream. The pump re-checks done before every Send, so a trigger that races Unsubscribe cannot deliver to a consumer that's gone; Unsubscribe itself does not wait for the pump to exit. The read loop also classifies read errors: a DecodeError is recoverable and keeps the loop alive, any other non-EOF error is terminal and ends the pump (errTerminalRead) instead of spinning. Signed-off-by: woodsonl <65194841+woodsonl@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The broker handled every inbound message inline on the read loop. Each
handleMessagecan run a synchronous worker relay bounded byrpcWorkerCallTimeout(5s), so one slow proxy, cluster, or settings callblocked every other client request behind it.
This dispatches through a small fixed pool. Cross-request ordering is preserved
where it matters by the per-handler mutexes (
workloadEmitMufor workloadapply → fan → emit, per-state mutexes for subscription bookkeeping). JSON-RPC
does not promise cross-request response ordering: each response carries its own
id, and the codec's write mutex keeps concurrent responses from interleaving.
The relay had a matching problem. It delivered to each subscriber with a
blocking send, so a subscriber that stopped reading could stall the pump for
everyone.
Delivernow coalesces under the subscriber's own lock and drops theoldest pending message when a subscriber falls behind, so one stuck client
cannot hold up the shared stream.
The read loop also classifies its read errors, matching the change in the
service read loops: a
DecodeErroris recoverable and keeps the loop alive,any other non-EOF error is terminal.
Release intent
Changelog title
Broker no longer lets one slow worker stall the control plane
Changelog body
The broker now handles client requests on a small worker pool instead of
inline on the read loop, so a slow worker call no longer blocks unrelated
requests behind it. Relay deliveries to a subscriber that has stopped reading
coalesce instead of blocking the stream for every other subscriber.
Bumps
Scope
Included: the broker dispatch pool, the relay's coalescing delivery, and the
broker read loop's error classification with its test.
Excluded: the same read-error classification in the individual service read
loops, which is proposed separately.
Validation
go build ./...andgo test ./...inservices/nvpair-ui-brokerservices/nvpair-ui-broker/relay/relay_test.gocovers the coalescingdelivery path, and
terminal_read_test.gocovers the read-loop cases.Risk
Low. Handler ordering is unchanged for the channels that depend on it because
the serialization lives in per-handler mutexes, not the read loop. A subscriber
that falls behind now drops its oldest pending message rather than stalling,
which is the behavior the relay's design already implied.
Checklist
git commit -s), certifying the Developer Certificate of Origin.services/versions.jsonis written by automation — do not edit it by hand.