Skip to content

fix(services): terminate the JSON-RPC read loop on a non-recoverable read error - #103

Closed
woodsonl wants to merge 1 commit into
NVIDIA:developfrom
woodsonl:fix/jsonrpc-read-loop-terminal-errors
Closed

woodsonl wants to merge 1 commit into
NVIDIA:developfrom
woodsonl:fix/jsonrpc-read-loop-terminal-errors

Conversation

@woodsonl

@woodsonl woodsonl commented Sep 22, 2026 •

Copy link
Copy Markdown

Description

Every service read loop treated all non-EOF read errors the same way: log the
error and continue. That is correct for a bad frame, which the scanner has
already advanced past. It is wrong for a terminal transport error such as an
over-long frame that bufio.Scanner cannot resync past. There the next Read
returns the same error, so the loop spins on it forever.

This classifies read errors by type. A DecodeError (bad JSON, wrong version)
stays recoverable and keeps the loop alive. Any other non-EOF error is
terminal: the broker's read loop returns errTerminalRead so Serve tears
down cleanly, and the producer goroutine stops feeding the channel once it hits
a terminal error rather than blocking forever.

Release intent

Changelog title

Services stop cleanly on a terminal stream read error

Changelog body

A service that hit a non-recoverable read error on its JSON-RPC stream, such
as an over-long frame the scanner cannot resync past, used to log the error
and retry forever. Read errors are now split into recoverable frame errors,
which are skipped, and terminal transport errors, which end the read pump.

Bumps

  • services: patch
  • nvpair-cluster-manager: patch
  • nvpair-engine-manager: none
  • nvpair-errors: patch
  • nvpair-job-scheduler: patch
  • nvpair-manual-nodes: patch
  • nvpair-node-info: none
  • nvpair-node-scanner: patch
  • nvpair-node-settings: patch
  • nvpair-proxy: patch
  • nvpair-tui: patch
  • nvpair-ui-broker: patch
  • nvpair-workload-manager: patch

Scope

Included: the readLoop error classification in every service that reads
JSON-RPC from a stream, the DecodeError codec alias the classification needs,
and the broker's recoverableDecode/errTerminalRead helpers with a test.

Excluded: nothing else. This is one behavior change applied uniformly.

Validation

  • go build ./... and go test ./... in every touched module
    (nvpair-cluster-manager, nvpair-errors, nvpair-job-scheduler,
    nvpair-manual-nodes, nvpair-node-scanner, nvpair-node-settings,
    nvpair-proxy, nvpair-tui, nvpair-ui-broker, nvpair-workload-manager)
  • services/nvpair-ui-broker/terminal_read_test.go covers the three cases:
    a non-EOF transport error ends the loop with errTerminalRead, EOF exits
    cleanly, and a malformed frame keeps the loop alive.

Risk

Low. The recoverable path is unchanged; only the terminal path now stops
instead of looping. An engine or peer that sends an over-long frame will now
end that connection rather than pin a CPU, which is the intended behavior.

Checklist

  • I have read the Contributing Guidelines.
  • Every commit is signed off (git commit -s), certifying the Developer Certificate of Origin.
  • New or existing tests cover the change.
  • Relevant documentation is updated.
  • I checked the diff, changed filenames, and commit messages for credentials, private data, internal URLs, internal issue identifiers, and generated artifacts.
  • I recorded the validation commands and results above.
  • I declared version bumps in the release-intent block above. services/versions.json is written by automation — do not edit it by hand.

…loops

Every service's read loop treated all non-EOF read errors the same way:
log and `continue`. That is right for a bad frame (the scanner advances
past it) but wrong for a terminal transport error such as an over-long
frame that bufio.Scanner cannot resync past — the loop then spins on the
same error forever instead of ending the pump.

Classify by error type instead. A DecodeError is recoverable and keeps
the loop alive; any other non-EOF error is terminal. The broker's read
loop returns errTerminalRead so Serve tears down cleanly, and the
producer goroutine stops feeding the channel once it hits a terminal
error rather than blocking forever.

Signed-off-by: woodsonl <65194841+woodsonl@users.noreply.github.com>
@woodsonl woodsonl closed this Sep 24, 2026
@woodsonl
woodsonl deleted the fix/jsonrpc-read-loop-terminal-errors branch September 24, 2026 05:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant