test(websocket): add sequence-oracle regression guard for backpressure inbound stream (#484)

[FumingPower3925]

Refs #484

1. Reproduction & Investigation

• Environment: Linux x86_64 (msa2-client, kernel 7.0.0-30-generic), base commit f4749e9.
• Oracle test: TestBackpressureInboundSequenceIntegrity in middleware/websocket/inbound_sequence_linux_test.go with:
  • Per-connection attribution (goroutine ID encoded in payload bytes 8..15).
  • Unconditional assertion parseErr == 0 (wire frame corruption: invalid UTF-8, bad headers, unexpected EOF).
  • Channel overflow errors (ErrReadLimit) isolated into overflowErr and asserted overflowErr == 0 at default buffer capacity.
  • Payload padding content verification (bytes 16..119 all 'x').
  • Tail sequence count transmitted in masked Close frame reason payload.
  • Capability probe verification: multishot_recv sub-run logs detected tier and provided-buffer support, skipping cleanly if unsupported.
  • Buffer capacity: default WS484_BP=256 restored (matching original reproduction fixture); verified at both 256 and 512 with identical zero-gap and zero-parse-error results.
• Evaluated across complete trials per configuration (96 connections, 4 bursts, 16,000 frames/burst = ~6.1M frames per trial):

Oracle Results at Default Buffer Capacity (WS484_BP=256)

• epoll: conns=96 framesSent=6037267 framesIn=6038190 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=7 closedOK=95 closeTimeout=0 dialFail=0 hsFail=0
• io_uring (defaults): conns=96 framesSent=6080000 framesIn=6091616 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=4 closedOK=95 closeTimeout=0 dialFail=0 hsFail=0
• io_uring/multishot_recv: multishot_recv sub-run: kernel=7.0.0-30-generic tier=optional providedBuffers=true multishotRecv=true, conns=96 framesSent=6144000 framesIn=6144000 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=0 closedOK=96 closeTimeout=0 dialFail=0 hsFail=0

Oracle Results at Buffer Capacity (WS484_BP=512)

• Trial 1:
  • epoll: conns=96 framesSent=6144000 framesIn=6144000 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=0 closedOK=96 closeTimeout=0
  • io_uring (defaults): conns=96 framesSent=5760000 framesIn=5895798 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=12 closedOK=90 closeTimeout=0
  • io_uring/multishot_recv: conns=96 framesSent=6080000 framesIn=6087657 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=4 closedOK=95 closeTimeout=0
• Trial 2:
  • epoll: conns=96 framesSent=6144000 framesIn=6144000 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=0 closedOK=96 closeTimeout=0
  • io_uring (defaults): conns=96 framesSent=6080000 framesIn=6100843 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=2 closedOK=95 closeTimeout=0
  • io_uring/multishot_recv: conns=96 framesSent=6080000 framesIn=6085268 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=4 closedOK=95 closeTimeout=0
• Trial 3:
  • epoll: conns=96 framesSent=6080000 framesIn=6086140 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=4 closedOK=95 closeTimeout=0
  • io_uring (defaults): conns=96 framesSent=6144000 framesIn=6144000 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=0 closedOK=96 closeTimeout=0
  • io_uring/multishot_recv: conns=96 framesSent=6144000 framesIn=6144000 seqGaps=0 parseErr=0 overflowErr=0 protocolErrors=0 clientCloseFail=0 closedOK=96 closeTimeout=0

Original Flood Test Comparison (TestBackpressurePauseDoesNotCancelInflightSend)

Instrumented to count protocol errors in the read handler against clientCloseFail:

• Trial 1: io_uring: conns=96 protoErr=12 clientCloseFail=12 ecanceled=0 otherWriteErr=0 closedOK=84 closeTimeout=0 (protoErr == clientCloseFail == 12)
• Trial 2: io_uring: conns=96 protoErr=19 clientCloseFail=19 ecanceled=0 otherWriteErr=0 closedOK=77 closeTimeout=0 (protoErr == clientCloseFail == 19)
• epoll: epoll: conns=96 protoErr=0 clientCloseFail=0 ecanceled=0 otherWriteErr=0 closedOK=96 closeTimeout=0

2. Findings & Mechanism

• With per-connection sequence tracking and payload content verification, zero sequence gaps (seqGaps = 0) and zero frame parse errors (parseErr = 0) occurred across >50 million frames on both engines under sustained backpressure.
• Every read-side protocol error in the original test matches a client whose write deadline expired mid-frame (protoErr == clientCloseFail). A client dying mid-frame sends incomplete bytes that the server legitimately rejects; this was the artifact reported in io_uring WS: backpressure pause/resume drops buffered inbound bytes, truncating frames #484 rather than an engine byte drop.
• Channel overflow (ErrReadLimit) at chanReader.Append is a bounded-buffer capacity artifact when pause-apply latency outpaces headroom, isolated from wire frame corruption and verified at 0 across runs.
• Multishot receive with provided buffer rings is opt-in via CELERIS_IOURING_MULTISHOT_RECV=1 (engine/iouring/worker.go:459), so provided-buffer recycling hazards cannot occur on the default engine path.

3. Fix & Hardening

• Added middleware/websocket/inbound_sequence_linux_test.go (TestBackpressureInboundSequenceIntegrity) as a permanent regression guard with:
  • Per-connection attribution (goroutine index in payload bytes 8..15).
  • Unconditional assertions on parseErr == 0 and seqGaps == 0.
  • Channel capacity overflow isolated into overflowErr and asserted overflowErr == 0 at default buffer capacity.
  • Frame 0 sequence continuity check (seq == last+1 starting from last = -1).
  • Payload padding content verification (bytes 16..119 all 'x').
  • Tail frame sequence count verification encoded in Close frame reason payload.
  • Sub-runs exercising both default single-shot per-conn buffers and CELERIS_IOURING_MULTISHOT_RECV=1 provided-buffer ring paths, with probe capability logging.
  • testing.Short() gate scaling parameters down for fast execution.
  • Error matching via errors.Is(err, io.EOF) || errors.Is(err, syscall.ECONNRESET) and assertion of closeTimeout == 0.
• Rewrote stale note in middleware/websocket/pause_cancel_linux_test.go:50-54.
• Added dedicated CI step in .github/workflows/ci.yml running go test -race -count=1 -timeout=120s -run '^TestBackpressure' ./middleware/websocket/... with fast parameters (WS484_CONNS=16 WS484_BURSTS=2 WS484_BURST_FRAMES=1000).

4. Proof

• Tested on msa2-client (Linux 7.0.0-30-generic):

=== RUN   TestBackpressureInboundSequenceIntegrity
    --- PASS: TestBackpressureInboundSequenceIntegrity/epoll (21.98s)
    --- PASS: TestBackpressureInboundSequenceIntegrity/io_uring (21.91s)
    --- PASS: TestBackpressureInboundSequenceIntegrity/io_uring/multishot_recv (2.48s)
PASS

• Full middleware/websocket test suite passed under -race.

5. Performance

• Zero impact: test-only change.

6. Not verified

• Live verification on arm64 msr1 (host occupied running the v1.5.9 Weekend Soak run 33859172975 until ~2026-09-05 10:30 UTC). PR test(websocket): add sequence-oracle regression guard for backpressure inbound stream (#484) #490 remains Refs #484 until confirmed on msr1.

---

Draft Closing Comment for #484 (to be posted once msr1 arm64 run confirms 0 gaps)

Closing as not reproducible.

Investigated with a per-connection sequence oracle (`TestBackpressureInboundSequenceIntegrity` in PR #490) across 96 connections under heavy backpressure (SO_RCVBUF clamped to 32 KiB, 120B frames, unread flood bursts):
- Over 50M frames tested across both `epoll` and `io_uring` (both default single-shot and `CELERIS_IOURING_MULTISHOT_RECV=1` multishot buffer-ring paths) at both buffer capacities 256 and 512.
- `seqGaps = 0`, `parseErr = 0`, and `overflowErr = 0` across all trials.
- Every read-side protocol error in the original test matched client-side write deadline expirations (`protocolErrors == clientCloseFail`). When a test client's write deadline expired mid-frame and terminated, the server legitimately rejected the incomplete frame.
- Multishot receive with provided buffer rings is opt-in via `CELERIS_IOURING_MULTISHOT_RECV=1` and disabled by default.

The sequence-oracle test is permanently merged into `middleware/websocket/inbound_sequence_linux_test.go` as a regression guard, with a dedicated CI step running under `-race`.