What happens
A connection whose handler is parked on something other than a read - a send, ioxd_delay, a socket client call - while its peer keeps sending has its multishot recv filling the per-connection receive queue (struct spsc, RX_QUEUE = 64 delivered buffers). When the queue is full, ioxd__conn_on_recv returns the buffer, ends the input with -ENOBUFS and cancels the recv (lib/io/conn.c, the ioxd__spsc_full branch). The handler's next read fails and the connection is closed: the peer sees a reset, or a reply cut short.
Where it was seen
The HttpArena entry's streamed echo (POST /echo: read 8 KB into the reply slab, send it, read the next 8 KB) on the arena's CI runner. A 100 KB body over TLS arrived there as small completions - well under 2 KB each - so more than 64 buffers queued while the handler waited on its first send, and the client got 0 bytes back (HttpArena PR #1473, run 34418961856). On this machine the same body arrives in 2 KB completions, 50 buffers, and never reaches the mark, so make check and the arena validator both pass.
Reproduced locally with the tiny build's queue (-DRX_QUEUE=4, as make check-tiny builds it): a 100 KB echo over TLS or plain dies after 16 KB, curl exit 18/56.
What it means
The queue is meant to bound how much one connection can hold of the shared buffer group (ioxd__conn_on_recv notes: "rather than let one peer hoard the buffer group, end its input"), but a slow consumer with a fast producer is an ordinary situation - any handler that sends while a large body is still arriving - and it ends in a dropped connection rather than in flow control.
One possible shape
Pause the recv at the mark the way a prologue pauses it (pausing, landing in RECV_PAUSED), let the socket's own window hold the peer, and re-arm from the reader as the queue drains; keep whatever the kernel had already posted before the cancel landed (the ring would have to grow for that burst, since the kernel can post the rest of a CQ batch after the decision). Commit 85b61bd was an implementation along those lines; it was reverted because it was not asked for, and is only a reference.
Also observed, not explained
On the same CI runner, with that implementation pinned, the Content-Length 100 KB echo passed and the chunked 100 KB echo still came back with 0 bytes (run 34420184914), in under a second, so not a timeout. Locally it passes with every chunk size and split pattern tried. Possibly a different problem in the chunked body path, possibly not; it has not been diagnosed.
What happens
A connection whose handler is parked on something other than a read - a send,
ioxd_delay, a socket client call - while its peer keeps sending has its multishot recv filling the per-connection receive queue (struct spsc,RX_QUEUE= 64 delivered buffers). When the queue is full,ioxd__conn_on_recvreturns the buffer, ends the input with-ENOBUFSand cancels the recv (lib/io/conn.c, theioxd__spsc_fullbranch). The handler's next read fails and the connection is closed: the peer sees a reset, or a reply cut short.Where it was seen
The HttpArena entry's streamed echo (
POST /echo: read 8 KB into the reply slab, send it, read the next 8 KB) on the arena's CI runner. A 100 KB body over TLS arrived there as small completions - well under 2 KB each - so more than 64 buffers queued while the handler waited on its first send, and the client got 0 bytes back (HttpArena PR #1473, run 34418961856). On this machine the same body arrives in 2 KB completions, 50 buffers, and never reaches the mark, somake checkand the arena validator both pass.Reproduced locally with the tiny build's queue (
-DRX_QUEUE=4, asmake check-tinybuilds it): a 100 KB echo over TLS or plain dies after 16 KB, curl exit 18/56.What it means
The queue is meant to bound how much one connection can hold of the shared buffer group (
ioxd__conn_on_recvnotes: "rather than let one peer hoard the buffer group, end its input"), but a slow consumer with a fast producer is an ordinary situation - any handler that sends while a large body is still arriving - and it ends in a dropped connection rather than in flow control.One possible shape
Pause the recv at the mark the way a prologue pauses it (
pausing, landing inRECV_PAUSED), let the socket's own window hold the peer, and re-arm from the reader as the queue drains; keep whatever the kernel had already posted before the cancel landed (the ring would have to grow for that burst, since the kernel can post the rest of a CQ batch after the decision). Commit 85b61bd was an implementation along those lines; it was reverted because it was not asked for, and is only a reference.Also observed, not explained
On the same CI runner, with that implementation pinned, the Content-Length 100 KB echo passed and the chunked 100 KB echo still came back with 0 bytes (run 34420184914), in under a second, so not a timeout. Locally it passes with every chunk size and split pattern tried. Possibly a different problem in the chunked body path, possibly not; it has not been diagnosed.