Skip to content

Fix ClickHouseDataReader: stop at the mid-stream exception block, and throw when the body ends after it - #753

Open
polyglotAI-bot wants to merge 2 commits into
mainfrom
polyglot/midstream-exception-at-end-of-body
Open

polyglotAI-bot wants to merge 2 commits into
mainfrom
polyglot/midstream-exception-at-end-of-body

Conversation

@polyglotAI-bot

@polyglotAI-bot polyglotAI-bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Fixes #750.

When a query fails after the server has sent the 200 OK, the server writes an in-band exception block into the body: \r\n__exception__\r\n<tag>\r\n<message>\n<length> <tag>\r\n__exception__\r\n (WriteBufferFromHTTPServerResponse::cancelWithException), where <tag> is the value of the X-ClickHouse-Exception-Tag header. ClickHouseDataReader looked for this block only in catch (IOException). So:

  1. When the body ends cleanly after the block at a row boundary, the reader reported a normal end of the result. The error was lost, and the result was silently truncated. This is always the case for 1-byte rows (UInt8, Int8, Bool, Enum8), and for fixed-width rows when the block length is a multiple of the row width. Today a real server breaks the chunked body after the block, so HttpClient throws HttpIOException, but a proxy that re-frames the body ends it cleanly.
  2. Before the error surfaced, the reader decoded the bytes of the block as rows and returned them. This also occurs on the real-server path: on main, the reader returns 78 rows decoded from the block before the exception.

Changes

  • ClickHouseDataReader: Read() and the POCO fast path TryMaterializeNextRow<T>() now use one helper, HasNextRow(), at each row boundary:
    • At the end of the body, if the last bytes received are the closing marker with the tag (<tag>\r\n__exception__\r\n), it throws the ClickHouseServerException, parsed from the ring buffer of ExceptionTagAwareStream. When a long message has pushed the opening marker out of the 4 KiB ring buffer, it throws with the existing "Unknown error (could not parse exception message)" text instead of reporting a complete result. This check does not allocate unless the closing marker is there.
    • When the next bytes in the read buffer are the opening marker exactly as the server writes it (\r\n__exception__\r\n<tag>), it reads the rest of the body (at most 16 KiB, the server's MAX_EXCEPTION_SIZE), parses it and throws, so that no row is decoded from the block. A cheap inline test (\r, then \n in the buffer) comes first, so that most rows pay no call.
  • ExceptionTagAwareStream: MatchOpeningMarker (exact prefix compare, None/Partial/Full), EndsWithClosingMarker (ring-buffer tail, no allocation), and TryParseMidStreamException(byte[]) (the existing parse, on a given buffer).
  • PooledReadBufferStream: NextBufferedByte and Peek(minimumCount), which compacts the buffer and reads more when the opening marker continues past the buffered bytes. It reads only while the bytes still match the marker. For a tagged response, FromHttpResponseAsync makes the read buffer at least as long as the marker (33 bytes), so that a very small ReadBufferSize does not turn the check off.

Streaming trade-off: when the bytes of a row received so far are a prefix of the marker (for example a single UInt8 column whose value 13 is the last byte of a server write), Read() now returns that row only when the next bytes arrive.

The row-boundary check accepts only the server's exact framing. A block written in another way (for example without the leading \r\n, as in the existing mock fixtures), or a block that does not start at a row boundary, is still found by the lenient checks at the end of the body, but the rows decoded from it before that are returned.

Test

New and strengthened tests fail on main (19 of them) and pass with this change:

  • MidStreamExceptionMockTests (mock responses, no server):
    • Read_ExceptionBlockAfterLastRow_ReturnsOnlyTheRowsThenThrowsServerException (parametrized): UInt8 rows; aligned Int32 rows; a 64-byte read buffer that splits the marker after 3 bytes; the marker's \r as the last buffered byte; one byte per read (as a chunked body can give); ReadBufferSize 1; a 6000-character message (longer than the ring buffer). Each asserts exactly the real rows, then ClickHouseServerException with code 395 and the exact message. On main: no exception.
    • TryMaterializeNextRow_ExceptionBlockAfterLastRow_ReturnsOnlyTheRowsThenThrowsServerException: the POCO fast path.
    • Read_ExceptionBlockNotAtRowBoundaryAndBodyEndsCleanly_ThrowsServerException (parametrized): block 1 byte into an Int32 row; block without the leading CRLF; block inside a row with a message longer than the ring buffer. On main: no exception.
    • Read_RowsThatLookLikeTheMarkerWithoutTheTag_AreReturnedAsData (contrast): rows that start like the marker, and a body that ends with __exception__, but with no tag. They stay data.
  • ResponseDecompressionTests.FromHttpResponseAsync_WithCompressedExceptionBlockAfterLastRow_ReturnsOnlyTheRows: the same through each response codec.
  • Real server (25.11+): ShouldDetectMidStreamException_AfterResponseIsCommitted and QueryAsync_MidStreamExceptionAfterResponseIsCommitted_ThrowsClickHouseServerException now also assert that every row returned is a row the server sent (added assertions only). On main: row 200000 of 200078 is not a row the server sent (78 rows decoded from the block).
  • Unit tests for MatchOpeningMarker, EndsWithClosingMarker (also across the end of the ring buffer) and PooledReadBufferStream.Peek.

Full ClickHouse.Driver.Tests suite on net10.0 against server 26.10.4.207: 14920 passed, 0 failed. The library also builds for net6.0 and net8.0.

Performance

The row-boundary check adds work to every row. Measured with an ad-hoc Stopwatch harness (not committed): a mock response (no network), Read() loop with no accessor calls, best of 9, Release, net10.0. A /benchmark-compare run would confirm it against a server.

Body main this PR
20M UInt8 rows, all 0 110.2 ms 111.3 ms (+1%)
20M UInt8 rows, pseudo-random 110.3 ms 112.8 ms (+2%)
5M Int64 rows, pseudo-random 65.0 ms 65.9 ms (+1%)
5M String rows of 13 characters (each row starts with byte 0x0D) 146.4 ms 156.5 ms (+7%)
20M UInt8 rows, all 13 (\r): worst case 110.1 ms 139.5 ms (+27%)

Run-to-run noise was about ±3%. The cost is per row, so it is smaller for rows with more columns. The end-of-body check runs once per result. If the per-row cost is not acceptable, the end-of-body check alone fixes the lost error (point 1) with no per-row cost; the row-boundary check (point 2) is separate code in HasNextRow and can be removed.

Related

#751 (a block longer than the 4 KiB ring buffer is lost): a block that starts at a row boundary is now read into its own buffer, so the real-server case of #751 gets the full message (checked against 26.10 with throwIf(number = 200000, repeat('x', N)): N = 6000 and 15000 give ClickHouseServerException code 395 with the whole message; main gives HttpIOException). A block that does not start at a row boundary, on a body that the server breaks, still uses the ring buffer, so this PR does not close #751.

Our open PR #479 (in-band exception on the raw streaming path, ClickHouseRawResult) also changes ExceptionTagAwareStream.cs and MidStreamExceptionTests.cs. Its new mode is opt-in and not used by the reader, so the two changes do not interact; the PR that merges second may need a text merge.

Checklist

  • Unit and integration tests covering the common scenarios were added
  • A human-readable description of the changes was provided to include in CHANGELOG (changelog.d/750-midstream-exception-end-of-body.fixes.md)

Pre-PR validation gate

  • Deterministic repro confirmed (mock tests and a real-server test fail on main)
  • Root cause documented above
  • Fix targets the root cause (the reader checked for the block only on an IOException)
  • Tests fail without the fix, pass with it
  • No existing test weakened (two real-server tests got added assertions; one test helper got an optional type parameter)
  • Convention compliance verified per AGENTS.md and ClickHouse.Driver/AGENTS.md (changelog fragment, parametrized tests, measured per-row cost)

🤖 Generated with Claude Code


Note

Medium Risk
Changes core row-reading logic for all tagged responses with a per-row peek cost; behavior is well-tested but affects every Read() on failing queries after rows start streaming.

Overview
Fixes #750 by teaching ClickHouseDataReader to detect ClickHouse’s in-band exception block on every row boundary, not only when an IOException occurs.

Read() and TryMaterializeNextRow<T>() now call HasNextRow() instead of a simple end-of-stream peek. For responses with X-ClickHouse-Exception-Tag, the reader:

  • Peeks for the server’s opening marker (\r\n__exception__\r\n<tag>) and throws ClickHouseServerException before decoding rows from the block.
  • On a clean body end, checks EndsWithClosingMarker() and throws rather than treating a truncated failed query as success (e.g. after a proxy ends the response).

Supporting changes: ExceptionTagAwareStream gains MatchOpeningMarker, EndsWithClosingMarker, and TryParseMidStreamException. PooledReadBufferStream adds Peek / NextBufferedByte. Tagged responses bump the read buffer to at least the marker length. Docs and a changelog entry describe behavior and limits; tests cover mock HTTP, compression, POCO path, and live-server mid-stream failures.

Reviewed by Cursor Bugbot for commit b8b7bb7. Bugbot is set up for automated code reviews on this repo. Configure here.

… throw when the body ends after it

The reader looked for the server's in-band exception block only after an
IOException. When the body ended cleanly after the block at a row boundary,
the result was reported as complete; and before the error surfaced, the
bytes of the block were returned as rows.

- At the end of the body, a closing marker with the tag now throws the
  ClickHouseServerException (parsed from the ring buffer when possible).
- At a row boundary, the opening marker as the server writes it
  ("\r\n__exception__\r\n<tag>") is matched in the read buffer, so the block
  is read and thrown before it is decoded as rows.

Fixes: #750

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The release note overstates row validity, while reader documentation omits the lookahead delay, clean-EOF exception behavior, and potential invalid rows from mid-row exception blocks.

2 open issues
What changed in this PR

This PR addresses #750 in the HTTP client by detecting mid-stream server exceptions before their bytes become rows when standard framing starts at a row boundary. It also prevents a cleanly ended response from hiding a trailing server error.

Changes:

  • Shares exception checks between Read() and POCO materialization.
  • Adds buffered lookahead and tagged closing-marker detection.
  • Expands mock, compression, and live-server test coverage, and adds a changelog entry.
File Description
ClickHouse.Driver/​Formats/​PooledReadBufferStream.cs Adds non-consuming buffered lookahead.
ClickHouse.Driver/​Formats/​ExceptionTagAwareStream.cs Adds opening- and closing-marker detection and buffer parsing.
ClickHouse.Driver/​ADO/​Readers/​ClickHouseDataReader.cs Checks for exception blocks at row boundaries and response completion.
ClickHouse.Driver.Tests/​Http/​ResponseDecompressionTests.cs Tests exception detection through response compression codecs.
ClickHouse.Driver.Tests/​Formats/​PooledReadBufferStreamTests.cs Tests lookahead across refills and end of stream.
ClickHouse.Driver.Tests/​Formats/​ExceptionTagAwareStreamTests.cs Tests marker matching and ring-buffer wraparound.
ClickHouse.Driver.Tests/​ADO/​MidStreamExceptionTests.cs Adds boundary cases and strengthens returned-row assertions.
changelog.d/​750-midstream-exception-end-of-body.fixes.md Records the reader’s exception-handling fix.

Review effort: Balanced

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on c7ba96e. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

security-ai-gate-reviewer Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

AI review gate — auto-approved

b8b7bb7aaf56 is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • b8b7bb7aaf56 approved — green: Cursor Bugbot
  • c7ba96e046fb approved — green: Cursor Bugbot

…ngelog claim

docs/http.mdx: new "Errors after the first rows" section under the
ADO.NET reader (row-boundary check, clean end of the response, and its
limits: blocks that do not start at a row boundary, the 4 KiB tail, and
the wait for a row whose bytes so far match the start of the block),
linked from POCO reads.

changelog.d: limit the claim to blocks that start at a row boundary
with the server's framing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on b8b7bb7. Details in the approver's comment on this pull request.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants