Skip to content

Mid-stream errors: a server message over ~3.7 KB surfaces as HttpIOException, not ClickHouseServerException (4 KiB ring buffer) #751

Description

@polyglotAI-bot

Describe the bug

When the server fails mid-stream and its exception message is longer than about 3.7 KB, ClickHouseDataReader throws the transport error (HttpIOException: The response ended prematurely, or EndOfStreamException) instead of ClickHouseServerException. The error code and the server message are lost.

ExceptionTagAwareStream keeps only the last 4096 bytes of the body (BufferCapacity = 4096). The server writes an in-band exception block of up to 16 KiB (MAX_EXCEPTION_SIZE = 16 * 1024 in src/Server/HTTP/WriteBufferFromHTTPServerResponse.h). When the block is longer than 4096 bytes, its opening marker __exception__\r\n<tag> is no longer in the ring buffer, so TryExtractMidStreamException() returns null and the reader re-throws the IOException.

Long messages occur in normal errors, for example a parse error that quotes a long string value.

Steps to reproduce

  1. Run the code example below with Compression=false against a server that writes the in-band block (checked with 26.10.4.207).
  2. The server message for this query is 4,539 bytes (Cannot parse string '9x9x…' as Int32); the whole block is 4,610 bytes.
  3. The reader throws HttpIOException instead of ClickHouseServerException.

Results for throwIf(number = 200000, '<message>') with the message lengths below (same settings):

Message length Bytes from the opening marker to the end of the body Exception
100, 3000, 3600 up to about 4,020 ClickHouseServerException
3700, 3800, 6000, 20000 about 4,120 and more (6000: 6,419) HttpIOException: The response ended prematurely

QueryAsync<T> (TryMaterializeNextRow<T>) gives the same result (6000: HttpIOException). A mock body whose block does not start at a row boundary (an Int32 column, one stray byte, then a block with a 6,000-byte message) gives EndOfStreamException for the same reason.

Expected behaviour

ClickHouseServerException with the server's code and message, for every block that the server writes (up to 16 KiB), as for a short message.

Code example

using var client = new ClickHouseClient("Host=localhost;Port=8123;Compression=false");
using var connection = client.CreateConnection();
using var command = connection.CreateCommand();
command.CustomSettings["max_block_size"] = 1000;
command.CustomSettings["http_response_buffer_size"] = 0;
command.CustomSettings["wait_end_of_query"] = 0;
command.CommandText =
    "SELECT number AS n, toInt32(if(number = 200000, repeat('9x', 2000), '1')) AS v " +
    "FROM system.numbers LIMIT 400000";

using var reader = command.ExecuteReader();
while (reader.Read()) { } // throws HttpIOException, expected ClickHouseServerException (Code: 6)

Workaround: wait_end_of_query=1. The server then buffers the result and returns HTTP 500, and the client throws ClickHouseServerException with the full message.

Error log

System.Net.Http.HttpIOException: The response ended prematurely. (ResponseEnded)

The server sent this block (curl, same query, end of the body):

\r\n__exception__\r\neeloxtjkfowvnctj\r\nCode: 6. DB::Exception: Cannot parse string '9x9x9x…' as Int32: … (CANNOT_PARSE_TEXT) (version 26.10.4.207 (official build))\n4539 eeloxtjkfowvnctj\r\n__exception__\r\n

Root cause

  • ClickHouse.Driver/Formats/ExceptionTagAwareStream.cs:14: the ring buffer has 4096 bytes. RecordBytes keeps only the last 4096 bytes received.
  • ClickHouse.Driver/Formats/ExceptionTagAwareStream.cs:131-133: TryExtractMidStreamException() needs the opening marker in that buffer, and returns null without it.
  • ClickHouse.Driver/ADO/Readers/ClickHouseDataReader.cs:724-730 (Read()) and :648-653 (TryMaterializeNextRow<T>()): on null, the catch (IOException) re-throws the transport error.

Suggested fix

Make the ring buffer hold the largest block that the server writes: MAX_EXCEPTION_SIZE (16 KiB) plus a few bytes, because the server can add a \n after a message that it cut to fit. The buffer is allocated per reader when the response has X-ClickHouse-Exception-Tag, which the server sends for every query (checked on 26.10); renting it from ArrayPool<byte> keeps the cost low. Alternative: keep the bytes from the first opening-marker candidate on.

A test can use the query above, or throwIf with a message of 6,000 and 16,000 characters. Keep the current behavior for short blocks, and for a body with no block (EndOfStreamException).

Related: #750 (a block at the end of the body that is read as rows) changes the same detection; a fix for it that reads a block at a row boundary into its own buffer covers the server case above, but not a block that starts inside a row or the end-of-body check that uses the ring buffer. #475 / #479 use ExceptionTagAwareStream on the raw path.

Configuration

Environment

  • Client version: main at 9267d72
  • Language version: C# (net10.0)
  • .NET version: 10.0.15
  • OS: Linux (Docker)

ClickHouse server

  • ClickHouse Server version: 26.10.4.207
  • ClickHouse Server non-default settings, if any: none (query settings in the code example)
  • CREATE TABLE statements for tables involved: none (system.numbers)

Found by automated analysis of the client while working on #750, and checked against a live server, not only by inspection.

Classification: impact: none · area: API & diagnostics (mid-stream errors)

Activity

  1. added
    bugSomething isn't working
    api-diagnosticsArea: public API shape, error reporting, logging, metrics, tracing
    on Nov 20, 2026
  2. polyglotAI-bot commented on Nov 20, 2026

    @polyglotAI-bot
    CollaboratorAuthor

    The fix for #750 in #753 covers the server case of this issue: a block that starts at a row boundary is now read into its own buffer, not the 4 KiB ring buffer. Against 26.10, throwIf(number = 200000, repeat('x', N)) with the settings above gives ClickHouseServerException code 395 with the whole message for N = 6000 and N = 15000.

    Still open after #753: a block that does not start at a row boundary, on a body that the server breaks (the catch (IOException) path), still uses the ring buffer and gives the raw HttpIOException. At a clean end of the body, #753 throws ClickHouseServerException with "Unknown error (could not parse exception message)" in that case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    api-diagnosticsArea: public API shape, error reporting, logging, metrics, tracingbugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions