Repository navigation
Description
Describe the bug
With compression on (the default), a SELECT that fails after the server started to write the result gives a wrong ClickHouseServerException:
ErrorCodeis a number from a type name of the result header (Int32gives 32,UInt64gives 64), not the server code.Messagestarts with the binary header and rows, and the error text is at the end. It had up to 990,928 characters.
ExecuteReaderAsync, ExecuteScalarAsync and ExecuteNonQueryAsync give the same result. An error in the first block is not affected. Workaround: UseCompression=false, or the setting wait_end_of_query=1.
Steps to reproduce
- Use the default
ClickHouseClientSettings(compression on). - Run the code example below.
- Compare
ErrorCodewith theX-ClickHouse-Exception-Codeheader of the response (159).
Expected behaviour
ErrorCode is the server code from X-ClickHouse-Exception-Code (159, 395), and Message is the server error text (Code: 159. DB::Exception: Timeout exceeded: ...), as with the workaround. docs/http.mdx says: "With the default timeout_overflow_mode, the command throws a ClickHouseServerException with error code 159 (TIMEOUT_EXCEEDED)." With compression on, a timeout after rows were sent gives ErrorCode 64.
Code example
using var client = new ClickHouseClient(new ClickHouseClientSettings("Host=localhost;Port=8123"));
using var connection = client.CreateConnection();
using var command = connection.CreateCommand();
command.CommandText = "SELECT number, sleepEachRow(0.001) AS z FROM numbers(100000) SETTINGS max_block_size = 100";
command.CommandTimeout = 1;
try
{
using var reader = await command.ExecuteReaderAsync();
while (reader.Read()) { }
}
catch (ClickHouseServerException ex)
{
Console.WriteLine($"ErrorCode={ex.ErrorCode} Length={ex.Message.Length} CodeAt={ex.Message.IndexOf("Code: ")}");
}Error log
| Case (default settings if not stated) | UseCompression | ErrorCode | Message.Length | Code: at |
Message starts with |
|---|---|---|---|---|---|
Code example (CommandTimeout = 1) |
true | 64 | 8,268 | 8,123 | \x02\x06number\x01z\x06UInt64\x05UInt8 + rows |
| Code example | false | 159 | 144 | 0 | Code: 159. DB::Exception: Timeout exceeded |
Code example + wait_end_of_query=1 |
true | 159 | 145 | 0 | Code: 159. ... |
SELECT toInt32(number) AS n, throwIf(number = 1000000, 'boom') AS e FROM system.numbers LIMIT 10000000 |
true | 32 | 324,214 | 323,926 | \x02\x01n\x01e\x05Int32\x05UInt8 + rows |
throwIf(number = 200000, ...), same query, + max_block_size=1000, http_response_buffer_size=0, wait_end_of_query=0 |
true | 32 | 990,928 | 990,609 | header + rows |
Errors in the first block: throwIf(number = 5) FROM numbers(10), intDiv by zero, toInt64('abc'), unknown column |
true | 395, 153, 6, 47 (correct) | < 400 | 0 | Code: ... |
The server response, with curl: HTTP 500 with the correct X-ClickHouse-Exception-Code, and a body with the partial RowBinaryWithNamesAndTypes output before the plain error text.
$ curl -s -o body.gz -D - -H 'Accept-Encoding: gzip' \
'http://localhost:8123/?enable_http_compression=1&max_block_size=1000&http_response_buffer_size=0&wait_end_of_query=0' \
--data-binary "SELECT toInt32(number) AS n, throwIf(number = 200000, 'boom mid stream') AS e FROM system.numbers LIMIT 400000 FORMAT RowBinaryWithNamesAndTypes"
HTTP/1.1 500 Internal Server Error
X-ClickHouse-Format: RowBinaryWithNamesAndTypes
X-ClickHouse-Exception-Code: 395
Content-Encoding: gzip
(other headers omitted)
$ gzip -dc < body.gz | wc -c
850336
$ gzip -dc < body.gz | head -c 17 | od -c
0000000 002 001 n 001 e 005 I n t 3 2 005 U I n t
0000020 8
$ gzip -dc < body.gz | tail -c 330 | tr -c '[:print:]' '.'
...........Code: 395. DB::Exception: boom mid stream: while executing 'FUNCTION throwIf(equals(__table1.number, 200000_UInt32) :: 4, 'boom mid stream'_String :: 2) -> throwIf(equals(__table1.number, 200000_UInt32), 'boom mid stream'_String) UInt8 : 0'. (FUNCTION_THROW_IF_VALUE_IS_NON_ZERO) (version 26.10.4.207 (official build)).
The CommandTimeout query gives HTTP/1.1 408 Request Time-out with X-ClickHouse-Exception-Code: 159 and an 8,268-byte body that starts with the header and rows. In these runs the server put the partial output into the error body only with compression on. Without compression, the throwIf queries give HTTP 200 with an in-band exception block, and the client gives ErrorCode 395.
Root cause
ClickHouse.Driver/ClickHouseClient.cs:1195-1198(HandleError): for a body that is not empty, the exception message is the whole decompressed body. The client reads theX-ClickHouse-Exception-Codeheader only for an empty body (line 1212).ClickHouse.Driver/ClickHouseServerException.cs:36-48(ParseErrorCode): the code is the first run of digits anywhere in the string, also a digit in a type name or a row byte.
Polyglot AI can open a fix PR if a maintainer assigns this issue to @polyglotAI-bot.
Related: open PR #709 (for #705) reads the header code in HandleError, but does not change Message.
Configuration
Environment
- Client version:
mainat 9267d72 (after 1.5.0) - Language version: C#
- .NET version: net10.0 (SDK 10.0.404)
- OS: Linux (container)
ClickHouse server
- ClickHouse Server version: 26.10.4.207
- ClickHouse Server non-default settings, if any: none; the per-query settings are in the Error log table
Classification: impact: none · area: API & diagnostics (server errors)
Found by automated analysis (Polyglot AI) while working on #751. Verified against a live server.
Activity