Repository navigation
Add a harness that runs the ClickHouse core SQL tests through the driver - #624
Conversation
test/clickhouse-test-runner is a C++ program on the ODBC API (unixODBC, the ANSI driver) that imitates clickhouse-client, so that the core runner (tests/clickhouse-test of ClickHouse/ClickHouse) drives the driver unchanged. It sends each statement with SQLExecDirect, reads the columns with SQLDescribeCol / SQLColAttribute and SQLGetData (the C type of the reported SQL type), and writes the values as TabSeparated text with a type-directed renderer. The .reference files of the upstream tests are then the oracle for the driver's result sets. - The shim, its unit tests (GoogleTest) and a live test that compares its rendering with the server's FORMAT TSV output. - allowlist.txt: 9,177 tests, and 1,414 known failures in 15 groups (10 driver groups, 5 shim groups), each with a reproduction. - The version pin (26.10.4.207), scripts to fetch the upstream tests of that version, to run the allowlist (shards, KNOWN_FAILURES=<id>) and to run the same tests through the real clickhouse-client. - The workflow upstream-sql-tests.yml: nightly, manual (with a clickhouse_version input) and on changes to the harness; the driver is built once, then 4 shards. - README.md, AGENTS.md (the triage rules), CLAUDE.md, and a pointer in the root AGENTS.md. src/errorcodes.cpp is generated by scripts/gen-error-codes.sh from the upstream checkout. It is committed, because the build job compiles the shim before any checkout; the test jobs check that it is up to date. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
AI review gate — AI review passedClean review from: Cursor Bugbot. |
The test sets max_memory_usage = 2000000 so that the native protocol throws MEMORY_LIMIT_EXCEEDED inside FUNCTION moduloOrZero. Over HTTP the memory of the statement is different and depends on the output format: - real clickhouse-client (TCP), FORMAT Null: MEMORY_LIMIT_EXCEEDED in FUNCTION moduloOrZero, as the test expects; - curl (HTTP), FORMAT Null and FORMAT ODBCDriver2: no error, peak 1.17 MB; - curl (HTTP), FORMAT TSV: MEMORY_LIMIT_EXCEEDED, but in ParallelFormattingOutputFormat. The shim sends a statement with an expected error without FORMAT Null, so the result depends on the ODBCDriver2 output path of the server. The test passed in all local checks and failed on the CI runner (the query succeeded). This is an environment cause (the HTTP interface), not a driver bug, so the test is not listed (rule 7). AGENTS.md now names this case under the environment causes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On a new server, the system log tables (system.query_log and others) exist only after the first flush of the logs. The CI server runs no query before the tests, so a test that reads these tables fails when it is one of the first tests of a shard. 04409_system_documentation_source_and_entities expects the row of system.query_log in system.documentation; it failed as the 7th test of shard 4. On a new 26.10.4.207 server the row is missing, and one SYSTEM FLUSH LOGS adds it (the same with clickhouse-client). The workflow now runs SYSTEM FLUSH LOGS when the server answers. The README gives the same step for local runs, and AGENTS.md names the case under the environment causes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 06a65d0. Configure here.
ConnectorSec security review — head
|
… name The shim took a top-level `FORMAT <name>` at the end of a query as the output-format clause, also when `format` is a table or a column with an implicit alias: `SELECT * FROM format f`, `SELECT format TSV`, `SELECT 2 * format x`. It removed the words and sent a truncated statement, so the result was a syntax error instead of the server's result. The INSERT splitter had the same bug: `INSERT INTO t SELECT * FROM format f;` was read as inline data and took the next statements with it. The new operandExpected() in text.cpp tells if the token after a text is in an operand position: after a keyword such as SELECT, FROM, AS, BY or a comma or an operator. A keyword that is in an operand position itself is a name (`AS mod`, `ORDER BY offset`), so a FORMAT after it is still the clause. findFormatClause and insertDataStart both use it. topLevelWords now skips `$tag$...$tag$` heredocs, as scanStatementEnd and findFormatClause do: a FROM or SELECT in a heredoc no longer changes the decision of formatAfterSelectFrom. The old and the new code give the same result for each statement of the 12,961 upstream .sql tests, so the allowlist does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The log of CLICKHOUSE_CLIENT_CLI_LOG had the connection string with its PWD= value and the command line with the value of --password. CI uploads this log as an artifact when a job fails. The jobs use the passwordless default user now, but a run against a server with a password would write the password in clear to the artifact. The log now has the password as `<redacted>`: Runner::connectionString() has a redact_password argument, and redactedCommandLine() reads the options as parseArgs does, so it finds the value of --password also after an option that takes the next argument as its value. An empty password stays empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks. Fixed in 036de6e, as you suggested:
Tests: |
AI review gate — auto-approval withheldThe AI review gate is green, but the approver did not approve What to do: ask someone with write access who is not the PR author to approve this pull request. History (latest 5 heads, newest first)
|
ConnectorSec security review — head
|
- run-upstream-tests.sh gives the runner --dont_retry_failures. Without it, tests/clickhouse-test runs a failed test again when its output has one of its MESSAGES_TO_RETRY, and reports it as OK when it passes then, so a flaky test could enter the allowlist. RETRY_FAILED is now the only retry. - For TSVRawWithNames and TSVRawWithNamesAndTypes (and their aliases), the shim writes the names and types without escapes, as the server does (TabSeparatedRowOutputFormat::writeLine). A name with a tab or a newline was written as `\t` / `\n`. Live test: names with a tab, a backslash, a quote, a newline and non-ASCII text, compared with the server's output; the escaped formats are the contrast case. No test result changes: the allowlist is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Summary
This PR adds a harness that runs the ClickHouse core SQL tests (
tests/queries/0_stateless) through the driver. It finds driver bugs: the upstream.referencefiles are the oracle for the result sets that the driver gives to an application.test/clickhouse-test-runneris a C++ program that imitatesclickhouse-clientwell enough for the upstream runner (tests/clickhouse-test) to drive it without changes. The program is an ODBC application:SQLDriverConnectwithDriver=<path to libclickhouseodbc.so>, no DSN).SQLExecDirect. It reads the column metadata withSQLDescribeColandSQLColAttribute, and each value withSQLGetDataand the C type of the reported SQL type (for exampleSQL_C_NUMERICforSQL_DECIMAL,SQL_C_TYPE_TIMESTAMPforSQL_TYPE_TIMESTAMP,SQL_C_GUIDforSQL_GUID). Errors come fromSQLGetDiagRec.TabSeparatedtext, as the server does. A live test compares this text with the server's ownFORMAT TSVoutput for many expressions.session_idin the query string ofUrlfor all requests of an invocation. The command-line settings (log_comment,max_insert_threads,max_threads,send_logs_level) go in theUrlquery string, so with each query, not asSET.ODBCDriver2. An optionalrowbinarybackend (RowBinaryWithNamesAndTypes) is for driver development only. It is not in CI, and the README lists its gaps.The program is a separate CMake project. The root
CMakeLists.txtdoes not include it, so the driver build does not change. This is a port of the Go harness (ClickHouse/clickhouse-go#2141).Tracking issue: ClickHouse/integrations#814
Results
Server and tests pinned together:
26.10.4.207(test/clickhouse-test-runner/clickhouse-version; the upstream tagv26.10.4.207-stable)..sqltests: 9,356 OK, 2,679 FAIL, 926 skipped (tags), 844 s at-j 6. Each failure has a cause (environment, driver, or shim).odbcdriver2, HTTP). Noonly=lines (one protocol).[untriaged]group.KNOWN_FAILURES=allruns them: 1,414 of 1,414 fail, each with the error of its group.AGENTS.md, "Tests that are left out for load").04101_variant_adaptor_propagate_exceptionsis left out after the first CI run: it expectsMEMORY_LIMIT_EXCEEDEDatmax_memory_usage = 2000000, a limit set for the native protocol. Over HTTP the memory of the statement depends on the output format (curl: no error withFORMAT NullandFORMAT ODBCDriver2); it failed on the CI runner and passed locally (AGENTS.md, "Environment causes").-j 6(more parallel than CI): 9,177 OK, 0 FAIL each, 526-529 s (before04101_variant_adaptor_propagate_exceptionswas left out).buildjob builds the driver once (Release, Clang 18, asLinux.yml), and runs the unit tests of the shim with Clang 18 and GCC 11, with warnings as errors. Then 4 shards of about 2,300 tests each, at-j 4. Each test job runsSYSTEM FLUSH LOGSonce when the server answers: on a new server the system log tables (system.query_logand others) exist only after the first flush, and04409_system_documentation_source_and_entitiesfailed when it ran as one of the first tests of a shard. Shard 1 measured about 3 minutes on 4 CPUs.src/errorcodes.cpp(error names for the test hints) is generated byscripts/gen-error-codes.shfromsrc/Common/ErrorCodes.cppof the upstream checkout. It is committed, because thebuildjob compiles the shim before it gets the upstream tests. For the pinned version, the test jobs fail when the committed file is not the same as the generated one.Client bugs found
These are not fixed in this PR, and no issues are filed. The maintainers decide. All are over HTTP (the only protocol of the driver), with the ANSI driver.
Groups in
allowlist.txt(known failures):[type-name-string](798 tests): the driver gives the base type name only.SQL_DESC_TYPE_NAMEisStringforArray,Tuple,Map,Enum,IPv4/IPv6,Int128/Int256,Time64and other types, andNullable(...),LowCardinality(...)and type parameters are lost. So an application cannot tell anArray(String)from aString. The full name is in theODBCDriver2header, but no call returns it.SQLExecDirect("SELECT ['a']");SQLColAttribute(1, SQL_DESC_TYPE_NAME)→ actualString(SQL_VARCHAR); expectedArray(String).[escape-scan](120): ODBC escape processing changes string literals, without an error.replaceEscapeSequencesalso runs inside'...'.SQLExecDirect("SELECT 'a{}b'")→ actualab; expecteda{}b. (SQL_ATTR_NOSCAN = SQL_NOSCAN_ONstops it.)[sql-scan-ternary](117): the ternary operatorc ? a : bcannot be sent: each?outside quotes is a parameter marker, also withSQL_ATTR_NOSCAN. Maybe by design.SQLExecDirect("SELECT 1 ? 2 : 3")→ actualSYNTAX_ERROR; expected2.[numeric-64bit](30):SQL_C_NUMERICfails for values over 2^64-1. The internal value of the driver has 64 bits;SQL_NUMERIC_STRUCT.valhas 128.SELECT toDecimal128('18446744073709551616', 0);SQLGetDatawithSQL_ARD_TYPE=SQL_C_NUMERIC(38, 0) → actual "value is too big for internal representation"; expected the value.[year-zero](20):Date32/DateTime64in year 0000 read as 1970 through the struct types (silent wrong value).value_manip::normalize_datemaps year 0 to 1970.SELECT makeDate32(0, 1, 2);SQLGetData(SQL_C_TYPE_DATE)→ actual 1970-01-02; expected 0000-01-02 (SQL_C_CHARgives0000-01-02).[time-range](18):Timeis reported asSQL_TYPE_TIME, but holds -999:59:59 .. 999:59:59, so values outside 00:00:00 .. 23:59:59 fail asSQL_C_TYPE_TIME.SELECT toTime('24:00:00');SQLGetData(SQL_C_TYPE_TIME)→ actual[HY000] Cannot interpret '24:00:00' as TIME; expected a SQL type whose C type holds the value.[sql-scan](10): the parameter-marker scan knows only'...'and"...", not comments, backticks or$$...$$. Wrong values without an error, or[42000].SQLExecDirect("SELECT $$a?b$$")→ actuala{odbc_positional_1:Nullable(Nothing)}b; expecteda?b.SQLExecDirect("SELECT 1 -- what @ is this")→ actual[42000]; expected1.[float-subnormal](6):SQL_C_DOUBLE/SQL_C_FLOATfail for subnormal values (std::stod/std::stofgiveout_of_range).SELECT toFloat64(5e-324);SQLGetData(SQL_C_DOUBLE)→ actual "Cannot interpret '5e-324' as double"; expected 4.94e-324.[sql-scan-at-sign](5): ClickHouse syntax with@is taken as a named parameter.SQLExecDirect("DROP USER IF EXISTS u1@'%'")→ actual[42000] Syntax error or access violation; expected success.[datetime-output-format](3): withdate_time_output_formatother thansimple,SQL_C_TYPE_TIMESTAMPfails, because the driver parses the server's output text.SELECT toDateTime('2020-10-14 21:00:00', 'UTC') SETTINGS date_time_output_format = 'iso';SQLGetData(SQL_C_TYPE_TIMESTAMP)→ actual "Cannot interpret '2020-10-14T21:00:00Z' as TIMESTAMP"; expected the timestamp.Bugs found with the bug classes of the Go harness. No test in the allowlist shows them, because the shim does not use these functions:
[dt64-param-range]: a boundSQL_TYPE_TIMESTAMPparameter outside 1677-09-21 .. 2262-04-11 fails. The driver always sends it asDateTime64(9)(int64 nanoseconds), whatever theDecimalDigits(driver/utils/type_info.cpp).SQLBindParameter(SQL_C_TYPE_TIMESTAMP, SQL_TYPE_TIMESTAMP, 29, 9)with 1600-01-01 00:00:00.123,INSERTinto aDateTime64(3)column → actualDECIMAL_OVERFLOW ... cannot be parsed as Nullable(DateTime64(9)); expected the row. A literal in the SQL text works.[timeout-no-cancel]: after a driver timeout, the query keeps running on the server and the session stays locked.Timeoutdrops the HTTP request only, andSQLCancelonly closes the cursor.Timeout=1and asession_id;SQLExecDirect("SELECT sleep(3)")→[HY000] Timeout(expected SQLSTATEHYT00). Each next statement on the connection fails withSESSION_IS_LOCKEDuntil the server query ends; expected: the server query is cancelled.[driver-sql-in-session]:SQLGetInfo(SQL_DBMS_VER)and the catalog functions that build SQL (SQLTables,SQLColumns,SQLGetTypeInfo) run their SQL in the user's session. They are inquery_logas the user's queries, and fail when the session has another dialect.SET allow_experimental_kusto_dialect = 1,SET dialect = 'kusto', thenSQLGetInfo(SQL_DBMS_VER)orSQLTables→ actualSYNTAX_ERROR; expected the value or the list. (SQLExecDirectsends no query of its own.)[session-off-by-default](by design and documented, for a decision): withoutAutoSessionId=onor asession_idinUrl, each statement runs in a new session, without an error or a warning.SQLExecDirect("SET max_threads = 3")→SQL_SUCCESS; thenSELECT getSetting('max_threads')gives the server default.CREATE TEMPORARY TABLE t ...succeeds, thenSELECT count() FROM tgivesUNKNOWN_TABLE.Shim gaps (groups marked
shiminallowlist.txt; not driver bugs):[shim-output-format](280): output formats other than theTabSeparatedfamily andNull. The driver reads onlyODBCDriver2/RowBinaryWithNamesAndTypes, so the shim renders results itself, and does not implementVertical,Pretty*,JSON*,CSVand binary formats.[shim-output-setting](4): settings that change the server's text output (output_format_decimal_trailing_zeros,format_tsv_null_representation, ...).[shim-explain-ast-format](1),[shim-format-null-error-text](1),[shim-kql-split](1): see the group headers.Server-side or format causes, for a person to look at (not listed as known failures, see
AGENTS.md, "Environment causes"):ODBCDriver2has no marker for the totals row: withWITH TOTALS, the totals row comes as a data row (109 tests).ODBCDriver2sends no extremes withextremes = 1(13 tests).FORMAT Null) can end before the server queues itsQueryFinishrow inquery_log, so aSYSTEM FLUSH LOGSin the next request can miss it. A plaincurlshows the same race under load; the real client (TCP) does not. 92 tests are left out for it.Run locally
Build the driver (
build/driver/libclickhouseodbc.so), start the pinned server, then:KNOWN_FAILURES=<group-id>runs one group.scripts/run-with-real-client.shruns the same tests through the realclickhouse-client(for triage). The README has the details: the environment variables, how the shim imitatesclickhouse-client, the allowlist format, how to grow the allowlist, CI, and how to bump the version.AGENTS.mdhas the triage rules.Checks
clang-format18 with the repository's.clang-format: clean. (clang-format 18 refuses that file, because two keys occur two times; the README shows the check with a copy that keeps the last value of each key.)shellcheck -x,actionlint: clean.CMakeLists.txtdoes not includetest/).🤖 Generated with Claude Code
Note
Low Risk
Adds isolated test/CI tooling and documentation without changing driver runtime code; main impact is new GitHub Actions cost and workflow maintenance.
Overview
Introduces
test/clickhouse-test-runner, an ODBC-basedclickhouse-clientshim that lets upstreamtests/clickhouse-testrun ClickHouse0_statelessSQL tests against the ANSI driver, with TabSeparated rendering fromSQLGetData, session/FORMAT/hint handling, unit tests, andlive_test.py.Adds
.github/workflows/upstream-sql-tests.yml: build the driver and verify the shim under Clang/GCC, fetch pinned upstream tests (26.10.4.207), run a large allowlist in four CI shards, optional manual version override, and failure artifacts. Shell helpers handle upstream checkout, sharding, and triage via the real client.Root
AGENTS.mdnow points agents at harness triage rules; the shim is a separate CMake project and does not modify the driver build.Reviewed by Cursor Bugbot for commit 884a740. Bugbot is set up for automated code reviews on this repo. Configure here.