Repository navigation
fix: request flattened JSON and Dynamic output by default - #2149
joe-clickhouse wants to merge 11 commits into
Conversation
JSON deprecated object decoding only skipped SharedData offsets and left the Array(Tuple(String, String)) payload in the wire stream, which desynced later columns (#1854). Decode the full array and merge overflow paths into the scanned JSON object.
AI review gate — Draft pull request — AI review not started
What to do:
Re-run the gate with “Re-run jobs” after an approval or a finished review. A push re-runs it automatically. |
There was a problem hiding this comment.
Auto-approved by the AI review gate on c16c412. Details in the approver's comment on this pull request.
AI review gate — auto-approved
History (latest 5 heads, newest first)
|
|
Requests flattened Final verdict✅ Approve Automated static review (no build or test run) by Polyglot AI (gpt-6-astra) of |
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Changes requested in the review of c16c412: see the summary comment and the inline threads.
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Request changes: two blocking problems. The two inline comments have the details.
- On ClickHouse 26.10, all
TestIssue1854*tests pass, exceptTestIssue1854SessionRecovery/HTTP/database_sql, which fails as in CI. - The earlier
rolefinding onconn_http.godoes not reproduce. Its thread has the run result. - Not checked: ClickHouse Cloud, and servers 25.6 to 26.2.
| } | ||
|
|
||
| settings := make([]proto.Setting, 0, len(c.opt.Settings)+len(querySettings)) | ||
| useFlattenedJSON := c.opt.useFlattenedJSON(c.flattenedJSON, querySettings) |
There was a problem hiding this comment.
Blocking: Native database/sql connection fails every query after a session change.
- After
SET readonly = 1on a Native connection, each later query on it fails with164for this setting. database/sqlkeeps the connection, because an*Exceptionis notdriver.ErrBadConn. Native has no backoff likeobserveErroron HTTP.- This applies to a reserved
sql.Connand to a pooled connection.
Run on ClickHouse 26.10: 3 of 3 queries fail. With DisableAutoFlattenedJSONOutput (the requests of main), 3 of 3 succeed. The docs say that a rejection retires the Native socket. That is true only for the ClickHouse API pool.
Add the same per-connection backoff on Native, and replace the skip at line 300 with a Native OpenDB case.
Reproduction on ClickHouse 26.10
db := clickhouse.OpenDB(&opts) // Native, default options
conn, _ := db.Conn(ctx)
_, _ = conn.ExecContext(ctx, "SET readonly = 1")
err := conn.QueryRowContext(ctx, "SELECT toUInt8(getSetting('readonly'))").Scan(&v)
// 3 of 3: code: 164, Cannot modify 'output_format_native_use_flattened_dynamic_and_json_serialization' setting in readonly modeThere was a problem hiding this comment.
Fixed. Native now turns off the automatic setting for the connection after a 164 or 452 that names it, same as HTTP. The test no longer skips Native database/sql. Note that 26.8 closes the Native socket after this error and 26.10 keeps it open, so the test accepts both.
| var exception *clickhouse.Exception | ||
| require.ErrorAs(t, client.scan(ctx, "SELECT toUInt8(1)", &value), &exception) | ||
| require.EqualValues(t, 164, exception.Code) | ||
| check1854Alive(t, client) |
There was a problem hiding this comment.
Blocking: CI fails here on 26.9 and 26.10 (HTTP, database/sql).
- From 26.9, the server default of
network_compression_methodisZSTD. The test client sendsLZ4. compressionSettingsis set at dial (older than this PR). AfterSET readonly = 1, each request on the kept connection fails with164fornetwork_compression_method.- The backoff matches only the flattened setting, so the connection does not recover.
Run on 26.10: without compression, the second query succeeds. With LZ4, 3 of 3 fail. 26.3 and 26.8 pass because their default is LZ4. CI: https://github.com/ClickHouse/clickhouse-go/actions/runs/44917268066
Set opts.Compression = nil in this test, so that it checks only the flattened backoff.
There was a problem hiding this comment.
Fixed, the test now sets Compression to nil. The compression setting failing after SET readonly is older than this PR, so I left that out of scope here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
An ignored deadline-setup error can let connection probing block beyond timeout and context expiry, while HTTP decode errors leave the automatic setting enabled and the overflow test removes settings required by supported ClickHouse 24.8 servers.
3 open issues
What changed in this PR
This PR addresses #1854 by requesting flattened JSON and Dynamic output on ClickHouse 25.6+ when the user’s settings permit it. It builds on #2004’s safeguards for unsupported legacy data.
Changes:
- Adds connection probes, automatic request settings, and an option to disable the feature.
- Rejects unsupported legacy shared data with actionable errors.
- Adds regression coverage and updates documentation and examples to use the automatic default.
| File | Description |
|---|---|
| tests/utils.go | Removes explicit flattened settings from test helpers. |
| tests/std/json_test.go | Uses automatic defaults in JSON tests. |
| tests/std/dynamic_test.go | Uses automatic defaults in Dynamic tests. |
| tests/issues/issue_1854_test.go | Tests legacy decode errors and explicit flattened output. |
| tests/issues/issue_1854_default_test.go | Covers defaults, overrides, profiles, and session recovery. |
| README.md | Introduces automatic output and the opt-out option. |
| lib/column/json_deprecated.go | Validates shared offsets and rejects unsupported data. |
| lib/column/json_deprecated_test.go | Tests legacy shared-data validation and errors. |
| lib/column/dynamic_deprecated.go | Rejects unsupported SharedVariant values. |
| examples/std/json_paths.go | Removes manual flattened-output setup. |
| examples/std/dynamic.go | Removes manual flattened-output setup. |
| examples/clickhouse_api/json_structs.go | Uses the automatic output default. |
| examples/clickhouse_api/json_strings.go | Removes the explicit flattened setting. |
| examples/clickhouse_api/json_paths.go | Uses the automatic output default. |
| examples/clickhouse_api/json_fast_structs.go | Uses the automatic output default. |
| examples/clickhouse_api/dynamic.go | Uses the automatic output default. |
| docs/data-types.mdx | Explains legacy read limits and alternatives. |
| docs/config-reference.mdx | Documents probes, overrides, and recovery behavior. |
| context.go | Adds internal probe and request flags. |
| conn.go | Integrates Native probing and automatic settings. |
| conn_send_query.go | Tracks automatic settings and bypasses settings for probes. |
| conn_process.go | Observes Native setting-rejection errors. |
| conn_json.go | Implements eligibility probes and rejection tracking. |
| conn_json_test.go | Tests probes, cancellation, recovery, and raw formats. |
| conn_http.go | Integrates HTTP probing and automatic settings. |
| conn_http_format.go | Excludes raw-format requests from automatic settings. |
| clickhouse_options.go | Adds the public opt-out option and DSN parsing. |
Review effort: Balanced
| if ctx.Err() != nil { | ||
| return ctx.Err() | ||
| } | ||
| c.startReadWriteTimeout(ctx) |
There was a problem hiding this comment.
This is the same as every other Native query path, which also ignores this error. Returning it here would fail the dial for connections that cannot take deadlines, like SSH tunnels. We track the general fix for writes on such connections separately, so I'm leaving it out of this PR.
|
|
||
| func (h *httpConnect) executeRequest(req *http.Request) (*http.Response, error) { | ||
| func (h *httpConnect) executeRequest(req *http.Request, automaticJSON bool) (_ *http.Response, err error) { | ||
| defer func() { h.jsonSerialization.observeError(automaticJSON, err) }() |
There was a problem hiding this comment.
The server checks settings before it writes the status line, so a 164 or 452 for this setting comes back as a non-200 response and executeRequest sees it. CI shows it as HTTP 500. Leaving this as is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review of 4db09d1 requests no changes.
The review of 4db09d1 requests no changes.
polyglotAI-bot
left a comment
There was a problem hiding this comment.
Review of 4db09d1 posted: see the summary comment and the inline threads.


Depends on #2004. Merge that PR first.
On ClickHouse 25.6 and later, the driver requests flattened output for
JSONandDynamicwhen the user’s settings allow it. The driver checks once per connection. To opt out, setDisableAutoFlattenedJSONOutputtotrue. If a session changes permissions after connecting, the next query can fail.The query from #1854 returned all four rows on ClickHouse 26.2 with default settings. The regression tests passed on 25.5 and 26.8 over TCP and HTTP.
Closes #1854.
Note
Medium Risk
Changes default connection setup (extra probe query) and per-query settings on 25.6+, which can affect quotas, readonly profiles, and JSON/Dynamic read behavior for existing deployments.
Overview
On ClickHouse 25.6+, the driver now probes once per connection (
system.settings) and, when allowed, automatically sendsoutput_format_native_use_flattened_dynamic_and_json_serialization=1on normal queries over Native and HTTP.Options.DisableAutoFlattenedJSONOutputand DSNdisable_auto_flattened_json_outputskip the probe and the automatic default; explicit connection/query settings still win. Probe failures from ClickHouse exceptions keep the connection and turn the feature off; hard probe errors fail the dial.HTTP adds per-connection backoff when automatic setting triggers READONLY (164) or SETTING_CONSTRAINT_VIOLATION (452).
QueryFormat/InsertFormatare markedrawFormatso they do not get the automatic setting.Legacy JSON shared data and Dynamic SharedVariant decoding now fail with a directed error (flattened setting, casts, profile hints) instead of silently mis-reading. Examples and tests drop manual
SETfor flattened output and add coverage for probes, opt-out, profiles, and issue #1854.Reviewed by Cursor Bugbot for commit c16c412. Bugbot is set up for automated code reviews on this repo. Configure here.