Skip to content

fix: request flattened JSON and Dynamic output by default - #2149

Draft
joe-clickhouse wants to merge 11 commits into
mainfrom
fix/1854-flattened-default
Draft

joe-clickhouse wants to merge 11 commits into
mainfrom
fix/1854-flattened-default

Conversation

@joe-clickhouse

@joe-clickhouse joe-clickhouse commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Depends on #2004. Merge that PR first.

On ClickHouse 25.6 and later, the driver requests flattened output for JSON and Dynamic when the user’s settings allow it. The driver checks once per connection. To opt out, set DisableAutoFlattenedJSONOutput to true. If a session changes permissions after connecting, the next query can fail.

The query from #1854 returned all four rows on ClickHouse 26.2 with default settings. The regression tests passed on 25.5 and 26.8 over TCP and HTTP.

Closes #1854.


Note

Medium Risk
Changes default connection setup (extra probe query) and per-query settings on 25.6+, which can affect quotas, readonly profiles, and JSON/Dynamic read behavior for existing deployments.

Overview
On ClickHouse 25.6+, the driver now probes once per connection (system.settings) and, when allowed, automatically sends output_format_native_use_flattened_dynamic_and_json_serialization=1 on normal queries over Native and HTTP. Options.DisableAutoFlattenedJSONOutput and DSN disable_auto_flattened_json_output skip the probe and the automatic default; explicit connection/query settings still win. Probe failures from ClickHouse exceptions keep the connection and turn the feature off; hard probe errors fail the dial.

HTTP adds per-connection backoff when automatic setting triggers READONLY (164) or SETTING_CONSTRAINT_VIOLATION (452). QueryFormat / InsertFormat are marked rawFormat so they do not get the automatic setting.

Legacy JSON shared data and Dynamic SharedVariant decoding now fail with a directed error (flattened setting, casts, profile hints) instead of silently mis-reading. Examples and tests drop manual SET for flattened output and add coverage for probes, opt-out, profiles, and issue #1854.

Reviewed by Cursor Bugbot for commit c16c412. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

AI review gate — Draft pull request — AI review not started

  • Cursor Bugbot does not review draft pull requests, so the gate cannot pass yet.

What to do:

  • Mark the pull request Ready for review. Bugbot then reviews it.
  • The bot re-runs this gate automatically when you mark the pull request ready or the review lands; or comment /ai-gate check.

Re-run the gate with “Re-run jobs” after an approval or a finished review. A push re-runs it automatically.

@joe-clickhouse
joe-clickhouse marked this pull request as ready for review November 21, 2026 15:07
@joe-clickhouse
joe-clickhouse marked this pull request as draft November 21, 2026 15:15

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on c16c412. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

Copy link
Copy Markdown

AI review gate — auto-approved

c16c412c87b9 is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • c16c412c87b9 approved — green: Cursor Bugbot

Comment thread conn_http.go Resolved
Comment thread tests/issues/issue_1854_default_test.go Resolved
@polyglotAI-bot

polyglotAI-bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Requests flattened JSON and Dynamic output by default in both Go APIs over Native TCP and HTTP, with explicit overrides and errors for unsupported legacy storage. The earlier role finding was withdrawn, and Native database/sql recovery now has code and regression coverage. No remaining findings warrant changes. ✅ Approve.

Final verdict

✅ Approve

Automated static review (no build or test run) by Polyglot AI (gpt-6-astra) of 4db09d1. Follow-up review of new commits after a change request; the first review was requested by @joe-clickhouse. A maintainer can ask for a new review by mentioning @polyglotAI-bot or requesting a review from it.

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested in the review of c16c412: see the summary comment and the inline threads.

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: two blocking problems. The two inline comments have the details.

  • On ClickHouse 26.10, all TestIssue1854* tests pass, except TestIssue1854SessionRecovery/HTTP/database_sql, which fails as in CI.
  • The earlier role finding on conn_http.go does not reproduce. Its thread has the run result.
  • Not checked: ClickHouse Cloud, and servers 25.6 to 26.2.

Comment thread conn.go Outdated
}

settings := make([]proto.Setting, 0, len(c.opt.Settings)+len(querySettings))
useFlattenedJSON := c.opt.useFlattenedJSON(c.flattenedJSON, querySettings)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: Native database/sql connection fails every query after a session change.

  • After SET readonly = 1 on a Native connection, each later query on it fails with 164 for this setting.
  • database/sql keeps the connection, because an *Exception is not driver.ErrBadConn. Native has no backoff like observeError on HTTP.
  • This applies to a reserved sql.Conn and to a pooled connection.

Run on ClickHouse 26.10: 3 of 3 queries fail. With DisableAutoFlattenedJSONOutput (the requests of main), 3 of 3 succeed. The docs say that a rejection retires the Native socket. That is true only for the ClickHouse API pool.

Add the same per-connection backoff on Native, and replace the skip at line 300 with a Native OpenDB case.

Reproduction on ClickHouse 26.10
db := clickhouse.OpenDB(&opts) // Native, default options
conn, _ := db.Conn(ctx)
_, _ = conn.ExecContext(ctx, "SET readonly = 1")
err := conn.QueryRowContext(ctx, "SELECT toUInt8(getSetting('readonly'))").Scan(&v)
// 3 of 3: code: 164, Cannot modify 'output_format_native_use_flattened_dynamic_and_json_serialization' setting in readonly mode

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed. Native now turns off the automatic setting for the connection after a 164 or 452 that names it, same as HTTP. The test no longer skips Native database/sql. Note that 26.8 closes the Native socket after this error and 26.10 keeps it open, so the test accepts both.

var exception *clickhouse.Exception
require.ErrorAs(t, client.scan(ctx, "SELECT toUInt8(1)", &value), &exception)
require.EqualValues(t, 164, exception.Code)
check1854Alive(t, client)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: CI fails here on 26.9 and 26.10 (HTTP, database/sql).

  • From 26.9, the server default of network_compression_method is ZSTD. The test client sends LZ4.
  • compressionSettings is set at dial (older than this PR). After SET readonly = 1, each request on the kept connection fails with 164 for network_compression_method.
  • The backoff matches only the flattened setting, so the connection does not recover.

Run on 26.10: without compression, the second query succeeds. With LZ4, 3 of 3 fail. 26.3 and 26.8 pass because their default is LZ4. CI: https://github.com/ClickHouse/clickhouse-go/actions/runs/44917268066

Set opts.Compression = nil in this test, so that it checks only the flattened backoff.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed, the test now sets Compression to nil. The compression setting failing after SET readonly is older than this PR, so I left that out of scope here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An ignored deadline-setup error can let connection probing block beyond timeout and context expiry, while HTTP decode errors leave the automatic setting enabled and the overflow test removes settings required by supported ClickHouse 24.8 servers.

3 open issues
What changed in this PR

This PR addresses #1854 by requesting flattened JSON and Dynamic output on ClickHouse 25.6+ when the user’s settings permit it. It builds on #2004’s safeguards for unsupported legacy data.

Changes:

  • Adds connection probes, automatic request settings, and an option to disable the feature.
  • Rejects unsupported legacy shared data with actionable errors.
  • Adds regression coverage and updates documentation and examples to use the automatic default.
File Description
tests/​utils.go Removes explicit flattened settings from test helpers.
tests/​std/​json_test.go Uses automatic defaults in JSON tests.
tests/​std/​dynamic_test.go Uses automatic defaults in Dynamic tests.
tests/​issues/​issue_1854_test.go Tests legacy decode errors and explicit flattened output.
tests/​issues/​issue_1854_default_test.go Covers defaults, overrides, profiles, and session recovery.
README.md Introduces automatic output and the opt-out option.
lib/​column/​json_deprecated.go Validates shared offsets and rejects unsupported data.
lib/​column/​json_deprecated_test.go Tests legacy shared-data validation and errors.
lib/​column/​dynamic_deprecated.go Rejects unsupported SharedVariant values.
examples/​std/​json_paths.go Removes manual flattened-output setup.
examples/​std/​dynamic.go Removes manual flattened-output setup.
examples/​clickhouse_api/​json_structs.go Uses the automatic output default.
examples/​clickhouse_api/​json_strings.go Removes the explicit flattened setting.
examples/​clickhouse_api/​json_paths.go Uses the automatic output default.
examples/​clickhouse_api/​json_fast_structs.go Uses the automatic output default.
examples/​clickhouse_api/​dynamic.go Uses the automatic output default.
docs/​data-types.mdx Explains legacy read limits and alternatives.
docs/​config-reference.mdx Documents probes, overrides, and recovery behavior.
context.go Adds internal probe and request flags.
conn.go Integrates Native probing and automatic settings.
conn_send_query.go Tracks automatic settings and bypasses settings for probes.
conn_process.go Observes Native setting-rejection errors.
conn_json.go Implements eligibility probes and rejection tracking.
conn_json_test.go Tests probes, cancellation, recovery, and raw formats.
conn_http.go Integrates HTTP probing and automatic settings.
conn_http_format.go Excludes raw-format requests from automatic settings.
clickhouse_options.go Adds the public opt-out option and DSN parsing.

Review effort: Balanced

Comment thread conn_json.go
if ctx.Err() != nil {
return ctx.Err()
}
c.startReadWriteTimeout(ctx)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the same as every other Native query path, which also ignores this error. Returning it here would fail the dial for connections that cannot take deadlines, like SSH tunnels. We track the general fix for writes on such connections separately, so I'm leaving it out of this PR.

Comment thread conn_http.go

func (h *httpConnect) executeRequest(req *http.Request) (*http.Response, error) {
func (h *httpConnect) executeRequest(req *http.Request, automaticJSON bool) (_ *http.Response, err error) {
defer func() { h.jsonSerialization.observeError(automaticJSON, err) }()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The server checks settings before it writes the status line, so a 164 or 452 for this setting comes back as a non-200 response and executeRequest sees it. CI shows it as HTTP 500. Leaving this as is.

Comment thread tests/issues/issue_1854_default_test.go Resolved
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@polyglotAI-bot
polyglotAI-bot dismissed their stale review November 21, 2026 17:56

The review of 4db09d1 requests no changes.

@polyglotAI-bot
polyglotAI-bot dismissed their stale review November 21, 2026 17:56

The review of 4db09d1 requests no changes.

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 4db09d1 posted: see the summary comment and the inline threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

panic: makeslice: len out of range when reading JSON column with deprecated v0 wire format

5 participants