Skip to content

fix(json): consume v0 SharedData to keep native stream aligned - #2004

Open
BetterAndBetterII wants to merge 8 commits into
ClickHouse:mainfrom
BetterAndBetterII:fix/json-v0-shared-data
Open

BetterAndBetterII wants to merge 8 commits into
ClickHouse:mainfrom
BetterAndBetterII:fix/json-v0-shared-data

Conversation

@BetterAndBetterII

@BetterAndBetterII BetterAndBetterII commented Sep 4, 2026 •

Copy link
Copy Markdown

Reject legacy Native JSON shared data and Dynamic SharedVariant values that the driver cannot decode. This includes Dynamic values inside a JSON column. Previously, reads could lose stream alignment or return a type-tagged binary value as a Go string.

On ClickHouse 25.6 and later, set output_format_native_use_flattened_dynamic_and_json_serialization=1 to read these values. On older servers, read JSON as a string or cast Dynamic to a concrete type.

The regression covers both Go APIs over Native TCP and HTTP. It checks the decode error, a subsequent query on the same handle, and the typed values with flattened output enabled.

Part of #1854. Enabling flattened output by default on 25.6+ is a follow-up.


Note

Medium Risk
Changes low-level Native column decode for experimental JSON/Dynamic types; behavior shifts from silent mis-reads to errors, which is safer but may surface failures in apps that previously got wrong data.

Overview
Legacy V1 Native decoding for JSON and Dynamic no longer skips or mis-handles shared storage. When JSON uses shared data (e.g. paths beyond max_dynamic_paths) or Dynamic uses SharedVariant, the driver returns a clear error instead of corrupting the native stream or surfacing binary garbage as strings.

decodeSharedData_v1 reads and validates per-row shared-data offsets; non-empty shared payloads trigger unsupportedSharedDataError with guidance to enable output_format_native_use_flattened_dynamic_and_json_serialization=1 on ClickHouse 25.6+, or to CAST in SQL on older servers. Dynamic decodeData_v1 applies the same rejection when any row targets SharedVariant.

Docs add a JSON and Dynamic section describing this behavior. Unit tests cover offset validation, empty roundtrips, and shared-variant paths; issue #1854 integration tests exercise both client APIs and Native/HTTP, including connection health after a failed scan and successful reads with flattened output enabled.

Reviewed by Cursor Bugbot for commit 53d2608. Bugbot is set up for automated code reviews on this repo. Configure here.

JSON deprecated object decoding only skipped SharedData offsets and left
the Array(Tuple(String, String)) payload in the wire stream, which
desynced later columns (ClickHouse#1854). Decode the full array and merge overflow
paths into the scanned JSON object.
@CLAassistant

CLAassistant commented Sep 4, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Comment thread lib/column/json_deprecated.go Resolved
Comment thread lib/column/json.go Resolved
Comment thread lib/column/json_deprecated_test.go Resolved
@github-actions

Copy link
Copy Markdown
Contributor

🤖 Claude review

The PR decodes the full JSON v0 SharedData column (Array(Tuple(String, String))) and does not skip only the UInt64 offsets. This repairs the stream desync from #1854. The stream fix is correct, but the new value merge is not complete, thus the verdict is request changes.

Key concerns:

  • A shared-data value with a type that the new decoder does not know (for example Array(JSON), DateTime64, Decimal, Tuple) is returned as raw binary bytes with the type String. The caller gets incorrect data and no error.
  • Only rowAsJSON merges the shared data. A scan into a struct or a map does not see the overflow paths.
  • There is no test against a real server and no tests/issues/issue_1854_test.go.

Verified by reading:

  • The binary type indexes agree with DataTypesBinaryEncoding.cpp.
  • The Nullable flag and the Array varuint length agree with serializeBinary.
  • The change is in the column decoder, thus it applies to native TCP and HTTP, and to the native and database/sql APIs.

Blind spots:

Verdict: ⚠️ Request changes

Inline comments are attached to the relevant lines. This summary updates in place on re-review.

@kavirajk
kavirajk requested review from a team and polyglotAI-bot as code owners November 17, 2026 07:01

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread lib/column/json.go Resolved
Comment thread lib/column/json_deprecated.go Resolved
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

AI review gate — AI review passed

Clean review from: Cursor Bugbot.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

Reviewed by Cursor Bugbot for commit 40a6e7e. Configure here.

Comment thread lib/column/json_deprecated_test.go Resolved
polyglotAI-bot
polyglotAI-bot previously approved these changes Nov 21, 2026

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve: no blocking problems. The three inline comments are non-blocking.

  • TestIssue1854 passes on ClickHouse 24.8, 25.3, 25.6, and 26.10. It fails on main in all 12 cases (2 protocols, 2 APIs, 3 cases).
  • A read of two blocks, where only the second block has shared data, returns the first 5 rows and then the error in rows.Err(). The next query on the same one-connection pool passes, on TCP and HTTP.
  • The open Bugbot finding on TestDynamicDeprecatedSharedVariantError is not correct. ch-go ColStr writes each string with a varint length, as PutString does, and the test gets the new error (run confirmed).

Comment thread docs/data-types.mdx Resolved
Comment thread lib/column/dynamic_deprecated.go Resolved
Comment thread lib/column/json_deprecated_test.go Resolved
@polyglotAI-bot

Copy link
Copy Markdown
Collaborator

Rejects unsupported legacy JSON shared data and Dynamic SharedVariant values in both Go APIs over Native TCP and HTTP. No findings requiring changes. ✅ Approve.

Final verdict

✅ Approve

Automated static review (no build or test run) by Polyglot AI (gpt-6-astra) of a348b54. Requested by @kavirajk. A maintainer can ask for a new review by mentioning @polyglotAI-bot or requesting a review from it.

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of a348b54 posted: see the summary comment and the inline threads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants