Skip to content

fix: decode control character escapes in Enum names - #2120

Merged
joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
xmasdev:fix-enum-control-escapes
Nov 20, 2026
Merged

joe-clickhouse merged 4 commits into
ClickHouse:mainfrom
xmasdev:fix-enum-control-escapes

Conversation

@xmasdev

@xmasdev xmasdev commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes #2119.
Fixes #1839.

The server writes control characters in Enum8 / Enum16 names as escape sequences in the type name (\b, \f, \n, \r, \t, \0). extractEnumNamedValues removed the backslash and kept the next byte, so the name 'a\nb' became anb. A scan returned the wrong string, Append("a\nb") failed with unknown element, and Append("anb") was accepted and stored a\nb on the server.

The parser now decodes each name into a buffer while it scans the type, and it maps those six escapes to the control bytes. Other escaped bytes (\\, \') are kept as they are, so 'a\\nb' is still a backslash followed by n.

Because the name is built during the scan, the skippedValueTokens offsets are gone. Those were not reset between names and were not adjusted after each removal, which is the bug in #1839. On main the type Enum8('a\'b\'c'=1,'d\'e'=2,'f'=3) panics with slice bounds out of range. So this change also covers what #1844 fixes. I don't want to step on that PR, and I can rebase on top of it if you prefer to merge it first.

Checklist

  • Unit and integration tests covering the common scenarios were added

TestExtractEnumNamedValues has new cases for the control escapes, an escaped backslash, several escapes in one type and bytes the server does not escape. Test2119 in tests/issues appends and scans the names with Enum8, Enum16, Array(Enum8) and Nullable(Enum8) over native and HTTP. I ran it on 26.9. It fails on main and passes with this change.


Note

Medium Risk
Changes core Enum type parsing used for all Enum columns; behavior shifts for escaped names but is covered by new unit and integration tests.

Overview
Fixes Enum8/Enum16 type parsing so label names match what ClickHouse sends and accepts on append/scan.

extractEnumNamedValues now builds each name in a buffer while scanning the type string instead of slicing then stripping backslashes with skippedValueTokens. A new unescapeEnumChar maps server escapes (\b, \f, \n, \r, \t, \0) to control bytes; other escapes like \\ and \' stay literal. That corrects wrong names (e.g. 'a\nb' → a\nb not anb) and removes the buggy offset list that caused panics on enums with several escaped quotes (e.g. Enum8('a\'b\'c'=1,…)).

Unit cases in enum_test.go cover control chars, backslashes, and multi-quote types. Integration tests Test1839 and Test2119 exercise CAST/queries and round-trip Enum8/Enum16/Array/Nullable over native and HTTP.

Reviewed by Cursor Bugbot for commit 34f7dde. Bugbot is set up for automated code reviews on this repo. Configure here.

The server writes control characters in Enum names as \b, \f, \n, \r,
\t and \0 in the type name. The parser removed the backslash and kept
the next byte, so 'a\nb' was read as "anb" and could not be appended
with its real name.

The name is now decoded into a buffer during the scan. This also removes
the escape offsets, which were not reset between names and went out of
range for names with more than one escape.
@cla-assistant

cla-assistant Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@joe-clickhouse
joe-clickhouse requested a review from a team as a code owner November 20, 2026 18:19
@joe-clickhouse

Copy link
Copy Markdown
Contributor

Thanks for this, nice fix. Building the name while scanning is much cleaner than the offset list, and it gets rid of the panic too.

Since this also fixes #1839 we added a small regression test for that issue's repro and merged main into the branch. We'll close #1844 in favor of this one.

@polyglotAI-bot polyglotAI-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve: no blocking problems. The one inline comment is non-blocking.

  • Test2119 and Test1839 fail on main (panic: slice bounds out of range) and pass on this branch, on ClickHouse 26.10.
  • A probe round-trips a name with each byte 0x00–0xFF in Enum8 and Enum16, native and HTTP, and the server's hex() matches. Probes for database/sql, Map, Variant and Dynamic also pass.
  • Not checked: servers older than 26.10. The CI test, lint and CodeQL runs wait for workflow approval. golangci-lint v2.11 and ./lib/... pass locally.

Comment thread lib/column/enum.go Resolved

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Co-signing @polyglotAI-bot's approval of a48a18c: the AI review gate is green on this commit.

@github-actions

Copy link
Copy Markdown
Contributor

AI review gate — Human approval in place of AI review

Approved on the head commit by polyglotAI-bot (write access, not the author), standing in for the AI review:

  • Cursor Bugbot is still reviewing this commit.

What to do:

  • Nothing to do yet if the review is running. The gate turns green when it finishes clean.
  • No review showing up? Post bugbot run as a new top-level PR comment. If that does nothing, Bugbot is probably not enabled for this repository: ask a repository administrator.

Evaluated at 34f7dde5. This required workflow does not re-run on its own once the AI review later concludes — check the Cursor Bugbot check on this commit directly for its current state.

@joe-clickhouse
joe-clickhouse merged commit 5c0b299 into ClickHouse:main Nov 20, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enum: names with \n, \t, \0 (and \r, \b, \f) escapes decode to the wrong string Bug: parsing escaped enum names with multiple escape sequences

3 participants