Skip to content

fix: correct escaped single quote parsing in Enum type strings - #1844

Closed
chaewonkong wants to merge 8 commits into
ClickHouse:mainfrom
chaewonkong:fix/enum-escaped-quote-parsing
Closed

chaewonkong wants to merge 8 commits into
ClickHouse:mainfrom
chaewonkong:fix/enum-escaped-quote-parsing

Conversation

@chaewonkong

@chaewonkong chaewonkong commented Apr 21, 2026 •

Copy link
Copy Markdown

Summary

Fix incorrect parsing of escaped single quotes in Enum8/Enum16 type strings.

Two bugs existed in extractEnumNamedValues in lib/column/enum.go:

  1. skippedValueTokens was never reset between values, causing escape positions from a previous enum
    value to bleed into the next one.
  2. When removing multiple backslashes from a single value, positions were not adjusted for prior
    removals, causing off-by-one corruption.

Checklist

Delete items not relevant to your PR:

  • Unit and integration tests covering the common scenarios were added

@chaewonkong chaewonkong changed the title fix: fix: correct escaped single quote parsing in Enum type strings Apr 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Summary

This PR fixes two real bugs in extractEnumNamedValues (lib/column/enum.go) that caused corrupted Enum8/Enum16 value names when escaped single quotes were present. Bug 1: skippedValueTokens was never cleared between values, bleeding escape offsets from one value into the next. Bug 2: when stripping multiple backslashes from a single value the removal loop did not compensate for the leftward shift caused by earlier removals. Both fixes are minimal and correct: the reset (skippedValueTokens[:0]) and the skipped-i adjustment solve exactly what is described. The unit tests added verify both scenarios.

Should fix

Missing regression test in tests/issues/issue_1839_test.go.
The project's workflow guidelines (CLAUDE.md) explicitly state: "Regression tests for bug fixes go in tests/issues/ named after the issue number." The unit tests in lib/column/enum_test.go verify the parsing logic in isolation, but there is no integration test that round-trips an Enum8 column whose names contain escaped single quotes through a real ClickHouse instance. An integration test would cover the full stack (type-string received from the server → parsed → inserted/queried) and guard against regressions from future protocol or server changes. Please add tests/issues/issue_1839_test.go with a table-driven test that creates a table with an Enum8 or Enum16 column using names like a'b and a'b'c, inserts a row, and reads it back, for both the native (clickhouse.Open) and database/sql (clickhouse_std) APIs.

Nits

  • nit: There is a blank line between the two new table entries in enum_test.go. The rest of the table has no blank lines between entries — remove the extra blank line for consistency.

Verdict

Request changes

@github-actions

Copy link
Copy Markdown
Contributor

Summary

This PR fixes two bugs in extractEnumNamedValues (lib/column/enum.go) that corrupt enum values when the Enum8/Enum16 type string contains escaped single quotes:

  1. Bleeding state: skippedValueTokens was never cleared between enum entries, so backslash positions from earlier values were reapplied when post-processing later values.
  2. Off-by-one after multiple removals: When stripping more than one backslash from a single value, positions were not adjusted for the characters already removed, leaving stale bytes in the result.

Both fixes are correct. The adjusted-position formula (skipped-i) accurately accounts for the i elements already deleted. The skippedValueTokens = skippedValueTokens[:0] reset is placed correctly — after the value is committed to values, before the next iteration — and reuses the backing array. The PR includes unit tests and a real-ClickHouse integration test that cover both failure modes.

Should fix

Integration test covers only native TCP; not HTTP or std API (tests/issues/1839_test.go). The bug lives in extractEnumNamedValues, which runs at column-type setup time, so it manifests identically over both protocols and both APIs. Adding an HTTP or database/sql variant would provide end-to-end confidence that the fix propagates through both code paths and satisfies the project checklist. A minimal addition:

func Test1839Std(t *testing.T) {
    testEnv, err := clickhouse_tests.GetTestEnvironment("issues")
    require.NoError(t, err)
    db, err := clickhouse_tests.TestClientWithDefaultSettingsStd(testEnv)
    // ... same table + insert + query pattern
}

Nits

  • nit: The two new unit-test cases in enum_test.go use Enum8. The existing suite has Enum8/Enum16 twins for several scenarios (e.g. "Enum8 single zero value" / "Enum16 single zero value"). Adding Enum16 variants for the escaped-quote cases (even if extractEnumNamedValues is shared) keeps the test matrix consistent.

Verdict

Approve

@github-actions

Copy link
Copy Markdown
Contributor

Summary

This PR fixes two bugs in extractEnumNamedValues (lib/column/enum.go) that caused corrupted Enum8/Enum16 parsing when type strings contain escaped single quotes. Bug 1: skippedValueTokens was never cleared between enum values, so escape positions from one value bled into the next. Bug 2: when stripping multiple backslashes from a single value, positions were not adjusted for earlier removals, producing off-by-one corruption. Both fixes are minimal, correct, and well-targeted. The unit tests in enum_test.go directly cover the broken cases, and the integration test in tests/issues/1839_test.go verifies the full round-trip. The approach is sound.

Should fix

  • Protocol coverage — extractEnumNamedValues runs whenever ClickHouse sends a column type string, which happens on both the native TCP and HTTP paths. The integration test only exercises clickhouse.Native. Please add an HTTP-protocol sub-test, following the pattern in tests/issues/1801_test.go:

    for _, protocol := range []clickhouse.Protocol{clickhouse.Native, clickhouse.HTTP} {
        t.Run(fmt.Sprintf("%v", protocol), func(t *testing.T) {
            conn, err := clickhouse_tests.GetConnection("issues", t, protocol,
                clickhouse_tests.TestClientDefaultSettings(testEnv), nil, nil)
            // ...
        })
    }

Nits

  • nit: Import groups in tests/issues/1839_test.go combine github.com/stretchr/testify/require (external) and clickhouse_tests "github.com/ClickHouse/clickhouse-go/v2/tests" (internal) into one block. The project convention (see tests/issues/1801_test.go) is three separate groups — stdlib → external → internal — each separated by a blank line.

Verdict

Approve — the core logic fixes are correct and the unit-test coverage is solid. Addressing protocol coverage in the integration test before merge would be ideal.

@chaewonkong

Copy link
Copy Markdown
Author

The CI failure in test (1.25, 25.10) seems unrelated to this PR. Test1229 is flaky and fails intermittently on main.

@chaewonkong
chaewonkong force-pushed the fix/enum-escaped-quote-parsing branch from 1d91cf7 to 5d0ac85 Compare May 5, 2026 06:47
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

🤖 Claude review

Fixes two bugs in extractEnumNamedValues (lib/column/enum.go) that corrupted Enum8/Enum16 value names containing escaped single quotes: escape offsets leaked from one value into the next, and multiple escapes within one value were removed at stale offsets. The fix is correct and well covered; approving.

Verified by trace:

  • For 'a\'b\'c' the recorded offsets are 1 and 4; subtracting the removal count i lands the second removal on the shifted backslash, yielding a'b'c. Offsets are appended in increasing order, so the adjustment is always exact.
  • Resetting skippedValueTokens after each value fixes the bleed. Both new unit cases fail on the old code (c'd became cd, a'b'c became a'b\c).
  • The in-place append still mutates src behind the cursor, but only within the already-consumed value range, so later values are unaffected (pre-existing, safe).

Coverage: parser unit tests for both enum widths, plus tests/issues round-trips over native TCP and HTTP. A database/sql variant would exercise the same shared parser and is not needed here.

Follow-ups (not blocking):

  • Test file name should follow the issue_<N>_test.go convention.
  • The parser only strips the backslash, so server-emitted control-character escapes (\n, \t, \0, ...) in enum names still decode incorrectly. Pre-existing and outside this PR's scope; noted as a general nit.

Blind spots: could not run the integration tests against a live server in this review; the parser behaviour was verified by hand-tracing the new unit cases.

Verdict: ✅ Approve

General findings

  • 💡 Nit — nit: name the regression test tests/issues/issue_1839_test.go
    The repo convention for new regression tests is tests/issues/issue_<N>_test.go (see issue_1862_test.go, issue_1868_test.go); the bare 1839_test.go form is the legacy pattern. Renaming keeps the newer tests discoverable together.

  • 💡 Nit — nit: control-character escapes in enum names are still decoded incorrectly (pre-existing)
    The escape branch drops only the backslash, so the fix covers \' and \\ but not the other sequences ClickHouse emits when formatting enum names (writeQuotedString escapes \n, \t, \r, \0, \b, \f). An enum value containing a newline would be reported as 'a\nb' and decoded as anb, so Append("a\nb") would fail with unknown element.

    • Not introduced by this PR; fine to track as a follow-up rather than block here.
    • Proof would be a round-trip test with Enum8('a\nb' = 1) on both protocols.

Inline comments are attached to the relevant lines. This summary updates in place on re-review.

@must108 must108 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @chaewonkong, thanks for the contributions. Please address the nits from agent, LGTM otherwise.

@joe-clickhouse

Copy link
Copy Markdown
Contributor

Thanks for the work here @chaewonkong. #2120 just merged which supersedes this. I ran all the test cases you introduced here against what I just merged and they all pass. Closing. Thanks again for the work though! Appreciate the effort.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: parsing escaped enum names with multiple escape sequences

5 participants