Skip to content

Format float and double values with std::to_chars, not std::to_string - #634

Open
polyglotAI-bot wants to merge 2 commits into
masterfrom
polyglot/float-to-chars
Open

polyglotAI-bot wants to merge 2 commits into
masterfrom
polyglot/float-to-chars

Conversation

@polyglotAI-bot

@polyglotAI-bot polyglotAI-bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Fixes #632.

from_value<float>::to_value<std::string> and from_value<double>::to_value<std::string> in driver/utils/type_info.h used std::to_string, which is printf("%f"): fixed notation with 6 decimal places. Two paths use these functions:

  • Parameters, all formats. Statement::prepareHttpRequest() converts a bound SQL_C_DOUBLE or SQL_C_FLOAT value to this text. 1e-7 was sent as 0.000000 and stored as 0.
  • Reads with RowBinaryWithNamesAndTypes. SQL_C_CHAR, SQL_C_WCHAR and SQL_C_BINARY of a Float32 or Float64 value go through convert_via_proxy<std::string> to the same functions (0.000000, 42.000000). ODBCDriver2 keeps the text of the server and does not use them.

The functions now write the shortest text that converts back to the same value, with std::to_chars. A float is formatted as a float (0.1f gives 0.1). std::to_chars does not depend on the locale; std::to_string used LC_NUMERIC of the application.

One difference from the fix suggested in the issue. std::to_chars with no format selects the shorter notation, so 100000.0 gives 1e+05 and 1700000000.0 gives 1.7e+09. The server rejects that text for integer and DateTime64 parameters, and it accepted the old text 1700000000.000000 for DateTime64(9). So the notation is the notation of the server: fixed for 1e-6 <= |x| < 1e21, scientific for other values. In this range the text is the text that the server writes for the same value, so RowBinaryWithNamesAndTypes and ODBCDriver2 give the same text. Outside it only the exponent is different (1e-07, server 1e-7). If you prefer the plain std::to_chars(first, last, src), it is a one-line change in floatToChars.

SQL_C_NUMERIC. SQL_C_NUMERIC of a Float32 or Float64 value goes through the same text to the Decimal/Numeric parser, which does not accept an exponent. Two new to_value<SQL_NUMERIC_STRUCT> specializations give that parser the fixed notation for all values. The results for the values that worked before stay the same (1000000, -2.5 → -2, 1e-7 → 0 at scale 0). Two behavior changes: a scale above 6 now keeps the digits (1e-7 at scale 9 was 0, now 0.000000100), and the fractional digits are truncated, not rounded to 6 decimals first (0.9999999 at scale 0 was 1, now 0, like ODBCDriver2).

Changes

  • driver/utils/type_info.h: new floatToChars (std::to_chars with the server notation, or a given notation); the float and double to_value<std::string> use it; new float and double to_value<SQL_NUMERIC_STRUCT> with the fixed notation.
  • driver/test/statement_parameter_binding_ut.cpp: the parameter text of bound SQL_C_DOUBLE / SQL_C_FLOAT values (Float64, Float32, Int32, Decimal parameter types).
  • driver/test/type_conversion_ut.cpp: text ↔ float/double round trips (both notations and their limits, -0, inf), and float/double → SQL_NUMERIC_STRUCT. These tests do not need a server, so they also run on macOS and Windows.
  • driver/test/statement_parameters_it.cpp: SQL_C_DOUBLE / SQL_C_FLOAT parameter → server → SQLGetData round trips.
  • driver/test/misc_it.cpp: SQLGetData of Float32 / Float64 as SQL_C_CHAR / SQL_C_WCHAR and as SQL_C_NUMERIC, in both formats. The ODBCDriver2 rows and the SQL_C_NUMERIC rows pin the output from before the change.

Test

Server 26.10.4.207, Debug build with Clang 18, unixODBC. The program repro.c from the issue now gives 1e-7, 123.456789123, 0.3333333333333333, 2.5e-10 for the bound values, and 1e-07, 123.456789123, 1e-40, 42 for SQL_C_CHAR with RowBinaryWithNamesAndTypes.

  • On master, all new tests fail except the rows that pin the old output (the ODBCDriver2 text, the SQL_C_NUMERIC rows, inf, -inf, and 0.000001, which %f gives correctly). With this change they pass.
  • With the plain std::to_chars and no SQL_NUMERIC_STRUCT specializations, the SQL_C_NUMERIC rows fail (Cannot interpret '1e+06' as Decimal/Numeric).
  • ctest -E "-dsn-[45]$": 16/16 passed. pytest: 41/41 passed with the ANSI and with the Unicode DSN.
  • Server 25.6.13.41 (the CI server, test/docker-compose.yml): ctest -E "-dsn-[45]$" 16/16, pytest 41/41 with each DSN. This server parses some float texts with an error of 1 ULP: 9.99e-07 gives 9.989999999999999e-7, and 3.4028235e+38 as Float32 gives 3.4028233e38, also in SELECT toFloat64('9.99e-07'). Server 26.10 parses them exactly, and the driver sends the correct text. So the round trips in statement_parameters_it.cpp use values that 25.6 parses exactly (9.5e-07, 3e+38). type_conversion_ut checks the driver text of 9.99e-07 and 3.4028235e+38 without a server.
  • Checked with curl and param_x: the server accepts the new texts for Float32, Float64, Decimal(P, S), integer, String and DateTime64(9) parameters (1e+20 into Decimal(15, 10) fails, as the old text did, because the value does not fit).

Floating-point std::to_chars needs macOS 13.3 in Apple's libc++. That is the value of CMAKE_OSX_DEPLOYMENT_TARGET.

Pre-PR validation gate

  • Deterministic repro confirmed (repro.c from the issue, and the new unit tests)
  • Root cause documented above
  • Fix targets the root cause
  • Test fails without fix, passes with fix
  • No existing tests broken
  • Convention compliance verified per README.md / CONTRIBUTING.md / the CI workflows (this repo has no changelog file)

🤖 Generated with Claude Code


Note

Medium Risk
Changes core float-to-text conversion used for ODBC parameters and result fetch paths; behavior shifts for edge magnitudes, Numeric binding, and locales, though coverage is broad in new tests.

Overview
Replaces std::to_string (fixed six-decimal printf style) for float and double in type_info.h with locale-independent std::to_chars via new floatToChars, using fixed notation when 1e-6 ≤ |x| < 1e21 and scientific otherwise so parameter text matches what the ClickHouse server accepts and RowBinaryWithNamesAndTypes reads align with ODBCDriver2.

Adds to_value<SQL_NUMERIC_STRUCT> for floats that routes through fixed decimal text (no exponents) for the Decimal/Numeric parser, which changes behavior for high scale and truncation vs the old six-digit round-trip path.

Tests lock the new formatting: bound SQL_C_FLOAT/SQL_C_DOUBLE parameter strings, string↔float unit tests (including inf), server round-trips for Float32/Float64, and SQLGetData as char/wchar/numeric across wire formats.

Reviewed by Cursor Bugbot for commit 1426297. Bugbot is set up for automated code reviews on this repo. Configure here.

std::to_string is printf("%f"): fixed notation with 6 decimal places.
A bound SQL_C_DOUBLE or SQL_C_FLOAT parameter was sent to the server
as this text, so 1e-7 was stored as 0. With
RowBinaryWithNamesAndTypes, SQL_C_CHAR and SQL_C_WCHAR of a Float32 or
Float64 column returned the same text ("0.000000", "42.000000").

Write the shortest text that converts back to the same value
(std::to_chars), with the notation of the server: fixed for
1e-6 <= |x| < 1e21, scientific for other values. A float is
formatted as a float. std::to_chars does not depend on the locale.

SQL_C_NUMERIC of a Float32 or Float64 value parses this text with the
Decimal/Numeric parser, which does not accept an exponent, so that
conversion uses the fixed notation for all values.

Fixes: #632

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@polyglotAI-bot
polyglotAI-bot requested review from a team and slabko as code owners November 19, 2026 17:32

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on 690ed54. Details in the approver's comment on this pull request.

@security-ai-gate-reviewer

security-ai-gate-reviewer Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

AI review gate — auto-approved

1426297bae89 is approved by security-ai-gate-reviewer[bot], an automated identity, not a human. The approval stands on the required AI review: Cursor Bugbot concluded green on this commit and the AI review gate is SUCCESS (its own comment has the verdict), and the change touches no path that requires a human reviewer. It is bound to this commit: a new push dismisses it and the approver re-evaluates.

History (latest 5 heads, newest first)

  • 1426297bae89 approved — green: Cursor Bugbot
  • 690ed54a1caa approved — green: Cursor Bugbot

The parameter round-trip tests compare the text that comes back from
the server. Server 25.6 (the CI server in test/docker-compose.yml)
parses "9.99e-07" as 9.989999999999999e-7, and "3.4028235e+38" as the
Float32 3.4028233e38, also for SELECT toFloat64('9.99e-07'). Server
26.10 parses both exactly. The driver sends the correct text.

Use "9.5e-07" and "3e+38", which the server parses exactly. They test
the same notation branches, and both fail with std::to_string.
type_conversion_ut still checks the driver text of "9.99e-07" and
"3.4028235e+38" without a server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@security-ai-gate-reviewer security-ai-gate-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Auto-approved by the AI review gate on 1426297. Details in the approver's comment on this pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parameters: SQL_C_DOUBLE and SQL_C_FLOAT values are sent with 6 decimal places (1e-7 is stored as 0)

1 participant