Skip to content

[C++] DICTIONARY decoding truncates strings containing embedded NUL bytes #1034

Closed

Description

@ColinLeeo

This is a historical tracking issue for a bug already fixed on develop by #1022, merged on 2026-10-18 as 963b55237f84224a958504dd31693dfcef0afb63. It does not report a remaining bug on the current default branch. This issue should be marked completed using that existing fix; no additional fix is requested in #1026.

Observed behavior

C++ DICTIONARY decoding truncated a TEXT/STRING value at its first embedded NUL byte. For example, the three-byte string std::string("a\0b", 3) (hex 61 00 62) was read back as the one-byte string "a" (hex 61). The equivalent PLAIN-encoded string preserved all three bytes.

The behavior was reproduced while reviewing the standalone branch of #1026, whose head c01ee105d397b1ed16a4daf335db9d8176036bac does not contain the independently merged mainline fix. It was also reproduced on that PR's base commit 1180b13675e11a0230708af2cc70d037e0e12623; this is not a regression introduced by #1026. No released-version range is claimed here.

Reproduction

  1. Create a C++ table with a STRING field using DICTIONARY encoding and UNCOMPRESSED compression.
  2. Write the values std::string("a\0b", 3), an empty string, and "z", passing their explicit byte lengths to common::String.
  3. Close and reopen the file, read the table, and compare both the byte lengths and contents.
  4. On the affected code, the first value has length 1 instead of 3. Repeating the same round trip with PLAIN encoding preserves length 3.

Expected: DICTIONARY preserves the complete byte sequence, including embedded and trailing NUL bytes.

Root cause and existing resolution

DictionaryDecoder uses SerializationUtil::read_var_str(std::string&, ByteStream&). The affected implementation decoded the encoded length but constructed the output using std::string(tmp_buf), which stops at the first NUL byte.

#1022 changes this to str.assign(tmp_buf, len) and adds byte-stream and dictionary regression coverage, including a Java-generated dictionary fixture. The current default branch, develop, already contains this fix. This record documents the review finding and its existing resolution rather than requesting duplicate work.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions