You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Support dictionary encoding for numeric types (DOUBLE/FLOAT/INT32/INT64) #1069
TsFile's DICTIONARY encoding is only wired for STRING/TEXT. Numeric types (DOUBLE/FLOAT/INT32/INT64) fall through to return nullptr in cpp/src/encoding/encoder_factory.h, so low-cardinality numeric data cannot use dictionary encoding.
Motivation
Parquet auto-detects low cardinality and dictionary-encodes float64 columns (RLE_DICTIONARY), storing indices instead of 8-byte values. This is the dominant reason Parquet compresses low-cardinality numeric data better than TsFile.
Benchmark on a 5214-column wide table of minute-level stock features (discrete/quantized values, ~1% distinct — e.g. 288k values with only ~2–4k distinct), all 1-chunk + zstd:
None of TsFile's existing numeric encodings match Parquet: they target smoothness/delta (Gorilla, TS_2DIFF, Chimp, Camel) or runs (RLE, RLBE), while this data is low-cardinality but scattered (no smoothness, no long runs).
Proposal
Extend DICTIONARY (or add a numeric dictionary encoding) to support DOUBLE/FLOAT/INT32/INT64. A numeric dictionary builds a map from each distinct value (bit pattern, or scaled integer) to a compact index, stores the dictionary once, then stores indices per value — the same mechanism Parquet uses.
Investigated the current state on both implementations:
Java: TSEncoding.TYPE_SUPPORTED_ENCODINGS lists DICTIONARY only for TEXT/STRING, and TSEncodingBuilder.Dictionary.getEncoder() throws UnSupportedDataTypeException for any other type. DictionaryEncoder itself is string-only (HashMap<Binary, Integer> + List<Binary>).
C++: the decoder/encoder factories accept DICTIONARY only for STRING/TEXT as well.
So numeric dictionary encoding would be a new feature in both implementations, and it has to be introduced together (shared on-disk format, cross-language read compatibility, schema-layer type restrictions, tests).
For that reason this is deferred from the 2.5.1 milestone — it can be picked up later as a combined introduction in Java and C++. Removed from the milestone and marked deferred.
Activity
ColinLeeo commentedon Nov 16, 2026
Investigated the current state on both implementations:
TSEncoding.TYPE_SUPPORTED_ENCODINGSlistsDICTIONARYonly forTEXT/STRING, andTSEncodingBuilder.Dictionary.getEncoder()throwsUnSupportedDataTypeExceptionfor any other type.DictionaryEncoderitself is string-only (HashMap<Binary, Integer>+List<Binary>).DICTIONARYonly forSTRING/TEXTas well.So numeric dictionary encoding would be a new feature in both implementations, and it has to be introduced together (shared on-disk format, cross-language read compatibility, schema-layer type restrictions, tests).
For that reason this is deferred from the 2.5.1 milestone — it can be picked up later as a combined introduction in Java and C++. Removed from the milestone and marked
deferred.