Repository navigation
Closed
Labels
P0c++Pull requests that update c++ codePull requests that update c++ codefeatureNew featureNew featurejavaPull requests that update Java codePull requests that update Java codepythonPull requests that update Python codePull requests that update Python code
Milestone
Description
Motivation
Cross-language compatibility is currently tested with a table-model fixture matrix that covers only part of what each implementation can write. The gaps let encoding and compression bugs reach releases and push maintainers into ad-hoc manual testing.
Current state (verified)
- Java:
java/tsfile/src/test/java/org/apache/tsfile/compatibility/TableModelEncodingCompressionCompatibilityTest.java - C++:
cpp/test/reader/table_view/table_model_encoding_compression_compatibility_test.cc - Orchestration:
.github/workflows/compatibility-test.yml— each language generates fixtures, and the other language validates them (both directions).
Coverage today:
- Table model only.
- Data types: INT32, DATE, INT64, TIMESTAMP, FLOAT, DOUBLE, plus BOOLEAN on the Java side.
- Encodings: CHIMP and RLBE for the six numeric types; TS_2DIFF for FLOAT/DOUBLE (plus an explicit
max_point_number=0variant); CAMEL for DOUBLE; PLAIN for null-pattern fixtures; Java-only BOOLEAN RLE default fixtures. - C++ ships encoders for PLAIN, DICTIONARY, RLE, TS_2DIFF, GORILLA, ZIGZAG, CHIMP, SPRINTZ, RLBE, and CAMEL; Java supports the same set plus deprecated DIFF/REGULAR/BITMAP/FREQ. DICTIONARY for STRING/TEXT, GORILLA, SPRINTZ, ZIGZAG, RLE for integer types, and the STRING/TEXT/BLOB data types are not cross-validated.
- The Java and C++ matrices are maintained separately and already differ (Java-only BOOLEAN RLE fixtures; C++ writes PLAIN null fixtures).
- The tree model and v3 files are not covered by the shared fixture flow; Java's
CompatibilityTestreads repo fixtures for v3 only.
Proposal
- Single source of truth: define the matrix in one checked-in spec (for example a small YAML/CSV file or a shared generated manifest) consumed by both the Java and C++ generators, so the two cannot drift.
- Cover the full supported cartesian space, skipping cases explicitly with a reason rather than silently omitting them: every (data type × encoding) pair supported by each side, every compression both can build (UNCOMPRESSED, SNAPPY, GZIP, LZ4, ZSTD, LZMA2 as available in CI), and both models (tree and table).
- Add boundary and scale coverage: 1 row; exactly one page; page + 1; the TS_2DIFF 129-value block boundary; multiple pages; multiple chunks; multiple chunk groups; all-null and mixed-null pages; long RLE runs; dictionary cardinality extremes; string edge cases (empty, embedded NUL, UTF-8); numeric extremes; and precision properties.
- Keep both directions (Java → C++, C++ → Java) and add Python as a reader where the binding supports it.
- Decide the policy for encodings one side cannot write: implement them, or record them as known gaps that fail deliberately.
- Make CI report the covered/skipped matrix so gaps are visible in every run.
Acceptance criteria
- Java and C++ generate an identical fixture set from one shared definition; language-specific fixtures can only exist when explicitly marked as such.
- Every (type, encoding, compression) pair supported by both sides is generated and validated in both directions.
- Tree-model and table-model files are both covered.
- Known unsupported combinations are listed explicitly and tracked instead of missing.
- CI fails when a writer cannot produce a declared fixture or a reader cannot read it.
Related
- [Test] Expand cross-language compatibility coverage across scales and nullable BOOLEAN columns #1021 — expand cross-language compatibility coverage across scales and nullable BOOLEAN columns
- [CPP] Support reading BOOLEAN RLE written with Java defaults #1019 — C++ support for reading BOOLEAN RLE written with Java defaults
Metadata
Metadata
Assignees
Labels
P0c++Pull requests that update c++ codePull requests that update c++ codefeatureNew featureNew featurejavaPull requests that update Java codePull requests that update Java codepythonPull requests that update Python codePull requests that update Python code
Activity