Skip to content

[Feature][Test] Expand the Java–C++ cross-language compatibility matrix to full coverage #1057

Closed

Description

@ColinLeeo

Motivation

Cross-language compatibility is currently tested with a table-model fixture matrix that covers only part of what each implementation can write. The gaps let encoding and compression bugs reach releases and push maintainers into ad-hoc manual testing.

Current state (verified)

  • Java: java/tsfile/src/test/java/org/apache/tsfile/compatibility/TableModelEncodingCompressionCompatibilityTest.java
  • C++: cpp/test/reader/table_view/table_model_encoding_compression_compatibility_test.cc
  • Orchestration: .github/workflows/compatibility-test.yml — each language generates fixtures, and the other language validates them (both directions).

Coverage today:

  • Table model only.
  • Data types: INT32, DATE, INT64, TIMESTAMP, FLOAT, DOUBLE, plus BOOLEAN on the Java side.
  • Encodings: CHIMP and RLBE for the six numeric types; TS_2DIFF for FLOAT/DOUBLE (plus an explicit max_point_number=0 variant); CAMEL for DOUBLE; PLAIN for null-pattern fixtures; Java-only BOOLEAN RLE default fixtures.
  • C++ ships encoders for PLAIN, DICTIONARY, RLE, TS_2DIFF, GORILLA, ZIGZAG, CHIMP, SPRINTZ, RLBE, and CAMEL; Java supports the same set plus deprecated DIFF/REGULAR/BITMAP/FREQ. DICTIONARY for STRING/TEXT, GORILLA, SPRINTZ, ZIGZAG, RLE for integer types, and the STRING/TEXT/BLOB data types are not cross-validated.
  • The Java and C++ matrices are maintained separately and already differ (Java-only BOOLEAN RLE fixtures; C++ writes PLAIN null fixtures).
  • The tree model and v3 files are not covered by the shared fixture flow; Java's CompatibilityTest reads repo fixtures for v3 only.

Proposal

  1. Single source of truth: define the matrix in one checked-in spec (for example a small YAML/CSV file or a shared generated manifest) consumed by both the Java and C++ generators, so the two cannot drift.
  2. Cover the full supported cartesian space, skipping cases explicitly with a reason rather than silently omitting them: every (data type × encoding) pair supported by each side, every compression both can build (UNCOMPRESSED, SNAPPY, GZIP, LZ4, ZSTD, LZMA2 as available in CI), and both models (tree and table).
  3. Add boundary and scale coverage: 1 row; exactly one page; page + 1; the TS_2DIFF 129-value block boundary; multiple pages; multiple chunks; multiple chunk groups; all-null and mixed-null pages; long RLE runs; dictionary cardinality extremes; string edge cases (empty, embedded NUL, UTF-8); numeric extremes; and precision properties.
  4. Keep both directions (Java → C++, C++ → Java) and add Python as a reader where the binding supports it.
  5. Decide the policy for encodings one side cannot write: implement them, or record them as known gaps that fail deliberately.
  6. Make CI report the covered/skipped matrix so gaps are visible in every run.

Acceptance criteria

  • Java and C++ generate an identical fixture set from one shared definition; language-specific fixtures can only exist when explicitly marked as such.
  • Every (type, encoding, compression) pair supported by both sides is generated and validated in both directions.
  • Tree-model and table-model files are both covered.
  • Known unsupported combinations are listed explicitly and tracked instead of missing.
  • CI fails when a writer cannot produce a declared fixture or a reader cannot read it.

Related

Activity

  1. added this to the 2.5.1 milestone on Nov 2, 2026
  2. added
    javaPull requests that update Java code
    pythonPull requests that update Python code
    c++Pull requests that update c++ code
    on Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0c++Pull requests that update c++ codefeatureNew featurejavaPull requests that update Java codepythonPull requests that update Python code

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions