Skip to content

fix: address CodeQL error findings and size overflow fallouts - #1134

Open
ColinLeeo wants to merge 3 commits into
apache:developfrom
ColinLeeo:colin/codeql-quality-errors
Open

ColinLeeo wants to merge 3 commits into
apache:developfrom
ColinLeeo:colin/codeql-quality-errors

Conversation

@ColinLeeo

Copy link
Copy Markdown
Contributor

Summary

The daily CodeQL Quality sweep reports 1,313 alerts on our own code. This PR clears every error-severity finding and the java/integer-multiplication-cast-to-long warnings in main code.

Error findings (9):

  • Tablet overrides equals() without hashCode(); add a hash over the fields compared before the deep value checks.
  • ExtractValueFilterOperators.ExtractValueCompareFilter.equals() now guards the cast with instanceof (still requires the exact class, preserving current semantics).
  • FileUtils.copyDirectory() used a bare startsWith(src) containment test, so a sibling such as /a/bc counted as a child of /a/b; the guard now ends at a path separator.
  • Removed four containers whose contents were never read in tests (ReadInPartitionTest, BitmapDecoderTest, TableViewTest, TsFileV4ReadWriteInterfacesTest).
  • test_batch_arrow.py no longer conditionally binds pa inside a try; it uses pytest.importorskip("pyarrow").

Integer-overflow warnings (16): byte-size estimates and month durations now perform the multiplication in long in LongRleEncoder, IntRleEncoder, LongZigzagEncoder, IntZigzagEncoder, BitmapEncoder, DeltaBinaryEncoder, RegularDataEncoder and TimeDuration. Behavior is unchanged for realistic sizes; the estimates can no longer wrap for very large inputs.

The remaining sweep is dominated by notes (empty blocks, commented-out code, missing override annotations) plus intentional-lifetime C++ warnings such as cpp/stack-address-escape; those need separate design decisions rather than one-line fixes.

Test plan

  • ./mvnw -B clean verify -P with-java,with-python -Denable.lzma2=ON (Java, C++, Python; 1853 C++ tests and 667 Python tests passed)
  • ./mvnw spotless:check -P with-java,with-python

- add the missing Tablet.hashCode and make the extract-filter equals()
  type check explicit
- guard the copyDirectory containment check with a path separator so a
  sibling directory is not mistaken for a child
- widen size arithmetic to long in the encoders and TimeDuration so byte
  size estimates and month durations cannot overflow int
- remove unused containers/variables flagged in tests and use
  pytest.importorskip for pyarrow

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Bitmap size estimation still risks integer overflow, and Tablet hashing can produce different hashes for equal tablets, breaking hash-based lookups.

2 open issues
What changed in this PR

This PR addresses CodeQL findings in TsFile’s Java implementation and Java/Python tests, focusing on equality contracts, directory-copy containment, and overflow-prone arithmetic.

Changes:

  • Add tablet hashing, guard filter equality casts, refine directory containment, and clarify cached-Long identity checks.
  • Promote encoder-size and month-duration calculations to long.
  • Remove unused test collections and simplify optional PyArrow imports.
File Description
python/​tests/​test_batch_arrow.py Use pytest.importorskip for PyArrow.
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​tableview/​TableViewTest.java Remove unused row collection while preserving iteration.
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​read/​TsFileV4ReadWriteInterfacesTest.java Remove unused device-ID collection.
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​read/​ReadInPartitionTest.java Remove unused time-range setup.
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​encoding/​decoder/​BitmapDecoderTest.java Remove unused boolean test data.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​write/​record/​Tablet.java Add a partial-field hashCode implementation.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​utils/​TimeDuration.java Widen month-duration multiplication.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​read/​filter/​operator/​ExtractValueFilterOperators.java Guard equality casts while retaining exact-class checks.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​external/​commons/​io/​FileUtils.java Distinguish nested destinations from sibling paths.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​encoding/​encoder/​RegularDataEncoder.java Widen integer and long size estimates.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​encoding/​encoder/​LongZigzagEncoder.java Widen size-estimate multiplication.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​encoding/​encoder/​LongRleEncoder.java Widen group and value size calculations.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​encoding/​encoder/​IntZigzagEncoder.java Widen size-estimate multiplication.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​encoding/​encoder/​IntRleEncoder.java Widen group and value size calculations.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​encoding/​encoder/​DeltaBinaryEncoder.java Widen buffered-value size calculations.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​encoding/​encoder/​BitmapEncoder.java Widen bitmap size-estimate multiplication.
java/​common/​src/​main/​java/​org/​apache/​tsfile/​utils/​RamUsageEstimator.java Extract and document cached-Long identity detection.

Review effort: Balanced

ColinLeeo and others added 2 commits November 19, 2026 17:21
Removed 'schemas' from hash calculation to maintain consistency with equals() method.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- BitmapEncoder now evaluates `values.size() + 7` in long, so the addition
  itself cannot wrap before the byte-size estimate is widened.
- VectorMeasurementSchema hashed its type/encoding/compressor arrays by
  identity while equals() compares them by content; hash the contents so
  equal schemas share a hash.
- Tablet.hashCode keeps its partial-field contract and no longer hashes
  `schemas`, whose pre-fix hash was the only inconsistent input.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

There are no active review issues for the CodeQL-related changes.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants