Repository navigation
Conversation
- add the missing Tablet.hashCode and make the extract-filter equals() type check explicit - guard the copyDirectory containment check with a path separator so a sibling directory is not mistaken for a child - widen size arithmetic to long in the encoders and TimeDuration so byte size estimates and month durations cannot overflow int - remove unused containers/variables flagged in tests and use pytest.importorskip for pyarrow
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Bitmap size estimation still risks integer overflow, and Tablet hashing can produce different hashes for equal tablets, breaking hash-based lookups.
2 open issues
What changed in this PR
This PR addresses CodeQL findings in TsFile’s Java implementation and Java/Python tests, focusing on equality contracts, directory-copy containment, and overflow-prone arithmetic.
Changes:
- Add tablet hashing, guard filter equality casts, refine directory containment, and clarify cached-
Longidentity checks. - Promote encoder-size and month-duration calculations to
long. - Remove unused test collections and simplify optional PyArrow imports.
| File | Description |
|---|---|
python/tests/test_batch_arrow.py |
Use pytest.importorskip for PyArrow. |
java/tsfile/src/test/java/org/apache/tsfile/tableview/TableViewTest.java |
Remove unused row collection while preserving iteration. |
java/tsfile/src/test/java/org/apache/tsfile/read/TsFileV4ReadWriteInterfacesTest.java |
Remove unused device-ID collection. |
java/tsfile/src/test/java/org/apache/tsfile/read/ReadInPartitionTest.java |
Remove unused time-range setup. |
java/tsfile/src/test/java/org/apache/tsfile/encoding/decoder/BitmapDecoderTest.java |
Remove unused boolean test data. |
java/tsfile/src/main/java/org/apache/tsfile/write/record/Tablet.java |
Add a partial-field hashCode implementation. |
java/tsfile/src/main/java/org/apache/tsfile/utils/TimeDuration.java |
Widen month-duration multiplication. |
java/tsfile/src/main/java/org/apache/tsfile/read/filter/operator/ExtractValueFilterOperators.java |
Guard equality casts while retaining exact-class checks. |
java/tsfile/src/main/java/org/apache/tsfile/external/commons/io/FileUtils.java |
Distinguish nested destinations from sibling paths. |
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/RegularDataEncoder.java |
Widen integer and long size estimates. |
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/LongZigzagEncoder.java |
Widen size-estimate multiplication. |
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/LongRleEncoder.java |
Widen group and value size calculations. |
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/IntZigzagEncoder.java |
Widen size-estimate multiplication. |
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/IntRleEncoder.java |
Widen group and value size calculations. |
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/DeltaBinaryEncoder.java |
Widen buffered-value size calculations. |
java/tsfile/src/main/java/org/apache/tsfile/encoding/encoder/BitmapEncoder.java |
Widen bitmap size-estimate multiplication. |
java/common/src/main/java/org/apache/tsfile/utils/RamUsageEstimator.java |
Extract and document cached-Long identity detection. |
Review effort: Balanced
Removed 'schemas' from hash calculation to maintain consistency with equals() method. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
- BitmapEncoder now evaluates `values.size() + 7` in long, so the addition itself cannot wrap before the byte-size estimate is widened. - VectorMeasurementSchema hashed its type/encoding/compressor arrays by identity while equals() compares them by content; hash the contents so equal schemas share a hash. - Tablet.hashCode keeps its partial-field contract and no longer hashes `schemas`, whose pre-fix hash was the only inconsistent input.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
The daily
CodeQL Qualitysweep reports 1,313 alerts on our own code. This PR clears every error-severity finding and thejava/integer-multiplication-cast-to-longwarnings in main code.Error findings (9):
Tabletoverridesequals()withouthashCode(); add a hash over the fields compared before the deep value checks.ExtractValueFilterOperators.ExtractValueCompareFilter.equals()now guards the cast withinstanceof(still requires the exact class, preserving current semantics).FileUtils.copyDirectory()used a barestartsWith(src)containment test, so a sibling such as/a/bccounted as a child of/a/b; the guard now ends at a path separator.ReadInPartitionTest,BitmapDecoderTest,TableViewTest,TsFileV4ReadWriteInterfacesTest).test_batch_arrow.pyno longer conditionally bindspainside atry; it usespytest.importorskip("pyarrow").Integer-overflow warnings (16): byte-size estimates and month durations now perform the multiplication in
longinLongRleEncoder,IntRleEncoder,LongZigzagEncoder,IntZigzagEncoder,BitmapEncoder,DeltaBinaryEncoder,RegularDataEncoderandTimeDuration. Behavior is unchanged for realistic sizes; the estimates can no longer wrap for very large inputs.The remaining sweep is dominated by notes (empty blocks, commented-out code, missing
overrideannotations) plus intentional-lifetime C++ warnings such ascpp/stack-address-escape; those need separate design decisions rather than one-line fixes.Test plan
./mvnw -B clean verify -P with-java,with-python -Denable.lzma2=ON(Java, C++, Python; 1853 C++ tests and 667 Python tests passed)./mvnw spotless:check -P with-java,with-python