Skip to content

Add a selective metadata reader for a set of measurements - #1123

Merged
ColinLeeo merged 8 commits into
apache:developfrom
ColinLeeo:colin/measurement-selective-metadata
Nov 20, 2026
Merged

ColinLeeo merged 8 commits into
apache:developfrom
ColinLeeo:colin/measurement-selective-metadata

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Reading the TimeseriesMetadata of N measurements of one device currently walks the device metadata index and deserializes the containing leaf group once per measurement. A wide aligned projection therefore pays one index walk per column, and an absent measurement that a bloom filter cannot rule out re-reads the same leaf group.

This PR adds a set-based metadata reader to TsFileSequenceReader that resolves a whole measurement set against the leaf groups that hold it.

What changed

  • TsFileSequenceReader.readTimeseriesMetadataForMeasurements(device, offset, measurements, returnedMeasurements, mayExist, ignoreNotExistDevice, ioSizeRecorder): locates the device's measurement metadata index node (or reads the supplied offset for external files), routes each name to its floor child with a binary search over the node's children, and reads each visited internal node and each visited leaf group once. It returns the metadata of returnedMeasurements found in the visited groups.
    • There is no global sort. On a leaf node the walk stops once every group is selected.
    • mayExist is optional. A name is tested before it selects a group, so a group that only absent names point to is not read; a dense request needs only a few probes. The predicate must have no false negatives (a BloomFilter qualifies) and receives the bare measurement name, so a BloomFilter lookup must build the full device.measurement path. Offset-addressed external files should pass none, because their BloomFilter may not cover the series.
    • returnedMeasurements may be a superset of measurements, which lets a single-measurement call warm its group siblings. The absence guarantee applies only to measurements that are both searched and requested for return; a missing warmed sibling is inconclusive because its own group may not have been visited.
    • The device is looked up even for an empty request, so an absent device is reported the same way as for a non-empty one.
  • The result map is sized from the searched set, so warming a single series does not allocate for every sibling.
  • For a streaming group above 2 GiB, only the bytes actually read are reported; the chunk lists the deserializer seeks past are not counted as I/O.
  • Convenience overloads without the offset, without returnedMeasurements, and without mayExist.
  • The set-based reader has a distinct name, so a literal null offset is no longer ambiguous with the single-measurement overloads.
  • The non-predicate TsFileInput deserializer now delegates to the predicate one instead of duplicating its body.
  • The batch index search added earlier in this PR (getChildIndexEntries and the two strategies) is removed; the traversal no longer needs it.

Example

A file whose device has 700 series spread over 3 measurement leaf groups; read all 700 measurements, 680 of which are absent from the file:

  • Before: one readTimeseriesMetadata call per measurement; each absent name re-reads and re-deserializes its floor leaf.
  • After: one call; the 3 leaf groups are read once each. With a bloom-backed mayExist, a group that only absent names point to is not read at all.

In the IoTDB query benchmark that motivated this change (700-column device, 20 columns per file, 12 files, cold metadata cache), the batch read brought a sparse 20-of-700 projection from 8.0 ms to 2.0–2.6 ms per query in the earlier revision; the floor-routing revision removes the request sort that dominated the per-call cost.

Tests

  • TimeSeriesMetadataReadTest:
    • batch results match per-measurement reads, including each chunk's header offset, count, start/end time and numeric min/max;
    • absent measurement and absent device, and the offset-addressed variant;
    • sibling warming through returnedMeasurements;
    • an internal measurement node with an exact read count (11 reads for 20 series at degree 3: root, 3 leaf index nodes, 7 leaf groups);
    • what an absent name costs: m00 alone reads 3 nodes/groups, m00 plus the absent m10x reads 5, and with mayExist rejecting m10x it is back to 3;
    • an aligned fixture that includes the time column and a table-model fixture.
  • TimeseriesMetadataTest.testStreamingDeserializerSkipsRejectedChunkList: two chunk-bearing records; the rejected record's chunk list is not decoded and the input lands exactly on the next record, whose chunk metadata then matches.
  • MetadataIndexNodeTest: the single-key floor search.

Compatibility

Existing overloads are unchanged. The traversal follows the approach of #550.

🤖 Generated with Claude Code

@ColinLeeo
ColinLeeo requested a balanced review from Copilot November 17, 2026 08:52
ColinLeeo added a commit to ColinLeeo/iotdb that referenced this pull request Nov 17, 2026
The set-based readTimeseriesMetadata overload is not in the dated tsfile
snapshot the build used before. Point the dependency at the 2.5.1
snapshot, which will contain it once apache/tsfile#1123 is merged and
published.

The IoTDB change that uses the new API must not be pushed until that
snapshot exists in the configured repositories.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Batch lookups mishandle duplicate keys and mixed device ID types, while filtering decodes unnecessary chunk metadata and tests lack valid ordering and internal-node coverage.

5 open issues
What changed in this PR

Adds batch metadata lookup to the Java reader, reducing repeated index traversal and leaf-group reads when selecting multiple measurements from one device.

Changes:

  • Adds batch child-index searches using binary or merge strategies.
  • Adds device- and offset-based reader overloads, including optional sibling metadata retrieval.
  • Adds tests for batch results, missing measurements and devices, offsets, and sibling retrieval.
File Description
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​read/​TimeSeriesMetadataReadTest.java Tests batch reader results and API variants.
java/​tsfile/​src/​test/​java/​org/​apache/​tsfile/​file/​metadata/​MetadataIndexNodeTest.java Tests batch child lookup and boundary cases.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​read/​TsFileSequenceReader.java Adds grouped metadata traversal and reader overloads.
java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​file/​metadata/​MetadataIndexNode.java Adds batch searches with shared child-entry results.

Review effort: Balanced

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for moving the selective reader into TsFile. I checked out cb46a2d and ran MetadataIndexNodeTest and TimeSeriesMetadataReadTest locally (6 cases pass). Since this API exists for performance, I also wrote throwaway differential tests and a reader-level benchmark (not pushed).

Correctness: no wrong result found.

  • Batch index search: 600,000 random node/key sets (duplicate keys, empty nodes, exact and floor mode), both strategies against a brute-force floor. No mismatch.
  • Batch read against the single-measurement reader, name by name (id, type, statistics, and the offset and statistics of every chunk): 467,805 names on tree-model files (aligned and non-aligned devices, index degree 2/3/4/7/256, multi-level measurement index, absent names and devices, the offset-addressed variant) and 153,615 names on table-model files (null tags, all-null columns). No mismatch.

Performance: one call per (device, file), TsFile API only. Real files written with TsFileWriter: one aligned device with 700 INT64 columns (three measurement leaf groups), 3 chunks per series, and a sparse file in which the same device has 20 of the 700 columns. The harness counts positional reads and bytes at the TsFileInput and reports the median of 5 timed rounds after warm-up (JDK 21, files in the page cache; rounds are mostly within ±3% of the median).

Scenario master flow this PR prototype without the sort
Dense device, select all 700 columns 100.6 µs, 6 reads / 179 KB 152.6 µs, 4 reads / 179 KB 93.0 µs
Dense device, first 60% of the columns 70.0 µs 91.2 µs 60.7 µs
Sparse file (20 of 700 columns), select all 700 56.1 µs, 2 reads / 5 KB 55.0 µs, 2 reads / 5 KB 3.5 µs
Same, in a file with 1,000 other devices (19 BloomFilter false positives) 533 µs, 60 reads / 225 KB 81.7 µs, 3 reads / 11 KB 23.1 µs
Dense device, 5 columns 29.1 µs 19.2 µs 16.3 µs
2 present columns + 18 absent names that sort into the other two leaf groups 29.9 µs, 2 reads / 65 KB 60.7 µs, 4 reads / 179 KB 17.3 µs, 2 reads / 65 KB [a]
Warm query: 100 never-written columns still unresolved 8.4 µs, no read 65.4 µs, 4 reads / 179 KB 9.5 µs, no leaf-group read [a]
20-column device, all present 3.7 µs 4.5 µs 3.4 µs
Offset-addressed read (external TsFile), select all 700 columns 18,643 µs, 1,402 reads / 42 MB 149.6 µs, 4 reads / 179 KB n/a
  • "master flow" is what IoTDB master's TimeSeriesMetadataCache.get() does per column on the default path: cache probe, per-measurement BloomFilter check, then readTimeseriesMetadata(device, measurement, allSensors, ...). It already reads each leaf group once.
  • The last row is master's externalTsFile / cache-disabled branch, which calls the single-measurement reader once per column. This is the "Before" of the description.
  • [a] with the existence predicate proposed in the inline comment on the Javadoc.

What this shows:

  • The reader is a large win where master really reads per column: 80–160x for offset-addressed reads (1, 3 and 10 chunks per series), and 6.5x with 20x fewer bytes when the BloomFilter has false positives. Narrow projections also gain, because chunk metadata is decoded only for the returned series.
  • On the default path it is not faster than what master does today. The sparse case that motivated it is level (55.0 vs 56.1 µs, same bytes), so the gain measured in apache/iotdb#19198 comes from the caller making one call per file instead of a cache/lock/BloomFilter cycle per column. Dense requests are slower by a constant of about 51 µs per call: 117.3 vs 65.7 µs with 1 chunk per series, 152.6 vs 100.6 µs with 3, 243 vs 192 µs with 10.

Requesting changes for the two P1s. Each point has an inline comment:

  1. [P1] Sorting the request accounts for 85–90% of that constant. Bucketing names by child removes it and makes the reader faster than the master flow in every dense case I measured.
  2. [P1] An absent name makes the reader load the leaf group it sorts into, and the API gives callers no cheap way to prevent that. The Javadoc and the test claim more than the reader guarantees.
  3. [P2] The new overloads make a literal null argument ambiguous.
  4. [P2] Test gaps: weak read-count assertions, no aligned or table-model device.
  5. [P2] TsFile's own query path does not use the new reader, although it has the per-absent-name re-read.
  6. [P3] Semantics and documentation: floor semantics differ from getChildIndexEntry, an empty request skips the device check, @return wording.
  7. [P3] Small code points: double contains, duplicated deserializer, an unreachable branch.

Not covered: the branch for leaf groups larger than 2 GB was only read, not run. The numbers above isolate the reader; end-to-end numbers are in apache/iotdb#19198. I can share the harness and the prototype if that helps.

MetadataIndexNode.getChildIndexEntries() searches the floor child entry
of a sorted key set in one pass, choosing between a monotonic binary
search and a merge scan and reusing one pair instance per child. The new
TsFileSequenceReader.readTimeseriesMetadata(device, offset,
measurements, returnedMeasurements, ...) uses it to visit only the leaf
groups that hold a searched measurement, reads each visited group once,
and returns the metadata of the returned measurements.

A caller can resolve a whole projection without one index walk and one
leaf deserialization per column, and a single-measurement call can warm
its group siblings through returnedMeasurements.

Follows the batch index-search approach of apache#550.
Compare the index child to the key in the merge search, like the binary
searches do, so a plain and a string-array device id can be compared
without throwing. Keep the matched child eligible for duplicate keys in
both batch strategies.

Decode the chunk metadata of a leaf group only for the measurements the
call returns, using the existing filter-aware TimeseriesMetadata
deserializer (and a stream counterpart for groups larger than one
buffer), so a sparse request does not build chunk metadata for every
series it walks past.

Cover the internal measurement node with a 20-measurement fixture, assert
that absent names add no reads, and make the index-node tests use
lexicographically ordered children plus duplicate and mixed-device-id
cases.
Replace the global sort in the set-based reader with per-node floor
routing: each name is bucketed by the child it sorts into, and the leaf
level stops once every group is selected. Remove the now-unused batch
index search.

Let callers pass a mayExist predicate so a group that only absent names
point to is not read, and document what a missing name means with and
without the predicate.

Rename the set-based reader to readTimeseriesMetadataForMeasurements so
a literal null offset is no longer ambiguous with the single-measurement
overloads. The non-predicate TsFileInput deserializer delegates to the
predicate one instead of duplicating its body.

Strengthen the tests: assert the exact node/group read count, add
aligned (with the time column) and table-model fixtures, and cover what
an absent name costs with and without the predicate.
@ColinLeeo
ColinLeeo force-pushed the colin/measurement-selective-metadata branch from 919f314 to c99f54f Compare November 19, 2026 04:28
ColinLeeo added a commit to ColinLeeo/iotdb that referenced this pull request Nov 19, 2026
The set-based readTimeseriesMetadata overload is not in the dated tsfile
snapshot the build used before. Point the dependency at the 2.5.1
snapshot, which will contain it once apache/tsfile#1123 is merged and
published.

The IoTDB change that uses the new API must not be pushed until that
snapshot exists in the configured repositories.
@ColinLeeo
ColinLeeo requested a max review from Copilot November 19, 2026 08:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- Size the result map from the searched set, not the return set: the
  return set may include many cache-warming siblings that live in groups
  this call does not visit.
- For a streaming group above 2 GiB, record only the bytes actually read,
  excluding the chunk lists the selective deserializer seeks past.
- Document that mayExist receives the bare measurement name, so a
  BloomFilter lookup has to build the full device.measurement path, and
  scope the absence guarantee to searched measurements: a missing warmed
  sibling is inconclusive.
- Test the streaming predicate path on two chunk-bearing records: the
  rejected record's chunk list is skipped, the input lands on the next
  record, and the accepted record's chunk metadata matches.
- Compare each chunk's offset, count, time bounds and numeric value
  statistics in the batch-vs-single assertion.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Java’s UTF-16 ordering can misroute UTF-8-ordered C++ indexes and silently omit metadata; reader-level accounting and leaf-filtering coverage is also missing.

1 open issue
2 previously missed issues

In code that hasn't changed since last review

High severity Handle C++ UTF-8 ordering when routing metadata lookups

java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​read/​TsFileSequenceReader.java:1328

C++ orders measurement names by UTF-8 bytes (cpp/src/common/allocator/my_string.h:181), whereas this comparison uses Java's UTF-16 order. For a C++-written group beginning with \uF900 and containing \uD840\uDC00, searching for the latter returns -1 and silently omits existing metadata. Route conservatively for both Java- and C++-ordered indexes, and add a C++-to-Java regression with these names.

Medium severity Test that rejected leaf groups do not trigger reads

java/​tsfile/​src/​main/​java/​org/​apache/​tsfile/​read/​TsFileSequenceReader.java:1272

The predicate test uses m00 and m10x, which route to different internal children, so m10x is rejected before this leaf-level check is reached. Add a case such as m00 plus m03x: they share an internal child but select different leaf groups, so asserting the filtered read count verifies that a rejected name does not trigger a leaf-group read.

6 resolved since last review

Review effort: Balanced

- Java indexes are sorted by UTF-16 code units while C++ indexes use
  UTF-8 bytes. The two disagree for supplementary characters mixed with
  the U+E000..U+FFFF range, so a binary search on a C++ index could miss
  an existing group. Compute the floor child under both orderings and
  visit both candidates when they differ.
- Extract the streaming leaf-group loop so a reader-level test can drive
  it with a fake input: one accepted and one rejected chunk-bearing
  record, asserting the recorded bytes are the full record for the
  accepted one and the record minus its chunk list for the rejected one.
- Add m00 + m03x to the leaf-filter test: they share an internal child
  but select different leaf groups, so the rejected-name read count now
  verifies the leaf-level filter.

@JackieTien97 JackieTien97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rework. I went through the 12 points of my previous review again on 20e89a4 and re-ran the differential tests and the reader-level benchmark from last time. Both P1s are fixed, and I found no wrong result.

Previous points. 11 of the 12 are addressed in the code. The remaining one (TsFile's own query path using the new reader) is the follow-up we agreed on.

Verification on 20e89a4

  • The PR's 10 test cases pass locally (MetadataIndexNodeTest, TimeseriesMetadataTest, TimeSeriesMetadataReadTest).
  • Differential runs against the single-measurement reader, name by name: 307,695 names on tree-model files and 154,448 names on table-model files, without a predicate and with the file's BloomFilter as mayExist. No mismatch.
  • The call shapes with a literal null that were ambiguous at cb46a2d now compile.
  • The branch for leaf groups above 2 GiB, which I had only read last time: on a file with one 2.28 GB leaf group the results equal the single-measurement reader's, and the bytes handed to ioSizeRecorder equal the bytes really read (1,290,008,853; the 990 MB of chunk lists of the series that are not returned are skipped).

Performance. Same files and method as last time (one aligned device with 700 columns, 3 chunks per series; median of 5 timed rounds, in µs). Each cell is the set-based reader against the master flow measured in the same run. The old head was re-measured today.

Scenario cb46a2d 20e89a4
Dense device, select all 700 columns 172.3 vs 116.8 (48% slower) 100.7 vs 111.9 (10% faster)
Dense device, first 60% of the columns 91.9 vs 79.5 (16% slower) 71.3 vs 76.9 (7% faster)
Sparse file (20 of 700 columns), select all 700 55.8 vs 58.4 3.7 vs 58.2
2 present columns + 18 absent names in other leaf groups 4 reads / 179 KB, 52.3 vs 32.3 with mayExist: 2 reads / 65 KB, 18.2 vs 32.5
Warm query, 100 never-written columns 4 reads / 179 KB, 69.2 vs 7.9 with mayExist: no leaf-group read, 10.7 vs 7.9

The dense select-all is also level or faster in the other three configurations (1 and 10 chunks per series, 1,000 other devices in the file): 13%, 1% and 27% faster than the master flow. A few cells had one outlier round out of five; the medians are not affected.

One non-blocking suggestion is inline. Three notes that need no change in this PR:

  • apache/iotdb#19198 at 7041acb calls the renamed reader, but it still probes the BloomFilter for every pending name before the call. For the dense select-all that is 228.3 µs, against 111.9 µs for the master flow and 98.6 µs when the filter is passed as mayExist. The caller should switch to mayExist.
  • The pre-existing single-measurement reader readTimeseriesMetadata(device, measurement, ...) routes with UTF-16 order only: on the C++-ordered file described in the inline comment it finds 1 of 300 series. This PR does not touch it; it deserves a separate issue.
  • The PR description does not mention the changes of 20e89a4 yet (routing under both orderings and the two new tests).

Not covered: the numbers isolate the reader, I did not re-measure IoTDB end to end.

The two index orderings can disagree only where a surrogate code unit is
compared with a unit in U+E000..U+FFFF. A name whose units are all below
U+D800 (ASCII, Latin, common CJK) has the same relation to every child
under both orders, so one binary search yields the same floor. Only names
with a unit at or above U+D800 pay for the code-point search.
Always compute the floor under both index orderings instead of relying on
the code-unit threshold; the check couples the routing decision to a
property of the name encoding that we do not want to depend on.
@ColinLeeo
ColinLeeo merged commit 0d89263 into apache:develop Nov 20, 2026
20 checks passed
ColinLeeo added a commit to ColinLeeo/iotdb that referenced this pull request Nov 20, 2026
apache/tsfile#1123 is merged on develop (squash 0d89263e); the dated
snapshot 2.5.1-261120-SNAPSHOT is built from that commit and published to
Apache snapshots.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants