Skip to content

Batch scan TimeSeriesMetadata for wide-table queries - #7

Open
ColinLeeo wants to merge 1 commit into
pr-19198-basefrom
pr-19198-clean
Open

ColinLeeo wants to merge 1 commit into
pr-19198-basefrom
pr-19198-clean

Conversation

@ColinLeeo

Copy link
Copy Markdown
Owner

Description

A device's TimeseriesMetadata is stored in measurement leaf groups (up to max_degree_of_index_node series per group). Loading it column by column walks the device metadata index and deserializes the containing leaf group once per requested column, so a scan that selects almost all columns pays that traversal once per column, and every column absent from a file costs its own cache probe and BloomFilter check. In the motivating workload (one device with ~700 columns, ~20 written per day, 12 files) loading the metadata of one device across its files dominated the query.

The TsFile side of this change is apache/tsfile#1123, now merged on develop as 0d89263e; this PR pins the dated 2.5.1-261120-SNAPSHOT cut from that commit.

This is a clean-diff copy of apache/iotdb#19198, based on master at c594a9adf (branch pr-19198-base), so the PR shows only the net change.

What changed

TsFile side (apache/tsfile#1123)

  • TsFileSequenceReader.readTimeseriesMetadataForMeasurements(device, offset, measurements, returnedMeasurements, mayExist, ignoreNotExistDevice, ioSizeRecorder) locates the device's measurement metadata index node (or reads the supplied offset for external TsFiles), routes each name to its floor child with a binary search over the node's children, and reads each visited internal node and each visited leaf group once. Chunk metadata is decoded only for the measurements the call returns. returnedMeasurements may be a superset of measurements, which lets a single-column call warm its group siblings.
  • mayExist is optional: a name is tested before it selects a group, so a group that only absent names point to is not read. It must have no false negatives; a BloomFilter qualifies.
  • There is no global sort; the batch index search used by the earlier revision of this PR is gone.

DataNode

TimeSeriesMetadataCache.getDeviceTimeseriesMetadata is now:

  1. per-measurement cache pass;
  2. device-absence check on the time column via the BloomFilter;
  3. per-miss BloomFilter probe: a column the filter rules out is answered with null and not remembered, because probing the filter again is cheaper than one cache entry per absent column per file;
  4. the surviving columns go to the reader in one call under the per-(device, file) lock;
  5. every surviving column the reader does not return, or returns with count == 0, is answered with null and remembered with a negative cache entry, so a later query does not walk its leaf group again.

Offset-addressed external TsFiles skip BloomFilter filtering, because their BloomFilter may not cover the series. get() remains the single-column entry point used by tree-model scans. (Earlier revisions of this PR used a device-wide read gated by a 0.6 projection ratio; the selective reader replaces that design.)

tsfile.version moves to 2.5.1-261120-SNAPSHOT, cut from the merged develop (apache/tsfile branch snapshot/2.5.1-261120, squash commit 0d89263e) and published to the Apache snapshots repository. It must be replaced by a released TsFile version before merge.

Example

Table with 700 field columns; each of 12 TsFiles holds 20 of them (rotating); SELECT * for one device:

  • Before: 701 per-column cache lookups per file; every absent column took the per-device lock, a second cache check and a BloomFilter probe; the columns that got through the filter paid a leaf-group read and were remembered, the rest were re-probed on every query. 8,412 lookups and about 8,200 lock/probe cycles per query, and 420 leaf-group reads per cold query (35 per file).
  • After: one cache pass and one lock per file; about 700 BloomFilter probes per cold file (one per cache-missed column), one batch read per file (a single leaf group in this fixture), and about 34 negative entries per file (the columns that pass the filter). Warm queries do no leaf-group read; the BloomFilter-excluded columns are probed again.

The per-column probe is the price of keeping negative entries bounded by the BloomFilter false-positive rate instead of one entry per absent column per file. A dense 700-column device adds the same probe per column but keeps the reads and bytes unchanged.

Benchmark

Measured on the latest revision with the same machine and TsFile snapshot for both variants, cold metadata cache unless noted; bytes and read counts come from QueryStatistics. The base column was measured on master with the same harness. Each figure is the median of five timed rounds after warm-up; the dense cold case varies by up to about 20% between runs, so the table reports ranges.

Scenario base (master) this PR
700 columns, select all, cold / warm 12.2–12.4 / 1.35 ms 15.4–17.8 / 1.4–1.7 ms
700 columns, first 60%, cold / warm 2.9–4.1 / 0.6 ms 5.1–5.8 / 0.5–0.8 ms
Sparse: 20 of 700 columns per file, cold / warm 7.7–7.8 / 5.0–5.3 ms 2.5–3.0 / 1.5–2.0 ms
Repeated select-all, working set larger than the 61 MB cache (steady state) 6.0–13.0 ms (median 4.7–5.3) 6.3–6.4 ms (median 4.2–4.4)
Bytes per query (select-all / 60% / sparse) 2021.0 / 1475.6 / 64.1 KB 2019.9 / 1475.1 / 59.2 KB

The sparse wide-table case that motivated the change is the main gain: cold 7.7–7.8 ms to 2.5–3.0 ms, warm 5.0–5.3 ms to 1.5–2.0 ms. The dense cold case varies by 15–20% between rounds on this machine (repeated runs of the previous revision show the same range, including a 30 ms outlier), so the table reports the observed range rather than a single sample; the bytes and read counts are identical across runs. The latest revision checks the aligned time column before the per-column cache pass and filters BloomFilter-excluded measurements before taking the device lock, so a warm sparse query no longer enters the locked recheck for columns that are absent from the file. Sparse cold still pays one BloomFilter probe per cache-missed column, which is what keeps negative entries bounded by the false-positive rate instead of one per absent column. The single-series get() fast path answers a cache hit on the first LRU lookup, like master.

Merge checklist

  • apache/tsfile#1123 is merged on develop (squash commit 0d89263e); the dated snapshot is cut from that commit.
  • Replace the dated 2.5.1-261120-SNAPSHOT in pom.xml with a released TsFile version before merge.

Tests

  • TsFile (apache/tsfile#1123): TimeSeriesMetadataReadTest covers batch vs per-measurement reads, absent names and devices, the offset-addressed variant, sibling warming, exact node/group read counts on an internal measurement index, aligned (time column) and table-model fixtures, and what an absent name costs with and without mayExist.
  • DataNode: TimeSeriesMetadataCacheTest adds "reader-confirmed absence is remembered, so the second query does not call the reader" and "a BloomFilter-excluded column is probed again instead of being remembered". It also covers cache hits, placeholders, BloomFilter exclusion and external TsFiles. AlignedSeriesScanBatchTest (dense, sparse and all-null projections end to end; the dense case asserts one reader call per file through a spied reader) and ReadPointAlignedCrossCompactionTest cover the device-absent path.

🤖 Generated with Claude Code

Resolve an aligned device's metadata with one set-based reader call per
(device, file) instead of one call per column. The cache checks the time
column first, filters absent series with the BloomFilter before taking the
device lock, and remembers read-confirmed absence. The single-column entry
keeps its own path with sibling warming.

TsFile side: apache/tsfile#1123 (merged as 0d89263e), consumed through the
dated 2.5.1-261120-SNAPSHOT.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant