Repository navigation
Description
Summary
Applications that load training data issue one time-range query per window. On that access pattern the public table query API costs a fixed ~0.30 ms per call, almost independent of the number of rows returned: 64 windows (2048 points each) take 20.6–23.1 ms, while a single prepared-series query for the same window takes 0.046 ms.
prepare_series + query_prepared already avoid most of the per-call cost, but the prepared path is (a) only reachable through locators produced by the Python dataset index, and (b) has several defects that limit how much it helps. This issue records what we measured and the concrete fixes that would make prepared series the normal fast path.
Workload
- 323 regular series × 105,120 points, 5-minute interval, one table, one string TAG (
series_id). - Values: float32 PLAIN; time: TS_2DIFF; no compression; page size 10,000 points.
- One window = 2048 rows; 64 windows per batch; warm page cache; one CPU;
threads=1; medians over repeated runs. - AMD EPYC 9J14, Python 3.11.18, tsfile 2.5.0 wheel plus locally built
develop(identical CMake flags).
Measurements (ms per single-window call)
| API path | ms/window |
|---|---|
query_table, batch_size = window length |
0.32–0.36 |
query_prepared, value column, time range, limit = length |
0.046 |
query_prepared, value column, time range, no limit |
0.079 |
query_prepared, valid column only |
0.076 |
two single-column query_prepared calls (value + valid) |
0.121 |
query_prepared_multi, two columns, time range |
0.26–0.27 |
query_prepared_multi, two columns, offset/limit |
1.79 |
| prepared-handle acquire | 0.004 |
Per 64-window batch: 20.6–23.1 ms with query_table, ~4–5 ms with prepared queries.
The cost is not proportional to rows: a 0-row query still costs 0.325 ms, a 10,000-row query 0.55 ms, and a full 105,120-row query 0.92 ms (8.8 ns/row).
Where the time goes per query_table call
query_table() itself is lazy (~0.008 ms). The first read_arrow_batch() performs all of the work:
- The table-level device index leaf node is read and deserialized to apply the tag filter.
SingleDeviceTsBlockReader::init_internal()callsTsFileIOReader::get_timeseries_indexes(), which re-reads and deserializes the device metadata node viaload_device_index_entry()and does not consultdevice_node_cache_, even thoughalloc_multi_ssi()later uses the cached node. Measured: repeated queries on the same series (cache hot) cost 0.316 ms vs 0.440 ms when cycling 323 series — the cache only saves ~28%.- A new
TsFileSeriesScanIterator+ChunkReaderis constructed and destroyed per query. ChunkReader::get_next_page()walks page headers from the start of the chunk, readingDEFAULT_READ_SIZE = 4096bytes per header; a 2048-point window triggers ~14 such reads (~57 KiB) even when the filter covers only the first page. There is no early stop once the page range is past the filter.- The intersecting page is decoded in full (10,000 points for a 2048-point window).
- The
TsBlockis allocated per query withmax_row_count = batch_size(e.g. 65,536 → ~0.8 MB); this alone costs 0.079 vs 0.040 ms.
The profile of the per-query work is dominated by allocator traffic and metadata construction; actual decoding (multi_DECODE_TV_BATCH + TS_2DIFF + time filter) is only ~6.5%.
Prepared path defects
PreparedSeriesTsBlockReaderhard-codesPREPARED_BATCH_ROWS = 65536, so without an explicitlimitevery prepared query preallocates a 65,536-row block. Passinglimithalves the call cost (0.079 → 0.046 ms).- Every prepared call still constructs a new SSI/
ChunkReader;PreparedSeriesonly caches the resolved index. query_prepared_multi(..., offset, limit)degrades to 1.79 ms/call, ~40× a single-column prepared call and ~7× the two-column time-range variant.- Two single-column prepared calls (0.121 ms) are ~2.2× faster than one
query_prepared_multi(0.27 ms) for two columns. - The prepared locator is only produced by the Python dataset index (
TsFileDataFrame(use_index=True)), so users not using that layer cannot callprepare_seriesby (table, tag values, field).
Proposed changes
- Make
get_timeseries_indexes()reuse the parsed device node fromdevice_node_cache_, or drop the redundant call ininit_internalsincealloc_multi_ssi()already resolves the same node. - Stop scanning page headers once the filter range is exhausted, and/or use a page index to seek.
- Push the row offset down to page granularity so a 2048-point window does not decode a 10,000-point page.
- In the prepared path: honor the requested row count in
PREPARED_BATCH_ROWS, reuse the SSI/ChunkReaderacross calls on the same prepared series, and expose a publicprepare(table, tags, field)entry point. - Fix
query_prepared_multiwithoffset/limitand investigate why two columns are slower than two single-column calls.
Notes
- PR perf(cpp): speed up wide-table reads (RLE SIMD, batch index loading, BOUNDARY bulk) #1072 (wide-table reads) does not change this workload: with the same build configuration it executes the same number of instructions as
develop(8.824G vs 8.825G for 1500 queries), because this dataset uses PLAIN values (no RLE) and one measurement per device (no per-measurement index fan-out). - Separately, we measured current
developat ~8.8G instructions vs 7.2G for 2.5.0 on the same 1500-query loop (both locally built with identical CMake flags), i.e. ~21% more instructions per query. We have not bisected this yet; it may deserve its own issue. - The benchmark script and the dataset generator parameters can be shared if useful.
Activity
perf(python): reuse prepared series for repeated single-device table …
perf(python): reuse prepared series for repeated single-device table …