Skip to content

[Perf][C++] Point queries rebuild index metadata on every call; prepared series queries are ~7x faster but need fixes #1080

Open

Description

@ColinLeeo

Summary

Applications that load training data issue one time-range query per window. On that access pattern the public table query API costs a fixed ~0.30 ms per call, almost independent of the number of rows returned: 64 windows (2048 points each) take 20.6–23.1 ms, while a single prepared-series query for the same window takes 0.046 ms.

prepare_series + query_prepared already avoid most of the per-call cost, but the prepared path is (a) only reachable through locators produced by the Python dataset index, and (b) has several defects that limit how much it helps. This issue records what we measured and the concrete fixes that would make prepared series the normal fast path.

Workload

  • 323 regular series × 105,120 points, 5-minute interval, one table, one string TAG (series_id).
  • Values: float32 PLAIN; time: TS_2DIFF; no compression; page size 10,000 points.
  • One window = 2048 rows; 64 windows per batch; warm page cache; one CPU; threads=1; medians over repeated runs.
  • AMD EPYC 9J14, Python 3.11.18, tsfile 2.5.0 wheel plus locally built develop (identical CMake flags).

Measurements (ms per single-window call)

API path ms/window
query_table, batch_size = window length 0.32–0.36
query_prepared, value column, time range, limit = length 0.046
query_prepared, value column, time range, no limit 0.079
query_prepared, valid column only 0.076
two single-column query_prepared calls (value + valid) 0.121
query_prepared_multi, two columns, time range 0.26–0.27
query_prepared_multi, two columns, offset/limit 1.79
prepared-handle acquire 0.004

Per 64-window batch: 20.6–23.1 ms with query_table, ~4–5 ms with prepared queries.

The cost is not proportional to rows: a 0-row query still costs 0.325 ms, a 10,000-row query 0.55 ms, and a full 105,120-row query 0.92 ms (8.8 ns/row).

Where the time goes per query_table call

query_table() itself is lazy (~0.008 ms). The first read_arrow_batch() performs all of the work:

  1. The table-level device index leaf node is read and deserialized to apply the tag filter.
  2. SingleDeviceTsBlockReader::init_internal() calls TsFileIOReader::get_timeseries_indexes(), which re-reads and deserializes the device metadata node via load_device_index_entry() and does not consult device_node_cache_, even though alloc_multi_ssi() later uses the cached node. Measured: repeated queries on the same series (cache hot) cost 0.316 ms vs 0.440 ms when cycling 323 series — the cache only saves ~28%.
  3. A new TsFileSeriesScanIterator + ChunkReader is constructed and destroyed per query.
  4. ChunkReader::get_next_page() walks page headers from the start of the chunk, reading DEFAULT_READ_SIZE = 4096 bytes per header; a 2048-point window triggers ~14 such reads (~57 KiB) even when the filter covers only the first page. There is no early stop once the page range is past the filter.
  5. The intersecting page is decoded in full (10,000 points for a 2048-point window).
  6. The TsBlock is allocated per query with max_row_count = batch_size (e.g. 65,536 → ~0.8 MB); this alone costs 0.079 vs 0.040 ms.

The profile of the per-query work is dominated by allocator traffic and metadata construction; actual decoding (multi_DECODE_TV_BATCH + TS_2DIFF + time filter) is only ~6.5%.

Prepared path defects

  • PreparedSeriesTsBlockReader hard-codes PREPARED_BATCH_ROWS = 65536, so without an explicit limit every prepared query preallocates a 65,536-row block. Passing limit halves the call cost (0.079 → 0.046 ms).
  • Every prepared call still constructs a new SSI/ChunkReader; PreparedSeries only caches the resolved index.
  • query_prepared_multi(..., offset, limit) degrades to 1.79 ms/call, ~40× a single-column prepared call and ~7× the two-column time-range variant.
  • Two single-column prepared calls (0.121 ms) are ~2.2× faster than one query_prepared_multi (0.27 ms) for two columns.
  • The prepared locator is only produced by the Python dataset index (TsFileDataFrame(use_index=True)), so users not using that layer cannot call prepare_series by (table, tag values, field).

Proposed changes

  1. Make get_timeseries_indexes() reuse the parsed device node from device_node_cache_, or drop the redundant call in init_internal since alloc_multi_ssi() already resolves the same node.
  2. Stop scanning page headers once the filter range is exhausted, and/or use a page index to seek.
  3. Push the row offset down to page granularity so a 2048-point window does not decode a 10,000-point page.
  4. In the prepared path: honor the requested row count in PREPARED_BATCH_ROWS, reuse the SSI/ChunkReader across calls on the same prepared series, and expose a public prepare(table, tags, field) entry point.
  5. Fix query_prepared_multi with offset/limit and investigate why two columns are slower than two single-column calls.

Notes

  • PR perf(cpp): speed up wide-table reads (RLE SIMD, batch index loading, BOUNDARY bulk) #1072 (wide-table reads) does not change this workload: with the same build configuration it executes the same number of instructions as develop (8.824G vs 8.825G for 1500 queries), because this dataset uses PLAIN values (no RLE) and one measurement per device (no per-measurement index fan-out).
  • Separately, we measured current develop at ~8.8G instructions vs 7.2G for 2.5.0 on the same 1500-query loop (both locally built with identical CMake flags), i.e. ~21% more instructions per query. We have not bisected this yet; it may deserve its own issue.
  • The benchmark script and the dataset generator parameters can be shared if useful.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    c++Pull requests that update c++ codeperformance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions