Skip to content

[Bug][C++] Repeated metadata lookups grow hot device-cache arenas #1009

Closed

Description

@ColinLeeo

Summary

Follow-up to #967 and the review discussion.

The bounded device-node LRU limits the number of retained devices, but repeated metadata lookups for a hot device still allocate into that cached node's PageArena. Its memory can therefore keep growing even when the cache contains only one entry and every query/scan iterator is released.

This allocation behavior predates #967; it is a remaining issue, rather than a regression introduced by the bounded LRU.

Reproduction and observed behavior

  1. Create a TsFile containing one non-aligned device, root.followup.d0, with one INT32 measurement, value, and one row.
  2. Keep one TsFileIOReader open. Warm the device cache once.
  3. Repeat alloc_ssi() / revert_ssi() for that same series 10,000 times, resetting the caller's arena after each iteration.

Core test logic, after opening reader on the file:

auto device = std::make_shared<StringArrayDeviceID>("root.followup.d0");
PageArena query_pa;
query_pa.init(512, MOD_TSFILE_READER);
auto load = [&]() {
    TsFileSeriesScanIterator* ssi = nullptr;
    int ret = reader.alloc_ssi(device, "value", ssi, query_pa);
    if (ssi) reader.revert_ssi(ssi);
    query_pa.reset();
    return ret;
};

ASSERT_EQ(load(), E_OK);
const auto before = reader.TEST_reader_memory_bytes();
for (size_t i = 0; i < 10000; ++i) ASSERT_EQ(load(), E_OK);
const auto after = reader.TEST_reader_memory_bytes();
EXPECT_EQ(reader.TEST_device_node_cache_size(), 1u); // passes
EXPECT_EQ(after, before);                          // fails

With a Debug build and ENABLE_TEST=ON, ENABLE_MEM_STAT=ON, the MOD_TSFILE_READER accounting reported:

After warm-up After 10,000 additional lookups
Cache entries 1 1
Accounted bytes 1,072 84,688

These are allocator accounting values, not process RSS. The reproduction was run on review revision 6a63ff35d9e94b889998dd69408ceecc1dad598d; the relevant reader and metadata allocation paths remain unchanged in the merged PR head, 1dfd2dc4d5daf91b9beca2da265a094126494633.

No row iteration is needed to trigger this: it happens during scan-iterator creation / measurement metadata lookup. Query teardown does not reclaim allocations retained by the cached arena.

Cause

A cache hit reuses the device's top measurement index node. load_timeseries_index_for_ssi() still resolves the measurement and reads/parses its timeseries metadata.

MetaIndexNode::binary_search_children() returns children_[l]->clone(pa_). For a MeasurementMetaIndexEntry, cloning duplicates the measurement name into the source node's arena. The binary search itself is not the retained allocation; the result copy is.

Those bytes are reclaimed when the owning arena is destroyed, rather than when the temporary result is released. A hot cache entry can stay resident indefinitely, so bounding entry count does not bound this growth.

Suggested fix and validation

  • Give lookup-result copies an explicit query/result-owned destination arena, or use another representation with a safe ownership lifetime. Keep a published cached node's arena unchanged during lookups.
  • Audit multi-level index traversal when changing this ownership. Some nodes use temporary arenas; simply returning a child pointer or removing cloning globally could create dangling references.
  • Add a repeated-hot-series regression test: retained metadata memory should plateau after warm-up while cache size stays constant. Also cover multi-level indexes and active iterators surviving cache eviction. Guard accounting assertions with ENABLE_MEM_STAT, while keeping functional checks enabled without it.

Optional subsequent performance improvement

Separately, consider lazily caching resolved, parsed (device, measurement) metadata with an owning handle, building on PreparedSeries / init_prepared(). A hit could avoid repeated measurement-index traversal and metadata I/O/deserialization. Filters, scan cursors and decoding buffers would remain query-local.

Such a cache should be scoped to the reader/file lifetime, have a memory budget, and share aligned time metadata where appropriate. The miss path would still need the allocation-lifetime fix above. Benchmark repeated and interleaved queries using metadata lookup/read/parse counts, query setup time and retained memory to quantify the benefit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions