Repository navigation
Description
Problem
An aligned C++ table query can spin indefinitely inside the first ResultSet::next() when the internal TsBlock cannot hold the decoder's fixed batch of 129 rows. The query itself returns E_OK; reading its first result never returns.
A reproducer using g_config_value_.tsblock_max_memory_ = 128 hangs with no time or field predicates. The same file/query completes and returns all 63 rows with the default 2 MiB setting (2097152). This is the internal TsBlock memory capacity, not the result API's requested output batch size.
Reproduction conditions
- Aligned table with TAG
idand INT64 fieldsv,w, andghost; PLAIN encoding and UNCOMPRESSED compression. ghostis declared in the schema but is never written.vis absent for one device, and the written value columns also contain nulls.- Three devices, each with 21 timestamps written across three flushes; the page writer limit is 5 points.
- Query
{"v", "ghost", "w", "v"}with no predicates, offset 0, no limit, and serial reading. Missing requested measurements force the per-column aligned read path. - Set the internal TsBlock memory budget to 128 bytes before creating the reader, then call
ResultSet::next().
Expected: return all 63 rows using appropriately sized internal batches. If a capacity is unsupported, return an explicit error rather than retrying without progress.
Actual: the first next() does not return; a bounded reproduction was terminated after 5 seconds.
Standalone C++ reproducer
Save this as small_tsblock_repro.cc and compile it against the built C++ library, for example from the repository root:
c++ -std=c++11 -I cpp/src small_tsblock_repro.cc \
-L /path/to/build/lib -ltsfile -Wl,-rpath,/path/to/build/lib \
-o small_tsblock_repro#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <string>
#include <vector>
#include "common/config/config.h"
#include "common/tablet.h"
#include "file/local_random_access_read_file.h"
#include "file/write_file.h"
#include "reader/table_query_executor.h"
#include "writer/tsfile_table_writer.h"
using namespace common;
using namespace storage;
void check(int ret) { if (ret != E_OK) { std::cerr << "error=" << ret << '\n'; std::exit(2); } }
int main(int argc, char** argv) {
libtsfile_init();
const auto saved = g_config_value_;
g_config_value_.page_writer_max_point_num_ = 5;
g_config_value_.tsblock_max_memory_ = argc > 2 ? std::atoi(argv[2]) : 128;
g_config_value_.parallel_read_enabled_ = false;
const std::string path = argv[1];
{
WriteFile file;
check(file.create(path, O_WRONLY | O_CREAT | O_TRUNC, 0666));
TableSchema schema("t", {
{"id", STRING, UNCOMPRESSED, PLAIN, ColumnCategory::TAG},
{"v", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
{"w", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
{"ghost", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD}});
TsFileTableWriter writer(&file, &schema);
for (int chunk = 0; chunk < 3; ++chunk) for (int d = 0; d < 3; ++d) {
std::vector<std::string> names{"id", "w"};
std::vector<TSDataType> types{STRING, INT64};
std::vector<ColumnCategory> categories{ColumnCategory::TAG, ColumnCategory::FIELD};
if (d != 1) { names.push_back("v"); types.push_back(INT64); categories.push_back(ColumnCategory::FIELD); }
Tablet tablet("t", names, types, categories, 7);
for (int r = 0; r < 7; ++r) {
int t = chunk * 7 + r;
check(tablet.add_timestamp(r, t));
const std::string id = "d" + std::to_string(d);
check(tablet.add_value(r, "id", id.c_str()));
if (t % 3 != 0) check(tablet.add_value(r, "w", int64_t(d * 1000 + t)));
if (d != 1 && t % 4 != 0) check(tablet.add_value(r, "v", int64_t(d * 100 + t)));
}
check(writer.write_table(tablet));
check(writer.flush());
}
check(writer.close());
}
std::cerr << "before query memory=" << g_config_value_.tsblock_max_memory_ << '\n';
LocalRandomAccessReadFile file;
check(file.open(path));
TableQueryExecutor executor(&file, -1);
ResultSet* result = nullptr;
check(executor.query("t", {"v", "ghost", "w", "v"}, nullptr, nullptr, nullptr, 0, -1, result));
std::cerr << "after query before first next\n";
bool next = false;
int ret = E_OK, count = 0;
while ((ret = result->next(next)) == E_OK && next) {
++count;
if (count > 200) return 3;
}
std::cerr << "after scan ret=" << ret << " count=" << count << " expected=63\n";
executor.destroy_query_data_set(result);
std::remove(path.c_str());
g_config_value_ = saved;
libtsfile_destroy();
return ret == E_OK && count == 63 ? 0 : 4;
}Run the two cases with an external 5-second timeout (e.g. timeout on Linux):
timeout 5s ./small_tsblock_repro /tmp/small-tsblock-128.tsfile 128
./small_tsblock_repro /tmp/small-tsblock-default.tsfile 2097152The small-capacity case stops after after query before first next. The default-capacity case prints after scan ret=0 count=63 expected=63 and exits with code 0. Terminate the small-capacity process externally; an iteration counter around next() cannot interrupt this hang.
Cause
AlignedChunkReader::decode_tv_batch checks row_appender.remaining() < BATCH with BATCH = 129 and returns E_OVERFLOW before consuming any input. For the time-plus-INT64 internal block in this reproducer, 128 bytes only allow 8 rows; the condition remains true even on an empty block and even though a page has fewer than 129 points.
decode_time_value_buf_into_tsblock preserves the unfinished page and converts that overflow to E_OK. The series iterator receives an empty successful block, sees that the chunk still has data, and retries. Since neither the input cursor nor available block capacity changes, it repeats without progress.
The multi-column multi_DECODE_TV_BATCH path already caps its effective batch with min(BATCH, row_appender.remaining()). The single-column typed path still has the unconditional 129-row requirement.
Verification and scope
- Reproduced on the base of [C++] Apply field predicates to aligned table query rows #1026,
1180b13675e11a0230708af2cc70d037e0e12623: timeout on the firstnext(). - Reproduced again on standalone PR head
c01ee105d397b1ed16a4daf335db9d8176036bacwith a macOS arm64 Debug library: 128 bytes times out after 5 seconds; 2097152 bytes exits successfully with all 63 rows. No predicates are supplied. - Source inspection of current
develop,eb733133b9eee0b1eb74f7671d809ecd730cedc3, confirms the same fixed-capacity guard remains. The runtime runs above used the PR head/base, not a separately rebuiltdeveloplibrary. - This is a pre-existing reader issue found during review of [C++] Apply field predicates to aligned table query rows #1026. Track its fix separately; no fix is included in that PR. A released-version range has not been established.
Suggested regression coverage should run the sparse/missing-column query with internal capacities below, at, and above 129 rows, check forward progress with a bounded timeout, and compare complete results with the default-capacity query.
Activity
[C++] Bound aligned batch decodes by remaining TsBlock capacity (#1045)