Repository navigation
Description
Problem
A C++ aligned table query using TimeIn can silently omit matching rows after decoding a page with no matching timestamps. The page decoder reports E_NO_MORE_DATA for an empty filtered result even when the same chunk still contains unread pages. The series iterator then skips the remainder of that chunk or ends the series.
This was independently reproduced on the current default branch, develop, at eb733133b9eee0b1eb74f7671d809ecd730cedc3. It is a pre-existing reader bug found while reviewing #1026; please track its fix separately from that PR.
Reproduction
- Write an aligned table with TAG
idand INT64 fieldsv,w, andghost, using PLAIN encoding and UNCOMPRESSED compression. - Use three devices, each with timestamps 0 through 20, written in three groups of seven rows with an explicit flush after each group. Set
page_writer_max_point_num_ = 5. - Leave
ghostunwritten, omitventirely for one device, and include nulls inv/w. Requesting the missing measurement makes the query use the per-column aligned read path. - Use serial reading and a 4096-byte internal TsBlock budget, which accommodates the decoder's 129-row batch for these single-column blocks.
- Query
{"v", "ghost", "w", "v"}withTimeIn({0, 2, 3, 6, 7, 10, 13, 19, 20}, false), no field predicate, offset 0, and no limit.
Expected: 9 matching timestamps per device, 27 rows total. Null field values must not remove rows that match the time predicate.
Actual: 7 timestamps per device, 21 rows total. Each device loses timestamps 19 and 20. The API returns E_OK and reports normal end of results; it does not report an error.
Expected per device: 0 2 3 6 7 10 13 19 20
Actual per device: 0 2 3 6 7 10 13
Standalone C++ reproducer
Save this as timein_empty_page_repro.cc. From the repository root, compile it against the built C++ library, for example:
c++ -std=c++11 -I cpp/src timein_empty_page_repro.cc \
-L /path/to/build/lib -ltsfile -Wl,-rpath,/path/to/build/lib \
-o timein_empty_page_repro
./timein_empty_page_repro /tmp/timein-empty-page-repro.tsfileThe program prints after scan ret=0 count=21 expected=27 and exits with code 4 on the affected implementation. Correct behavior is 27 rows and exit code 0.
#include <cstdio>
#include <cstdlib>
#include <iostream>
#include <string>
#include <vector>
#include "common/config/config.h"
#include "common/tablet.h"
#include "file/local_random_access_read_file.h"
#include "file/write_file.h"
#include "reader/filter/time_operator.h"
#include "reader/table_query_executor.h"
#include "writer/tsfile_table_writer.h"
using namespace common;
using namespace storage;
void check(int ret) { if (ret != E_OK) { std::cerr << "error=" << ret << '\n'; std::exit(2); } }
int main(int argc, char** argv) {
libtsfile_init();
const auto saved = g_config_value_;
g_config_value_.page_writer_max_point_num_ = 5;
g_config_value_.tsblock_max_memory_ = argc > 2 ? std::atoi(argv[2]) : 4096;
g_config_value_.parallel_read_enabled_ = false;
const std::string path = argc > 1 ? argv[1] : "/tmp/timein-empty-page-repro.tsfile";
{
WriteFile file;
check(file.create(path, O_WRONLY | O_CREAT | O_TRUNC, 0666));
TableSchema schema("t", {
{"id", STRING, UNCOMPRESSED, PLAIN, ColumnCategory::TAG},
{"v", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
{"w", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD},
{"ghost", INT64, UNCOMPRESSED, PLAIN, ColumnCategory::FIELD}});
TsFileTableWriter writer(&file, &schema);
for (int chunk = 0; chunk < 3; ++chunk) for (int d = 0; d < 3; ++d) {
std::vector<std::string> names{"id", "w"};
std::vector<TSDataType> types{STRING, INT64};
std::vector<ColumnCategory> categories{ColumnCategory::TAG, ColumnCategory::FIELD};
if (d != 1) { names.push_back("v"); types.push_back(INT64); categories.push_back(ColumnCategory::FIELD); }
Tablet tablet("t", names, types, categories, 7);
for (int r = 0; r < 7; ++r) {
int t = chunk * 7 + r;
check(tablet.add_timestamp(r, t));
const std::string id = "d" + std::to_string(d);
check(tablet.add_value(r, "id", id.c_str()));
if (t % 3 != 0) check(tablet.add_value(r, "w", int64_t(d * 1000 + t)));
if (d != 1 && t % 4 != 0) check(tablet.add_value(r, "v", int64_t(d * 100 + t)));
}
check(writer.write_table(tablet));
check(writer.flush());
}
check(writer.close());
}
std::cerr << "before query memory=" << g_config_value_.tsblock_max_memory_ << '\n';
LocalRandomAccessReadFile file;
check(file.open(path));
TableQueryExecutor executor(&file, -1);
ResultSet* result = nullptr;
check(executor.query("t", {"v", "ghost", "w", "v"}, new TimeIn({0, 2, 3, 6, 7, 10, 13, 19, 20}, false), nullptr, nullptr, 0, -1, result));
std::cerr << "after query before first next\n";
bool next = false;
int ret = E_OK, count = 0;
while ((ret = result->next(next)) == E_OK && next) {
++count;
std::cout << result->get_value<int64_t>(1) << " ";
if (count > 200) return 3;
}
std::cerr << "after scan ret=" << ret << " count=" << count << " expected=27\n";
executor.destroy_query_data_set(result);
std::remove(path.c_str());
g_config_value_ = saved;
libtsfile_destroy();
return ret == E_OK && count == 27 ? 0 : 4;
}Root cause
The final chunk for each device has two time pages:
| Page | Timestamps | Required behavior |
|---|---|---|
| First | 14 through 18 | No timestamps match; continue scanning the same chunk |
| Second | 19 and 20 | Both timestamps match and must be returned |
AlignedChunkReader::decode_tv_buf_into_tsblock_by_datatype changes a successful decode into E_NO_MORE_DATA whenever the output TsBlock is empty:
if (ret_tsblock->get_row_count() == 0 && ret == E_OK) {
ret = E_NO_MORE_DATA;
}That conflates a page with no matching rows with exhaustion of the data stream. The trace after decoding the 14-through-18 page shows:
ret=E_NO_MORE_DATA rows=0 has_more_data=true has_next_chunk=false
TsFileSeriesScanIterator::get_next only retries an empty block when ret indicates success. With E_NO_MORE_DATA, it advances directly to the next chunk if one exists; otherwise it returns the end-of-data status. In this reproduction the final chunk has no successor, so the column is treated as finished and its page containing timestamps 19 and 20 is never read.
TimeIn::satisfy(Statistic*) conservatively returns true, so this page reaches row-level filtering rather than being skipped by page statistics. The missing-column projection exposes the affected single-column aligned path. The underlying defect is the empty-page/end-of-stream status handling, not the set membership test itself.
Diagnostic verification
Using a macOS arm64 Debug build with SYSTEM LZ4 and other optional compressors disabled:
- Unmodified
developateb733133b9eee0b1eb74f7671d809ecd730cedc3: 21 of 27 rows returned. - In an isolated temporary source copy, removing only the empty-output-to-
E_NO_MORE_DATAconversion above restores all 27 expected timestamps. The no-time-filter control still returns all 63 rows. This is a targeted diagnostic experiment, not a fully validated production fix. - Restoring the original source and rebuilding reproduces 21 of 27 rows again.
- The same failure was previously reproduced on [C++] Apply field predicates to aligned table query rows #1026's base
1180b13675e11a0230708af2cc70d037e0e12623and standalone headc01ee105d397b1ed16a4daf335db9d8176036bac. No fix has been added to that PR. A released-version range has not been established.
This is separate from the small-capacity hang in #1035: this reproduction has enough internal block capacity, terminates normally, and returns incomplete results.
Suggested regression coverage should include fully filtered pages followed by matching pages within the same chunk, both with and without subsequent chunks, and check the sparse/missing-column read path as well as all-matching and no-matching controls.
Activity
[C++] Keep scanning after a fully filtered aligned page (#1038) (#1044)