Skip to content

[C++] Apply field predicates to aligned table query rows - #1026

Merged
ColinLeeo merged 17 commits into
apache:developfrom
jixuan1989:cpp/fix-table-field-filter
Oct 27, 2026
Merged

ColinLeeo merged 17 commits into
apache:developfrom
jixuan1989:cpp/fix-table-field-filter

Conversation

@jixuan1989

@jixuan1989 jixuan1989 commented Oct 18, 2026 •

Copy link
Copy Markdown
Member

Query acceptance UNCLEARED; source/write/CI acceptance also uncleared. Fresh cf4c9aa rounds contain 82 cases/98 comparisons with a repeated increase and 5 confirmation-only comparisons. Largest repeated increase: layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]. Largest against preceding measured eee: layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]. No numerical allowance was supplied; necessary/minimized extra cost or causal attribution is unproven. All samples and adverse results remain. No source push, merge, new issue or Copilot request. This is diagnostic publication only.

查询验收尚未通过;源码、写入和CI验收也未通过。 cf4c9aac全新轮次有82场景/98项比较重复上升,另有5项仅确认轮信号。最大重复上升:layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]。相对前次已测eee的最大值:layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]。用户未给数值容差;额外成本的必要性、最小性和因果归因均未证明。所有样本及不利结果保留。未推送源码、合并、创建issue或请求Copilot,本次仅发布诊断。

Fixes #1025.

Table queries previously ignored FIELD predicates. The change binds typed predicates once, reads predicate columns even when not returned, and evaluates each merged row with SQL NULL, FLOAT/DOUBLE NaN and logical NOT semantics before projection/OFFSET/LIMIT. It supports row/batch, aligned/non-aligned/mixed data and callback fallback. Current maintainer revisions also harden terminal errors/resource cleanup and use per-task decoder/compressor slots across shared-library callers. Query owns transferred filters; binding is single-use. Unsupported/invalid predicate state returns an error. This does not establish all allocation/submit/future exception safety.

精确维护者head为cf4c9aac。此前表查询忽略FIELD条件;本修改单次绑定有类型谓词,读取未返回但过滤所需的列,在投影/OFFSET/LIMIT前按合并时间行处理SQL NULL、FLOAT/DOUBLE NaN和逻辑NOT。覆盖行/批、对齐/非对齐/混合及回调回退;后续维护者修改强化终止错误与资源清理,跨共享库调用使用每任务decoder/compressor槽。query取得filter所有权,绑定约定一次;非法/不支持状态返回错误,并不证明所有分配/submit/future异常安全。

Fresh exact-head evidence: Windows1,163 full passes +3skips/10disabled (245focused); Linux245focused and317reader passes +1disabled, but full strict sanitizer fails in unchanged LZOkay. JDK24 affected139tests and1,896 standard full cross-reads pass. Actual Java boxed-number oracle matches55,296 masks/1,536bindings. Since073 changed C++ lines81/89=91.01%,branches92/136=67.65%; whole PR vs current develop lines1,075/1,263=85.11%,branches1,172/1,796=65.26%;22-file module lines78.14%,branches45.86%. Fresh Java module coverage unmeasured; zero Java production diff is N/A, not100%.

全新验证:Windows全量1163通过、3跳过/10禁用(重点245);Linux重点245、reader317通过/1禁用,但严格全量在未改LZOkay失败。JDK24受影响139项、标准1896次完整双向交叉读取通过;真实Java装箱数字参考核对1536次绑定/55296个mask。相对073的C++修改行81/89=91.01%、分支92/136=67.65%;相对当前develop整PR行1075/1263=85.11%、分支1172/1796=65.26%;22文件模块行78.14%、分支45.86%。新Java模块覆盖未测,零Java生产修改为N/A而非100%。

The186 new query cases retain all eight original benchmarks:96main+14long-name/context+76non-aligned/mixed layouts. New layouts test bucket0/1/50/100 AND v0 IS NULL, actual2/5/9FIELD projections, row/batch1024, time/TAG/requested parallel0/1. Bucket selectivity differs from combined hit rate; non-aligned hidden-filter-only and dense/unfiltered100% combined-hit timing remain unmeasured. Both old callback and native paths include identical physical/output bucket projection and full checking/materialization; this is whole-query cost, not isolated engine cost. Warm cache, pointwise block-bootstrap intervals; phases/P50/P95/throughput/whole-process memory are reported where measured. Original8 still lack phase/individual-quantile/memory metrics. Writer throughput is not newly accepted: historical2048-byte header preflight remains error6 on fresh cf4 reader. Cold/P99/compressed/scaling/query-only memory and other PR01/02 gaps remain.

新增186项(主96、长名/上下文14、非对齐/混合76)保留原8基准;布局采用bucket0/1/50/100且v0 IS NULL、实际2/5/9个FIELD、行/batch1024、时间/TAG/请求parallel0/1。bucket命中率不同于组合命中率;非对齐隐藏过滤列、稠密/无过滤100%组合命中计时仍未测。旧回调与native实际/输出均含相同bucket列,完整核对/物化成本计入;比较是整查询而非纯引擎成本。暖缓存、逐项块bootstrap,适用范围报告分项/P50/P95/吞吐/整进程内存,原8仍缺分项/单次分位数/内存。写入吞吐没有新验收:旧2048字节header预检对新cf4 reader仍error6;冷缓存/P99/压缩/扩展/query专属内存及PR01/02缺口保留。

Native Java Table reading independent sparse columns fails the interior NULL oracle on old C++, cf4 C++ and Java-written files. Raw Java per-series maps/union verify physical absence and all cells; map materialization is untimed and is not native Java Table NULL acceptance or Java performance. Shared caller parent10×4 and cf4 100×4 cases pass; the old race was not reproduced. CodeQL Analyze(java) failed near SARIF upload without an explicit root cause in the job log; local Java tests do not clear CI. All setup failures are retained, including corrected stack TimeBetween ownership misuse in the harness affecting all readers, not an old-only production regression.

Java原生Table在旧C++、cf4 C++及Java文件的独立稀疏列均有内部NULL错误;逐序列map/union核对实际缺值与全部单元,物化未计时,不能声称原生Table NULL或Java性能通过。共享调用父版10×4及cf4 100×4均通过,旧竞态未复现。CodeQL Analyze(java)在SARIF上传附近失败,日志无明确根因,本地Java不能清除CI。保留准备失败,含已修正、影响各reader的栈TimeBetween所有权脚本误用,并非旧版单独退化。

Complete EN/ZH report · all ratios · metrics · raw evidence and actual final independent reviews. Both final reviewers authorize diagnostic publication only; no source gate pass or historical retroactive approval. Historical evidence and the full preceding PR body retain earlier failed signals/gaps and review history. No new issue, production push, upstream merge or Copilot request.

以上链接提供完整双语报告、所有比率/指标、原始证据及真实独立终审;两位仅准许诊断发布,不代表源码门槛通过或历史追认。历史报告及完整旧正文继续保留此前失败信号、缺口和审阅记录。未新建issue、推送生产源码、主动合并或请求Copilot。

Unpublished local candidate: expanded query investigation (2026-10-27)

Current upstream status (checked after measurement): Upstream maintainer ColinLeeo merged this PR at2026-10-27T06:28:01Z, merge commit 0d42092cd51993dc82ab3f390d7866d6404d11a1; PR head remains cf4c9aacbaf009915327838569db97129eb4f584. Current checks:47success/1neutral/1skipped; Analyze(java) now succeeds (job). Its earlier failed state is retained as history. The frozen report and diagnostic reviews predate the upstream merge/status refresh. Their failed CI statements describe that snapshot, not the latest upstream result. There is no CI run for unpublished local0928, whose source/query/writer/CI gates remain false. The agent performed no merge or source push.

当前上游状态(计时后核查): 上游维护者ColinLeeo于2026-10-27T06:28:01Z合并本PR,merge commit为0d42092cd51993dc82ab3f390d7866d6404d11a1,PR head仍为cf4c9aacbaf009915327838569db97129eb4f584。当前检查47success/1neutral/1skipped,Analyze(java)现已成功(任务链接见上);旧失败作为历史保留。冻结报告及诊断终审早于上游合并/状态刷新,其中CI失败仅指当时快照,不代表最新上游结果。本地未发布0928没有CI运行,源码/查询/写入/CI门槛仍全部false;代理未执行合并或源码推送。

Current public head remains cf4c9aacbaf009915327838569db97129eb4f584; the implementation of local candidate 0928f7a82f7ca21d5d5dc90b2195538167917097 has not been pushed. A separate fixed282-case round on that candidate versus workspace 50a25ab8ab6589c483a639d5a37d3b8e8bbe6a0a and correct EEE eee39799948699c58e292ee6361f1da8154378c2 is complete:27243 processes,74628 formal queries (74304 full/324checked-prefix early-close),10206 excluded warmups; preflight2640full/18prefix separate. Original186 configurations plus96 aligned/nonaligned/mixed, dense/sparse, scan/100%/hidden FIELD-hit/projection cases retain all8original benchmarks. Initial18/confirmation36 paired blocks selected81cases once; no extensions/pooling. Complete cell/count/order/NULL oracles checked each consumed result.

Repeated/initial-only/confirmation-only increases are38/15/2 against workspace and42/5/3 against EEE. Worst repeated confirmation ratios:workspace1.140823726 (+14.08%), pointwise95 [1.069005129,1.222774720]; EEE1.123465096 (+12.35%), [1.065129776,1.190903690]. Both largest cases are unfiltered scans where the guarded FIELD type accessor is inactive; no getter/CPU/cache cause or necessary-minimal exception is established. The previous local28-case12EEE increases (worst7.33%) and all publiccf4 signals above remain unresolved independently. No equivalence or multiplicity-adjusted inference is claimed.

Full bilingual report · raw samples, all ratios/metrics, harness, protocol history and actual diagnostic reviews. File-open/combined metadata-plan/first-result/lifecycle, rows/s/logical FIELD-points/s, descriptive P50/P95 and whole-process peak memory are recorded within stated boundaries. New original8 phase metrics apply to these local versions only; historical publiccf4/PR01/02 gaps remain. Correct EEE callback costs include extra hidden-column materialization/checking where applicable. New18C++ products match bytes by six configurations; new-fixture Java cross-read and write throughput are unmeasured. Existing0928 correctness/Java/coverage/sanitizer evidence remains separate. All source/query/writer/CI gates remain false; CodeQL/LZOkay, whole-PR coverage<90%, long-header/Java alias/otherNULL, binary FIELD timing/encryption/cold/P99/codecs/scaling/query-only-memory gaps remain. Both actual independent sessions (gpt-6.1-sol and gpt-6-astra) reviewed the frozen evidence; this is diagnostic permission, not source acceptance. No source push, upstream merge, new issue or Copilot request.

完整中文译稿

公开head仍为cf4c9aacbaf009915327838569db97129eb4f584,本地候选0928f7a82f7ca21d5d5dc90b2195538167917097的实现未推送。该候选相对工作区50a25ab8ab6589c483a639d5a37d3b8e8bbe6a0a及正确EEE eee39799948699c58e292ee6361f1da8154378c2的新固定282场景轮次已完成:27243进程、74628正式查询(74304完整/324核对前缀提前关闭)、10206排除的预热,另有预检2640完整/18前缀。原186配置加96项对齐/非对齐/混合、稠密/稀疏、扫描/100%/隐藏FIELD命中率/投影场景,保留原8基准。初测18块/确认36配对块,81场景仅选择一次,不追加或合并;逐次核对已消费结果的单元、计数、顺序及NULL。

相对工作区重复/仅初测/仅确认增幅为38/15/2,相对EEE为42/5/3。最大重复确认比值相对工作区1.140823726(+14.08%),逐项95%区间[1.069005129,1.222774720];相对EEE为1.123465096(+12.35%),[1.065129776,1.190903690]。两个最大场景均为守卫不执行FIELD类型访问器的无过滤扫描,未证明getter/CPU/缓存因果或必要最小成本例外。此前本地28场景的12个EEE增幅(最大7.33%)及上述公开cf4信号独立保留;不声称等价或多重比较校正推断。

完整双语报告及原始样本、全部比率/指标、驱动、协议历史和真实诊断审阅见以上链接。按所述边界提供文件打开/混合元数据规划/首结果/生命周期、行/秒及逻辑FIELD点/秒、描述性P50/P95、整个进程峰值内存。原8新增阶段指标仅属于本地这些版本,历史公开cf4/PR01/02缺口保留。适用时EEE回调包含额外隐藏列物化/核对成本。18个新C++文件按六配置字节相同,新夹具Java交叉读取及写入吞吐未测;既有0928正确性/Java/覆盖率/sanitizer证据独立。源码/查询/写入/CI门槛全部false;CodeQL/LZOkay、整PR覆盖率<90%、长头部/Java别名/另一NULL、binary FIELD计时/加密/冷缓存/P99/codec/扩展性/查询专属内存缺口保留。两个实际独立会话(gpt-6.1-sol与gpt-6-astra)审阅了冻结证据,仅许可诊断,不代表源码验收。未推源码、合并上游、发布新issue或请求Copilot。

AI disclosure: The implementation, tests and description in this pull request were written by an AI coding agent (OpenAI Codex), which also performed the reported local validation.

AI 使用说明: 本 PR 的代码修改、测试和正文由 AI 编程代理(OpenAI Codex)编写,文中报告的本地验证也由该 AI 代理执行。

Copilot AI balanced review requested due to automatic review settings October 18, 2026 00:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The scalar fallback can lose matching NULL rows and return nullable Boolean rows outside the time filter.

3 open issues
What changed in this PR

Addresses #1025 by evaluating FIELD predicates in C++ aligned table queries while preserving the unfiltered fast path.

Changes:

  • Adds typed, NULL-aware row predicates and legacy callback dispatch.
  • Evaluates predicates before OFFSET/LIMIT and advances rejected rows.
  • Adds tests for types, NULLs, pagination, and callback errors.
File Description
cpp/​test/​reader/​table_field_filter_test.cc Adds predicate and query regression tests.
cpp/​src/​reader/​filter/​table_field_filter.h Defines borrowed typed row views and predicate API.
cpp/​src/​reader/​filter/​table_field_filter.cc Implements typed and legacy predicate dispatch.
cpp/​src/​reader/​block/​single_device_tsblock_reader.h Declares FIELD evaluation helper.
cpp/​src/​reader/​block/​single_device_tsblock_reader.cc Integrates filtering into scalar row processing.

Review effort: Balanced

}
}
if (can_multi) {
if (can_multi && field_filter == nullptr) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. The aligned value page's time statistics exclude NULL entries, so they cannot rule out candidate timestamps in the shared time page. I reproduced this with (t0,42),(t1,NULL),(t2,NULL) and a time window [t1,t2]: the prior PR returned 0 rows, while Java and the corrected C++ return 4 across two devices, on either writer's file. Commit 249b0d5 uses only shared time-page statistics for this conservative page check. Missing time statistics keep the page. The new test runs with row/batch-1/batch-4 and parallel on/off.

The revision also fixes the test-only TAG predicate leak behind the two Linux ASan CI failures: the query borrows this predicate, so it now has stack ownership. The old test reproduced 480 leaked bytes in six allocations; the revised Linux ASan/UBSan full suite passes with leak detection enabled.

Validation for 249b0d5: all 24 added regression configurations fail on the old PR head and pass after the fix; 74 focused, 979 Windows Release, 1000 all-codec Linux, and 999 Linux ASan/UBSan full-suite tests pass. Twenty added NULL-boundary Java/C++ own/cross-reader results match. Whole-PR changed executable lines: 87/89 (97.75%); review-only: 12/12 (100%). Query evidence retains the original eight cases and adds 30 native FIELD selectivity/projection cases with verified full results, open/metadata/first-row/full timing and process peak memory; comparison boundaries and unmeasured scope are documented.

Revision report · Source, raw test and performance evidence

AI disclosure: This analysis, implementation, validation and reply were produced by OpenAI Codex.

AI 使用说明:本分析、修改、验证和回复由 OpenAI Codex 完成。

}
}
if (can_multi) {
if (can_multi && field_filter == nullptr) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. The BOOLEAN NULL scalar branch skipped the timestamp predicate and could emit an out-of-window NULL before FIELD evaluation and OFFSET/LIMIT. Commit 249b0d5 checks that timestamp, backs off the provisional row, and continues without consuming a value for a NULL. The two-device reproduction previously returned 6 rows for the window; revised C++ and Java return the same 4, with both Java-written and C++-written inputs. The test also verifies callback timestamps and OFFSET/LIMIT in all six row/batch/parallel configurations.

Validation for 249b0d5: all 24 added regression configurations fail on the old PR head and pass after the fix; 74 focused, 979 Windows Release, 1000 all-codec Linux, and 999 Linux ASan/UBSan full-suite tests pass. Twenty added NULL-boundary Java/C++ own/cross-reader results match. Whole-PR changed executable lines: 87/89 (97.75%); review-only: 12/12 (100%). Query evidence retains the original eight cases and adds 30 native FIELD selectivity/projection cases with verified full results, open/metadata/first-row/full timing and process peak memory; comparison boundaries and unmeasured scope are documented.

Revision report · Source, raw test and performance evidence

AI disclosure: This analysis, implementation, validation and reply were produced by OpenAI Codex.

AI 使用说明:本分析、修改、验证和回复由 OpenAI Codex 完成。

}
}
if (can_multi) {
if (can_multi && field_filter == nullptr) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed. A sole all-NULL aligned FIELD has a zero-count value index and no value context, but still has shared timestamps that must reach a NULL-aware predicate. Commit 249b0d5 uses the existing time-only context specifically for that aligned, zero-value-count, non-null-FIELD-filter case. The empty-result shortcut remains for non-aligned or other time-filtered-out value queries. Before the revision the reproduction returned 0 rows; revised C++ and Java return 28, or the same 2 after a time window and OFFSET/LIMIT, on both writers' files. Always-false, legacy-NULL rejection, and wholly outside-window controls are covered too.

Validation for 249b0d5: all 24 added regression configurations fail on the old PR head and pass after the fix; 74 focused, 979 Windows Release, 1000 all-codec Linux, and 999 Linux ASan/UBSan full-suite tests pass. Twenty added NULL-boundary Java/C++ own/cross-reader results match. Whole-PR changed executable lines: 87/89 (97.75%); review-only: 12/12 (100%). Query evidence retains the original eight cases and adds 30 native FIELD selectivity/projection cases with verified full results, open/metadata/first-row/full timing and process peak memory; comparison boundaries and unmeasured scope are documented.

Revision report · Source, raw test and performance evidence

AI disclosure: This analysis, implementation, validation and reply were produced by OpenAI Codex.

AI 使用说明:本分析、修改、验证和回复由 OpenAI Codex 完成。

@ColinLeeo
ColinLeeo requested a balanced review from Copilot October 19, 2026 07:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

A FIELD predicate must see rows whose timestamps lie in the selected
time range even when the column's non-NULL values all lie outside it.
Replace the all-values-NULL check in the empty-context shortcut with an
aligned-source check so such devices fall through to the time-only
context instead of returning an empty result.
decode_tv_batch<int32_t> replaced this loop in apache#823, leaving the function
uncalled. Its NULL branch also predates the time-filter fix applied to
the other decode paths in this series.
@ColinLeeo

ColinLeeo commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for your pr 🌹 .

Follow-up review fixes on top of the two commits above:

1. Empty-context shortcut missed all-NULL-in-range rows (f2e157ed3)

The committed version keyed the empty-context shortcut on all_values_null_aligned (value statistic count == 0). That misses a case: a column whose non-NULL values all lie outside the selected time range is still all-NULL inside it, yet its statistic has count > 0, so the reader returned an empty result and the FIELD predicate never ran for those rows.

The fix relaxes the check to all_value_columns_aligned: when every requested column is aligned, the reader falls through to the time-only context, and the shared time axis delivers those rows to the predicate with all fields NULL.

New test NullOnlyChunkInTimeRangeReachesPredicate covers this (chunk_null has values only in the first chunk; with a time filter over the second chunk the predicate must still be called for every row). Verified against the pre-fix code first: the test fails 6/6 there (empty result, zero predicate calls).

2. Removed dead per-value INT32 decode (6f4a971ee)

AlignedChunkReader::i32_DECODE_TYPED_TV_INTO_TSBLOCK has no call sites since #823 replaced the INT32 dispatch with decode_tv_batch<int32_t>. Its NULL branch also still emits NULL rows unconditionally (the same gap this series fixes in the other decode paths), so it was deleted rather than patched.

Verification: field-filter tests 80/80, targeted reader regressions 160/160, full C++ suite 1229 passed / 0 failed.

@jixuan1989

Copy link
Copy Markdown
Member Author

Confirmed both changes in your follow-up. A whole-column positive value count does not establish any non-NULL value in the selected range, so the aligned shared-time fallback is needed here. The new six-configuration regression fails on the old library and passes at 6f4a971eefffcc81cf62a75b6ad8bc91460b0fa6. I also checked that the removed private helper had no call sites and that active aligned INT32 decoding still uses decode_tv_batch<int32_t>.

Local checks: 80 permanent focused cases, 24 extra boundary cases, and three actual non-aligned gap cases pass. Windows Release (codecs off, same retained baseline configuration): 985 passed, 3 skipped, 10 disabled. All-codec Linux combined ASan/UBSan focused/extra checks pass; reader scope is 343 passed, 2 external-fixture skips, 9 disabled. Forty-eight final-C++/Java own/cross reads match times, order and typed values. Maintainer-delta coverage is 4/4 executable lines (100%) and 5/6 branches (83.33%); affected reader files overall are 79.81% / 55.92%.

Query acceptance remains NOT cleared. The frozen protocol retained 68 initial cases and independently confirmed seven initial flags, with 20,928 fully consumed/checked formal queries. Three remain higher: null_w4_p1_b1024_tag0_parallel0 +0.826% (95% +0.125% to +1.454%), dense_w4_p10_b1024 +0.902% (+0.528% to +1.321%), and dense_w4_p100_b1024 +1.433% (+0.380% to +1.998%). Their necessity is unproven. NULL comparisons use a correct app-level reference including dense anchor materialization; the old incorrect zero-row native path is not timed as a fast equivalent. Native-after / old-reference ratios reach 1.7273x with different projection/predicate/materialization costs, so this is disclosed with that boundary rather than attributed solely to engine regression.

The broader sanitizer runs are not clean. An unchanged TS2DIFF signed overflow aborts the Java-fixture read; byte-identical old/new headers reproduce it on the same valid raw page. A separately filtered remainder then hits an unaligned read in the unchanged pinned LZOkay dependency. Failed logs are preserved, with no suppression. Reader-scope passes above must not be read as a full sanitizer pass. Writing was not retimed for this reader-only follow-up.

中文说明: 两处修改的功能判断已由上述测试确认,但查询门槛仍未通过:三个复核项分别上升约 0.83%、0.90%、1.43%,尚未证明代价必要。原生 NULL 路径/旧正确应用层参考最高 1.7273 倍,两者工作不同,不能据此单独归因于引擎。完整 sanitizer 检查也因未改动的 TS2DIFF/LZOkay 问题中止,原始失败日志保留,不能以 reader 专项通过替代全量通过。中英文警告和提交范围已置于 PR 正文顶部。

Bilingual report · Sources, all raw results and hash-verified evidence parts. Earlier measurements remain under their original head/scope. The independent agent found no unresolved actionable source issue; the complete different-model final review gate is not passed. No production code was authored/pushed by Codex in this follow-up.

AI disclosure: This analysis, local testing, documentation and reply were produced by OpenAI Codex. The two reviewed production commits were authored by maintainer ColinLee/ColinLeeo.

AI 使用说明: 本分析、本地测试、说明和回复由 OpenAI Codex 完成;被验证的两次生产代码提交由维护者 ColinLee/ColinLeeo 编写。

@ColinLeeo

Copy link
Copy Markdown
Contributor

After validation, the sanitizer issues are confirmed. The current sanitizer checks require stricter tracking. The related failures originate from TS2DIFF and the third-party LZOkay dependency, and will be tracked in separate issues and PRs.

@ColinLeeo

Copy link
Copy Markdown
Contributor

The current implementation is correct, but I do not think the row-oriented path is the right final execution model for aligned data.

For aligned series, all columns already share the same ordered time axis. The existing time-filter path also provides the necessary scan-pruning foundation: device/chunk/page statistics are used to determine what actually needs to be decoded. We should extend the same idea to FIELD filtering:

  1. apply the time filter and prune device/chunk/pages;
  2. collect all columns referenced by the FIELD filter;
  3. decode those columns as aligned batches;
  4. evaluate column predicates into per-column masks;
  5. combine the masks according to the logical expression into one final selection mask;
  6. materialize only the rows/columns required by the result and apply OFFSET/LIMIT.

This avoids per-row timestamp merging and TableFieldRow construction. Non-aligned series can continue to use the existing row-based path.

This batch-oriented design is also intended to address the performance regressions found in the current validation: it preserves aligned batch decoding and avoids the overhead of the per-row merge and filter path.

I plan to implement this as a follow-up rather than continuing to optimize the current per-row loop. Keeping this PR focused on correctness also makes the change easier to review.

@jixuan1989

Copy link
Copy Markdown
Member Author

Agreed to keep this PR focused on correctness and handle aligned batch execution in the follow-up you described. At 6f4a971ee, a non-null FIELD filter bypasses both the multi-column context and aligned bulk-copy paths; the current path merges timestamp cursors and constructs a TableFieldRow for each candidate. This supports the architectural direction, but does not establish the cause of each small old/new latency signal or prove a future speedup.

One API boundary to preserve: TableFieldFilter::satisfy_row() is an arbitrary callback over projected fields. It currently exposes neither referenced-column dependencies nor an expression tree, and unprojected FIELDs are not loaded automatically. Steps 2/4/5 therefore need an explicit predicate-dependency/expression or batch contract, or a compatible fallback for existing callbacks; collecting filter-only columns is an intentional extension rather than something the current planner can infer.

The existing frozen regression evidence is useful for the follow-up: shared timestamps must retain all-NULL selected ranges; value-only statistics must not discard those rows; NULL/binary/primitive values must retain their current semantics and lifetime; OFFSET/LIMIT count only selected rows across batches/devices; callback/decode errors remain terminal. I rechecked the Windows and Linux XML records for the six configurations of the NULL-range, all-NULL, typed-value, post-filter OFFSET/LIMIT and callback-error tests. These are existing passes on the unchanged head, not new execution or validation of the proposed batch implementation.

I also recorded your confirmation of the sanitizer failures as separate follow-up work. The published baseline reproductions and failed logs remain intact, and the full-sanitizer status stays not clean. Neither follow-up has a linked implementation in these comments yet.

Performance status remains unchanged: the published warm-cache MSVC comparisons retain +0.826%, +0.902%, and +1.433% confirmed median latency signals. Their necessity is unproven, so query acceptance is not cleared. Future batch measurements should compare fully checked equivalent queries with the current correct native path, and report the application-reference projection/materialization boundary separately. No new timing, source revision, or acceptance pass is claimed here.

中文说明: 同意保持当前 PR 的正确性范围,将对齐数据批量解码、掩码合并和选择后物化作为后续实现。源码确实显示 FIELD 过滤绕过共享多列/批量快路径,但这不能直接解释每个小幅旧新延迟差异,也不能预先证明后续提速。当前任意行回调没有引用列清单或表达式树,后续需要明确接口或兼容回退,并保留 NULL 共享时间轴、类型/字节与借用期、过滤后 OFFSET/LIMIT、错误终止等行为。已核对既有冻结测试 XML,没有重新执行测试,也没有验证尚未提交的批量实现。sanitizer 两项按维护者意见分别跟踪,完整检查仍未通过。暖缓存 MSVC 的 +0.826%、+0.902%、+1.433% 查询信号和必要性未证明的结论保留,性能门槛仍未通过;没有新增计时或生产代码修改。

Frozen bilingual report and limitations

AI disclosure: This source/test-evidence assessment, tracking and reply were prepared by OpenAI Codex. The proposed follow-up implementation is the maintainer's plan and has not been validated here.

AI 使用说明: 本源码与既有测试证据核对、跟踪及回复由 OpenAI Codex 完成;后续实现为维护者提出的计划,本回复未对其作已实现或已通过验证的声明。

Add AlignedFieldFilter (a columnar predicate tree) and SelectionMask
(a row bitmask), and thread them through the aligned chunk/block readers.

Single-threaded flow: page statistics (may_match/all_match) prune pages
that cannot match or mark pages that fully match; only filter-referenced
columns are decoded first; output columns are decoded lazily and only for
pages with at least one matching row; the predicate is evaluated once per
page into a bitmask, and the scatter loop only tests bits.

NOT is pushed down to leaf comparisons so NULL rows keep SQL three-valued
semantics, get_data_type_size is no longer called on string columns, and
query() error paths free the field filter.
@ColinLeeo

ColinLeeo commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

This PR now also includes a columnar/batch FIELD-predicate path for aligned table queries (commit f5d1243a4):

  1. Page-statistics pruning — AlignedFieldFilter::may_match/all_match classify each page as SKIP / FULL_PASS / BOUNDARY before decoding, so pages that cannot match are never decoded or row-evaluated.
  2. Filter-column-first, lazy output decode — only predicate-referenced columns are decoded first; output-only columns are decoded only when a page has at least one matching row. For low-selectivity predicates, most pages never decode their output columns.
  3. Once-per-page bitmask evaluation — the predicate tree is evaluated into a SelectionMask per page (a type-specialized loop, no per-row virtual callback and no per-row TableFieldRow construction); the scatter loop then only tests bits.

@ColinLeeo

ColinLeeo commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Updated: committed mask bulk copy and C++/Java selectivity benchmark

Measured on October 22, 2026, after committing and pushing 11bfc35c99f8e845939a6753f004ecee38c7a2ec to PR #1026.

Exact source and implementation

  • C++ revision: 11bfc35c99f8e845939a6753f004ecee38c7a2ec, committed and pushed. No uncommitted source patch is needed. The implementation bulk-copies contiguous selected mask runs for dense fixed-width columns, retaining NULL/string fallbacks and correct OFFSET/LIMIT, duplicate projection, extra time-column and source-block cursor behavior. The commit also fixes page-plan continuation, NaN statistics pruning and missing-column NULL evaluation.
  • Java sources are unchanged from f5d1243a44fd8a6adddb24d23e8718ab8ccd0fed and identical in the final C++ revision. Built as 2.5.1-SNAPSHOT, using Corretto OpenJDK 17.0.13 (-Xms1g -Xmx1g).
  • C++ library: Release (-O3 / -flto), Apple Clang 17.0.0; unchanged benchmark driver -O3. Source-tree IDs, binary hashes, dataset hashes and build metadata are in protocol.json.
  • C++ uses native structured AlignedFieldFilter. Java uses the public TableQueryExecutor / TsBlockReader and applies the equivalent predicate to returned blocks. The current Java table API does not expose a directly usable equivalent structured FIELD-predicate contract. This compares end-to-end filtered-query usage against a Java scan/filter reference, not identical internal predicate pushdown plans, isolated decoding speed, or language performance in general.

Data and queries

  • One host: Apple M5 Pro, 24 GiB RAM, macOS 26.6.2.
  • 2,000,000 aligned rows, one device, eight INT64 FIELDs (v0…v7), no NULLs.
  • Timestamp r; v0 = (r * 1543 + 17) % 10000; vc = r * 8 + c for c = 1..7.
  • Predicate v0 < percentage * 100 gives exact accepted fractions 0 / 1 / 10 / 50 / 100%. Matches are distributed across pages, not a contiguous time prefix.
  • Projection widths 1 / 4 / 8: v0 through v(width-1).
  • PLAIN values, TS_2DIFF time, UNCOMPRESSED; explicit flush every 100,000 rows; page limits 10,000 points / 128 KiB.
  • Identical deterministic logical data generated independently by the C++ and Java writers. Both readers query each exact same file; these files are logically equivalent, not byte-identical. The input files are the frozen files from the earlier report; their writer implementations did not change in this commit. File sizes and SHA-256 values are recorded.

Measurement and validation

30 scenarios × 2 implementations × 5 fresh processes × 5 timed queries = 1,500 new timed samples. None of the previous working-tree or paired-experiment timings are merged into this matrix.

Each process first executes one discarded exhaustive timestamp/order/row-count/projected-value validation query and two further discarded warm-up queries. Every timed query reopens its reader and verifies exact row count plus timestamp and all projected-value checksums. All scenarios pass; checksums match across readers and writer files.

Timing includes file open, metadata/query construction, scan/filter, consumption of selected timestamps and every projected value, and result/reader close. JVM startup, file generation, exhaustive validation and warm-up are excluded. Both use 1,024-row output blocks. The statistic is the median of five process medians, each over five timed queries. Ratio = Java reference median / C++ median; >1 means C++ is faster.

Cases are deterministically shuffled and implementation order alternated; measured processes do not overlap. This is a warm-OS-cache microbenchmark, not a cold-cache or tail-latency result.

C++-written TsFile

Projected FIELDs Accepted C++ (ms) Java reference (ms) Java / C++
1 0% 0.183 73.545 400.97×
1 1% 16.863 84.912 5.04×
1 10% 18.236 76.964 4.22×
1 50% 27.133 73.072 2.69×
1 100% 16.105 79.356 4.93×
4 0% 0.563 136.004 241.70×
4 1% 21.591 132.350 6.13×
4 10% 26.517 127.503 4.81×
4 50% 48.653 131.910 2.71×
4 100% 32.852 141.700 4.31×
8 0% 1.062 214.316 201.81×
8 1% 35.631 222.408 6.24×
8 10% 44.160 217.576 4.93×
8 50% 84.512 224.702 2.66×
8 100% 60.109 242.057 4.03×

Java-written TsFile

Projected FIELDs Accepted C++ (ms) Java reference (ms) Java / C++
1 0% 0.182 75.372 414.80×
1 1% 16.123 70.138 4.35×
1 10% 17.657 72.180 4.09×
1 50% 27.381 73.661 2.69×
1 100% 15.699 73.566 4.69×
4 0% 0.506 139.713 276.32×
4 1% 22.298 143.282 6.43×
4 10% 26.468 127.137 4.80×
4 50% 48.362 144.901 3.00×
4 100% 32.562 136.728 4.20×
8 0% 1.129 211.059 186.94×
8 1% 36.364 224.075 6.16×
8 10% 44.208 219.684 4.97×
8 50% 82.433 227.907 2.76×
8 100% 61.416 226.072 3.68×

For non-empty results, C++ was 2.66×–6.43× faster than this Java reference in this matrix. For 8 projected FIELDs / 100% accepted, C++ vs Java was 60.109 vs 242.057 ms (4.03×) on the C++-written file, and 61.416 vs 226.072 ms (3.68×) on the Java-written file.

The larger 0%-match ratios reflect C++ page-statistics pruning versus the Java reference scanning data before rejecting rows. They are not isolated decoding-speed ratios. This is dense INT64 data and this predicate/projection matrix on one host, not a universal performance guarantee or a before/after regression-acceptance result.

Separate paired bulk-copy experiment

A separate earlier paired experiment used the same checked benchmark and C++-written file to compare the pre-bulk correctness-fixed library and this exact final library. Three fresh process pairs per case, five timed queries after exhaustive validation and two warm-ups; order was balanced. It found 8 FIELDs / 100% accepted: 142.848 → 56.521 ms (2.53×) and 50% accepted: 122.115 → 77.474 ms (1.58×). This supporting experiment has 465 samples including its Java reference check and is retained separately in bulk-copy-paired-results.json; its samples are not merged with the new 1,500-sample matrix.

Correctness and formatting

The final implementation adds 48 tests relative to f5d1243a4 (38 correctness regressions plus 10 mask bulk-copy tests). Full C++ suite: 1,339 run, 1,336 passed, 3 skipped; 10 additional tests disabled. Related 180 tests repeated ten times, all passed. Spotless, clang-format and git diff --check passed. Full-suite output and repeated-run summary are in validation.json. No new clean sanitizer status is claimed.

Reproducible evidence and update history

Full report, exact protocol, source drivers and reproduction instructions · 1,500 new raw timing samples · All 300 measurement-process outputs · Validation output.

This edits the existing performance comment to reflect the pushed implementation, rather than appending new results to stale tables. The original pre-bulk working-tree evidence remains unchanged for traceability; no old timing samples are merged into the new matrix. The supporting paired bulk-copy experiment is separately retained in the new evidence.

AI disclosure: Implementation, tests, benchmark drivers, execution, result verification, analysis and this comment were prepared by OpenAI Codex.

Copy selected bitmap runs with dense fixed-width bulk appends while preserving NULL/string fallbacks, duplicate projections, post-filter row ranges, and source-block cursor ownership.

Keep page-plan decode mode stable across TsBlocks, retain NaN rows when floating statistics cannot prove a match, and evaluate missing aligned columns as NULL. Add focused and integration regression coverage.
@jixuan1989

jixuan1989 commented Oct 22, 2026 •

Copy link
Copy Markdown
Member Author

Fresh exact cf4c9aa validation is complete; diagnostic evidence only, source gate remains uncleared.

Query acceptance UNCLEARED; source/write/CI acceptance also uncleared. Fresh cf4c9aa rounds contain 82 cases/98 comparisons with a repeated increase and 5 confirmation-only comparisons. Largest repeated increase: layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]. Largest against preceding measured eee: layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]. No numerical allowance was supplied; necessary/minimized extra cost or causal attribution is unproven. All samples and adverse results remain. No source push, merge, new issue or Copilot request. This is diagnostic publication only.

查询验收尚未通过;源码、写入和CI验收也未通过。 cf4c9aac全新轮次有82场景/98项比较重复上升,另有5项仅确认轮信号。最大重复上升:layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]。相对前次已测eee的最大值:layout76/layout0-w1-p100-b1024 fixed/public 2.7010312 (+170.103%), pointwise95 [2.6883321,2.7058418]。用户未给数值容差;额外成本的必要性、最小性和因果归因均未证明。所有样本及不利结果保留。未推送源码、合并、创建issue或请求Copilot,本次仅发布诊断。

Windows1,163full passes; Linux245focused/317reader passes but unchanged LZOkay full-sanitizer failure; Java139affected tests,1,896standard cross-reads and55,296numeric masks pass in their stated scope. New layout76 expands sparse non-aligned/mixed coverage, but native Java Table interior NULL fails across all3writers; the disclosed raw-series oracle does not clear that scope. Parent shared-library negative was not reproduced. Since073 C++ lines91.01%,branches67.65%; wholePR85.11%/65.26%,module78.14%/45.86%. No fresh writer acceptance;2048header error6 persists. CodeQL failure cause unresolved. All initial, confirmation and setup failures retained; no added measurement rounds.

精确cf4c9aac全新验证完成,仅发布诊断,源码门槛未通过。Windows全量1163通过,Linux重点245/reader317通过但全量仍在未改LZOkay失败;Java受影响139项、标准1896次交叉读取及55296个数字mask在所述范围通过。新增76布局覆盖稀疏非对齐/混合,但三种writer产物的Java原生Table内部NULL均失败,显式逐序列参考不能清除该范围;旧共享库负控未复现。相对073 C++行91.01%/分支67.65%,整PR85.11%/65.26%,模块78.14%/45.86%。写入无新验收、2048header仍error6,CodeQL根因未决;保留全部初始/确认和准备失败,未追加轮次。

Report · complete raw diagnostics and final review records · previous detailed reply. Final independent reviews permit diagnostics only. No production source push or upstream merge.

AI disclosure: The implementation, tests and description in this pull request were written by an AI coding agent (OpenAI Codex), which also performed the reported local validation.

AI 使用说明: 本 PR 的代码修改、测试和正文由 AI 编程代理(OpenAI Codex)编写,文中报告的本地验证也由该 AI 代理执行。

Negating an ordered FLOAT/DOUBLE comparison flipped the operator
(NOT (x < 5) -> x >= 5). NaN makes every ordered comparison false, so the
negation must be true for NaN rows while the flipped form is false and
dropped them; when the literal itself was NaN, every non-NULL row was lost.
Ordered floating-point leaves now keep the operator and record the negation,
and evaluation computes !(value <op> literal) for non-NULL rows. Page
statistics no longer prove a full-page match or a skip for such leaves.

A composite node missing an operand could be classified as a full-page match
through its remaining operand. Page statistics then skipped evaluation, so a
malformed OR silently returned rows (and did not terminate in batch mode).
all_match() now requires both operands, and build_page_plan() rejects a
structurally invalid filter with E_INVALID_ARG before classifying pages.

AlignedChunkReader is placement-new'd and torn down through destroy() without
running its destructor, so the page-plan filter state added for FIELD
filtering leaked its backing buffers. destroy() now releases
filter_column_mask_ and the current page mask.

Tests cover NOT over all six operators and five literals (including NaN and
both infinities) for FLOAT and DOUBLE, with NULL rows from a device that
lacks the float columns; and a malformed OR that was previously accepted.
Clear borrowed contexts before deletion, make close idempotent, and publish multi-index ownership before fallible lookup. Keep multi-column read failures terminal until reset, validate page and value-view bounds, and reject residual NOT nodes instead of inverting nullable masks.

Add real-file fault injection, malformed page and buffer tests, repeated-close and partial-lookup coverage. Validate reset recovery across output blocks and failures after a result prefix.
Give each test case and invocation a separate TsFile path so CTest workers cannot truncate or remove one another's input. Reproduce the shared-name failure with four parallel workers, then verify 100 repetitions of every case after isolation.
Use each strided task's index for its decoder and compressor slot instead of header-inline worker TLS, which can differ between a caller executable and the reader DLL. This prevents concurrent tasks from sharing slot zero and corrupting timestamps. Complete partially initialized pools when retrying after reset.

Validate with the same hidden-symbol caller executable: the old code fails in the first iteration; the fix passes 1000 iterations (2000 fault/recovery tests). Full four-worker CTest also passes with no failures.
@ColinLeeo
ColinLeeo merged commit 0d42092 into apache:develop Oct 27, 2026
52 of 53 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][C++] Aligned table queries ignore the supplied field_filter

3 participants