Repository navigation
feat: paginate TsFile metadata details and bound scan memory - #50
Merged
Merged
Conversation
解析含 87,488 个 RowGroup 的 TsFile 时,后端把整文件的每个 chunk 逐条建成
DTO(约 200 万条)塞进 /api/meta 响应,堆被打爆;OOM 又打死了 Tomcat 的
Poller 线程,Acceptor 仍在 accept,导致服务"端口通但永无响应"假死。
- /api/meta/{fileId} 只返回摘要,rowGroups/chunks 恒为空数组(字段保留以兼容契约)
- 新增 /api/meta/{fileId}/row-groups 与 /chunks 分页端点,
offset/limit 归一化,limit 上限 200,hasMore 用 long 计算防溢出
- 摘要与分页都改为按设备增量读取,不再全量物化 TimeseriesMetadata
- 摘要缓存只存摘要;新增设备索引缓存,按 mtime+size 校验文件版本
- tsfile 依赖 2.3.0 -> 2.5.0
- 应用日志级别 DEBUG -> INFO(原包名写错导致线上一直是 DEBUG)
Tests: 77 run, 0 failures, 13 skipped(既有 @EnabledIf 集成用例因样本目录缺失而跳过)
配合后端分页契约,明细不再一次性拉全量。 - 新增 usePagedList:追加式分页状态机(hasMore 闸门、切文件 generation 防旧响应回写、空页即停防无限重试、失败停止自动加载并可重试) - RowGroupsTable 改用 el-table-v2 虚拟行,滚动到底自动加载下一页; 按设备搜索只过滤已加载行,计数如实显示"已加载 / 总数" - 新增 ChunksTable(此前无此组件),同款虚拟滚动与分页 - 元数据页接入两个新表格,chunks 明细首次可见 - TableFilterPanel/TreeFilterPanel 的设备选项改由分页 RowGroup 有界扫描, 扫描不完整时标出 (已扫 / 总数),不再依赖恒空的 metadata.rowGroups - 显式引入 table-v2 样式:ElTableV2/ElAutoResizer 在 script setup 中手动 import,unplugin-vue-components 不会为其注入 CSS,缺失时表头会落到数据 下方、行不裁剪互相重叠 验证:pnpm build 通过;Playwright 实测 87,488 RowGroup / 1,838,571 Chunk 的文件,两表滚动加载正常、表头正确、行不重叠、零 console 错误
TsFileHealthChecker 为文件里每个 chunk 建一条 HashMap<Long, IChunkMetadata> 记录,1,838,571 个 chunk 的文件要上百万个元数据对象,扫描时撑爆堆。 - 新增 ChunkMetadataIndex:四组平行基本类型数组(offset/measurementId/ startTime/endTime),单 chunk 28B,183 万 chunk 约 49MiB(原结构约 350MiB 起) - 构建时逐设备读取并周期性 clearCachedDeviceMetadata,不全量物化 TimeseriesMetadata - 查找用单向前进游标(遍历本身从文件头顺序前进,O(n)),倒退请求回退二分 - quickCheck 只需"有没有 chunk",改为命中第一个即返回 - 新增 tsfile.scan.max-chunks(默认 5,000,000):超阈值不报错不 OOM,降级为 结构与元数据检查并在扫描结果里给出 DATA_VALIDATION_SKIPPED(WARNING 级) 实测:729MB / 183 万 chunk 的文件在 -Xmx512m 的一次性实例下 30s 扫完 (HEALTHY,无 OOM);闸门设为 1000 时 138ms 降级返回 WARNING。 Tests: 322 run, 0 failures, 17 skipped
- 补 DATA_VALIDATION_SKIPPED 的 i18n(zh/en)与 TS 联合类型,并同步 Record<ScanErrorType, number> 分布类型;顺带补上同样缺失的 DATA_READ_ERROR 文案(后端 6 个枚举值现已与三处映射一一对应) - TreeFilterPanel 不再用 watch(selectedDevice) 清空测点:loadMetadata 会把 selectedDevice 程序化设为第一个设备,watcher 异步回调落在 chart 模式默认 测点写入之后,导致图表有数据但测点框为空。改挂 el-select-v2 的 @change, 只有用户主动切换才重置 - ChunksTable 12 列压到 11 列(编码方式+压缩方式合并为「编码/压缩」), 重排列宽使总和 1193 ≤ 容器 1196,消除 el-table-v2 的等比压缩截断; 偏移量/最小值/最大值等数值列不再出现省略号 验证:Playwright 实测线上——测点首次加载保留 8 条、手动切设备后回到占位符; Chunk 表偏移量 344172277、编码/压缩 TS_2DIFF / LZ4、最小值/最大值均完整
ITsFileReader.getAllTableSchema() 在树模型文件上不解引用空的 tableSchemaMap, 直接抛 NullPointerException;而所有调用方都以"空列表"为前提来判断是否回退到 TsFileSequenceReader 的树模型读取路径,于是整条数据路径 500。 tsfile 2.3.0 与升级后的 2.5.0 均能复现,非本次依赖升级引入。 - 新增 TableSchemaSupport.readAll:把 null 与 NPE 归一化为空列表 (TsFileParser 早有同款私有实现,提取为共享工具供其余调用点复用) - TsFileDataReader 7 处、TableService 2 处裸调用改走该工具 实测:树模型文件 /api/data/preview 与 /api/data/query 由 500 变 200, 数据预览页渲染 100 行、无错误横幅。 已知未修(既有缺陷,属测点过滤语义而非崩溃):树模型文件带 devices/ measurements 过滤时仍返回空——DataService 是读后过滤,而树模型行的测点键 为全路径,与请求里的短名匹配不上。
树模型读路径按存储顺序逐设备读取,用全局 limit 收集满即 break,而 deviceMeasurements 又是无序 HashMap。于是返回窗口永远被迭代顺序里的第一台 设备吃满,任何指向其它设备的过滤都匹配不到——设备筛选、测点筛选、以及依赖 它们的图表模式全部返回空(既有缺陷,2.3.0 上同样存在)。 - readDataByTimeRange 新增带 devices/measurements 的重载,由 DataService 的 数据预览与图表查询两处调用点传入 FilterConditions/请求里的条件 - readTreeModelData 用这些条件先行筛选要读的设备(按设备名精确匹配)与测点, 并改用 LinkedHashMap 让分页顺序稳定可预测 - 不带过滤时行为不变 实测(真实 25 设备 / 922 chunk 树模型文件,本地实例 + 线上各验一遍): devices=[ac_sig.a2158] 0 行 -> 3 行 measurements=[WorkState] 0 行 -> 3 行 devices+measurements 0 行 -> 3 行 图表模式:由"加载失败"/"暂无图表"变为渲染出序列 (图例 root.a.nayong.ac_sig.a2158 / WorkState,数据点 10,000) Tests: 323 run, 0 failures, 17 skipped(新增 1 个测点下推用例;现有写工具只能 产出单表带 device_name 字段的伪多设备样本,构造不出真多设备树模型文件,故设备 下推以真实样本端到端验证,测试里已注明)
树模型的层级是「设备 → 测点」,但测点下拉取的是 metadata.measurements —— 整个
文件的测点并集。以真实样本为例:文件级并集 122 个测点,而 ac_sig.a2158 只带
WorkState 一个。于是用户可以选到本设备并不存在的测点(后端过滤下推后正确返回
0 行,但 UI 无任何解释),图表模式也会把不属于该设备的测点预选进来。
- 后端新增 GET /api/meta/{fileId}/devices/{device}/measurements:
经已有的设备索引缓存解析设备名,只读该设备的元数据(readChunkMetadataInDevice),
跳过 VECTOR 时间列,按名称排序返回;设备不存在时返回空列表
- 前端测点下拉与图表默认序列改为跟随所选设备;级联请求失败时退回文件级并集,
不阻塞筛选面板
- 设备切换仍会清空已选测点(用户主动切换才触发,保持前次的修复意图)
实测(真实 25 设备树模型文件):
ac_sig.a2158 -> 1 个测点(WorkState)
cabinet.a2025 -> 20 个测点
不存在的设备 -> 0 个
图表模式首次加载:默认序列由 8 个(文件级)变为 1 个(该设备真实拥有)
Tests: 325 run, 0 failures, 17 skipped(新增 2 个端点用例)
表模型的层级是「表 → 设备(TAG 组合)→ 字段」,但设备下拉取的是全文件设备
(分页扫描 RowGroup,最多 800 条),与所选表无关;字段层则早已按表过滤。
现成的 /api/tables/{fileId}/devices?tableName= 不能直接用:它靠一次全表行查询
发现设备,实测某表 87,488 台设备时返回 16MB / 16.9s。
- 新增 GET /api/meta/{fileId}/devices?tableName=&query=&offset=&limit=:
从已缓存的设备索引推导(设备标识本身即 tableName.tag1.tag2),支持按表过滤、
大小写不敏感的子串搜索与分页;只物化请求的那一页,total 为匹配总数
- 前端设备下拉改为按所选表加载首页(50 条)并在输入时远程搜索(去抖 250ms),
切换表时清空已选设备并重新加载;已选设备的标签单独缓存,换结果集后不丢
实测:某表 87,488 台设备 —— 首页 8.4KB / 热态 13-20ms(冷态首调含设备索引构建,
与元数据摘要共用同一次构建);不存在的表名返回 0 条;搜索 "3540" 命中 464 条。
Tests: 327 run, 0 failures, 17 skipped(新增 2 个端点用例)
el-select-v2 的下拉里有两层可滚容器:外层 .el-select-dropdown__wrap (overflow:auto + max-height:274px)与内层虚拟列表 .el-vl__window。虚拟列表 高度超过外层上限时(下拉向上弹出、可用高度被压缩时最常见)两层同时滚动, 用户会看到两条滚动条。 - .v2-options-compact(本就用在该场景的标记类,此前没有任何样式)的作用域内 把外层改为 overflow:hidden,滚动只留给内层虚拟列表 - 两层都加 overscroll-behavior:contain,滚到底不再把滚动传给页面 实测:打开「字段」下拉后 popper 内可滚容器由两层变一层,连续滚轮滚动时 window.scrollY 恒为 0(不再穿透)。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Per-chunk data is materialised even though it is never needed, and the tree model read path exhausts its row window on the first device.
GET /api/meta/{fileId}returned every row group and every chunk of the file. For a real 87,488-row-group file that is ~1.8M chunk DTOs in a single JSON response. Parsing it exhausted the 2 GB heap; the resultingOutOfMemoryErrorkilled Tomcat's poller thread while the acceptor kept accepting connections, so the service stayed "up" for 7 days answering every request with zero bytes.TsFileHealthCheckerbuilt aHashMap<Long, IChunkMetadata>entry for every chunk, so scanning the same file had the same failure mode.ITsFileReader.getAllTableSchema()dereferences a null table schema map on tree model files and throwsNullPointerExceptioninstead of returning an empty list, so data preview and chart answered 500 for those files.Changes
Metadata
GET /api/meta/{fileId}returns a summary only;rowGroups/chunksremain in the contract but are always empty arrays.GET /api/meta/{fileId}/row-groupsand/chunkswithoffset/limit(default 50, max 200), returning{items,total,offset,limit,hasMore}.getAllTimeseriesMetadata(true)materialisation on the metadata path.mtime+size.el-table-v2virtualised rows with load-on-scroll for both tables (the chunk table did not exist before). Device options come from a bounded paged scan and are labelled "(loaded / total)" rather than pretending to be complete.el-table-v2styles are imported explicitly — components imported in<script setup>do not get CSS injected byunplugin-vue-components; without it the header renders below the body and rows overlap.Scan
ChunkMetadataIndex: parallel primitive arrays (offset / measurement id / start / end), 28 B per chunk ≈ 49 MiB for 1.83M chunks versus roughly 350 MiB+ for the object map. Forward-only cursor with binary-search fallback.quickCheckstops at the first chunk instead of building the whole index.tsfile.scan.max-chunks(default 5,000,000): above the cap the scan degrades to structure + metadata checks and reportsDATA_VALIDATION_SKIPPED(WARNING) instead of OOMing.Tree model
TableSchemaSupport.readAll(...)normalisesnull/NPE fromgetAllTableSchema()to an empty list; 9 call sites inTsFileDataReaderandTableServiceuse it.GET /api/meta/{fileId}/devices/{device}/measurementsreads a single device's metadata through the cached device index, and the picker plus the chart default series now follow the selected device, falling back to the file-wide union if the lookup fails./api/tables/{fileId}/devices?tableName=cannot be used for it — it discovers devices by running a full table query, measured at 16 MB / 16.9 s for a table with 87,488 devices. NewGET /api/meta/{fileId}/devices?tableName=&query=&offset=&limit=derives devices from the cached device index instead (a device identifier already istableName.tag1.tag2), so it pages and searches in milliseconds; the picker loads the first page and searches remotely.Dependency / config
logging.levelused a package name that does not exist, so the configured INFO never took effect).Verification
mvn test: 323 run, 0 failures, 17 skipped.pnpm buildpasses (vue-tsc included).-Xmx512m: scan completes in 30 s,HEALTHY, no OOM. With--tsfile.scan.max-chunks=1000it degrades in 138 ms toWARNING+DATA_VALIDATION_SKIPPED.devices=[ac_sig.a2158],measurements=[WorkState]and both combined went from 0 rows to 3 rows; chart mode now renders a series (root.a.nayong.ac_sig.a2158 / WorkState).ac_sig.a2158→ 1 measurement,cabinet.a2025→ 20, unknown device → 0. In the UI the chart default series for the first device dropped from 8 file-wide picks to the single measurement that device actually carries.3540→ 464 matches, and the picker's options follow the search.Note
The device filter in the tree model path is verified end-to-end against a real multi-device file rather than a unit test: the existing test writer can only produce a single-table file with a
device_namefield, which cannot express a genuine multi-device tree model file. The measurement pushdown does have a unit test.