Skip to content

fix: export all filtered rows instead of the current preview page - #51

Open
CritasWang wants to merge 1 commit into
apache:mainfrom
CritasWang:fix/export-all-filtered-data
Open

CritasWang wants to merge 1 commit into
apache:mainfrom
CritasWang:fix/export-all-filtered-data

Conversation

@CritasWang

Copy link
Copy Markdown
Contributor

Fixes #45

Problem

exportCSV() / exportJSON() serialized dataRows, which only holds the current preview page, so the exported subset changed whenever the user paged. The user guide promises the export covers every record matching the filters.

Paging the preview endpoint from the browser is not enough: with post-read filters the backend caps a single read at query.max-result-size (10,000), so the export still truncated there — measured 10,000 rows on a 16,031-row file.

Changes

  • New POST /api/data/export?format=csv|json streams the whole filtered result set. The server pages through the file itself (2,000 rows per page, offset pushed down), applies exactly the same filters as the preview, and flushes after every page, so the file never has to fit in memory and max-result-size no longer truncates. limit/offset in the request body are ignored.
  • CSV follows RFC 4180 (commas, quotes, line breaks quoted and escaped), columns are the file's column union in lexicographic order, and timestamps are ISO-8601 UTC with milliseconds — the same text the browser used to produce.
  • JSON streams one row object per line, identical in shape to the preview rows.
  • Frontend requests the blob from the new endpoint and keeps the export buttons disabled while it runs.
  • DataPreviewRequest now accepts advancedConditions: the frontend has been sending it, but the backend silently dropped it, so advanced filters never applied to the preview either.

Verification

Against a 16,031-row tree-model file in a local instance, driven through the real UI with Playwright:

before after
rows on the preview page 100 100
CSV data rows 100 16,031
JSON rows 100 16,031
  • ~100 ms for either format including the download; no console errors.
  • limit: 1, offset: 5 in the request body still exports 16,031 rows.
  • format=xlsx → 400 with a JSON error, path outside the allowed directories → 403, missing file inside them → 404 (resolved before any bytes are written, so no truncated file).
  • Peak RSS stayed flat across 200 consecutive exports.

mvn test: 297 run, 0 failures, 17 skipped (11 new). pnpm build passes.

Not addressed here

  • The export does not apply query.timeout-seconds; a long export holds the request thread. Capping it belongs at the gateway.
  • Each page re-scans the file from the start, so a very large export is O(n² / pageSize) in scan work. TsFileDataReader.streamData(...) can make this a single pass in a follow-up.
  • docs/API.md and docs/MANUAL_TEST_CASES.zh-CN.md (TC-C005) still describe the old page-scoped behaviour.

exportCSV()/exportJSON() serialized dataRows, which holds only the current
preview page, so the exported subset changed with pagination — while the user
guide promises everything matching the filters.

Paging the preview endpoint from the browser is not a fix either: with
post-read filters the backend caps a single read at query.max-result-size
(10,000), so the export still truncated there (measured 10,000 rows on a
16,031-row file).

- New POST /api/data/export?format=csv|json streams the whole filtered result
  set: the server pages through the file itself (2,000 rows per page, offset
  pushed down), applies exactly the filters the preview applies, and flushes
  each page — the file never has to fit in memory and max-result-size no
  longer truncates.
- CSV follows RFC 4180 (commas, quotes, line breaks); columns are the file's
  column union in lexicographic order; timestamps are ISO-8601 UTC with
  milliseconds, matching what the browser used to produce.
- JSON streams one row object per line, same shape as the preview rows.
- Frontend requests the blob from the new endpoint and keeps the export
  buttons disabled while it runs.
- DataPreviewRequest now accepts advancedConditions: the frontend has been
  sending it, but the backend silently dropped it (preview was affected too).

Verified on a 16,031-row tree-model file: preview page shows 100 rows, export
yields 16,031 rows for both CSV and JSON in ~100 ms; limit/offset in the
request are ignored; bad format -> 400, path outside allowed dirs -> 403,
missing file inside the dir -> 404.
Tests: 297 run, 0 failures, 17 skipped (11 new).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] CSV export only includes the current preview page instead of all filtered data

1 participant