Repository navigation
fix: export all filtered rows instead of the current preview page - #51
Open
CritasWang wants to merge 1 commit into
Open
CritasWang wants to merge 1 commit into
CritasWang wants to merge 1 commit into
Conversation
exportCSV()/exportJSON() serialized dataRows, which holds only the current preview page, so the exported subset changed with pagination — while the user guide promises everything matching the filters. Paging the preview endpoint from the browser is not a fix either: with post-read filters the backend caps a single read at query.max-result-size (10,000), so the export still truncated there (measured 10,000 rows on a 16,031-row file). - New POST /api/data/export?format=csv|json streams the whole filtered result set: the server pages through the file itself (2,000 rows per page, offset pushed down), applies exactly the filters the preview applies, and flushes each page — the file never has to fit in memory and max-result-size no longer truncates. - CSV follows RFC 4180 (commas, quotes, line breaks); columns are the file's column union in lexicographic order; timestamps are ISO-8601 UTC with milliseconds, matching what the browser used to produce. - JSON streams one row object per line, same shape as the preview rows. - Frontend requests the blob from the new endpoint and keeps the export buttons disabled while it runs. - DataPreviewRequest now accepts advancedConditions: the frontend has been sending it, but the backend silently dropped it (preview was affected too). Verified on a 16,031-row tree-model file: preview page shows 100 rows, export yields 16,031 rows for both CSV and JSON in ~100 ms; limit/offset in the request are ignored; bad format -> 400, path outside allowed dirs -> 403, missing file inside the dir -> 404. Tests: 297 run, 0 failures, 17 skipped (11 new).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #45
Problem
exportCSV()/exportJSON()serializeddataRows, which only holds the current preview page, so the exported subset changed whenever the user paged. The user guide promises the export covers every record matching the filters.Paging the preview endpoint from the browser is not enough: with post-read filters the backend caps a single read at
query.max-result-size(10,000), so the export still truncated there — measured 10,000 rows on a 16,031-row file.Changes
POST /api/data/export?format=csv|jsonstreams the whole filtered result set. The server pages through the file itself (2,000 rows per page, offset pushed down), applies exactly the same filters as the preview, and flushes after every page, so the file never has to fit in memory andmax-result-sizeno longer truncates.limit/offsetin the request body are ignored.DataPreviewRequestnow acceptsadvancedConditions: the frontend has been sending it, but the backend silently dropped it, so advanced filters never applied to the preview either.Verification
Against a 16,031-row tree-model file in a local instance, driven through the real UI with Playwright:
limit: 1, offset: 5in the request body still exports 16,031 rows.format=xlsx→ 400 with a JSON error, path outside the allowed directories → 403, missing file inside them → 404 (resolved before any bytes are written, so no truncated file).mvn test: 297 run, 0 failures, 17 skipped (11 new).pnpm buildpasses.Not addressed here
query.timeout-seconds; a long export holds the request thread. Capping it belongs at the gateway.TsFileDataReader.streamData(...)can make this a single pass in a follow-up.docs/API.mdanddocs/MANUAL_TEST_CASES.zh-CN.md(TC-C005) still describe the old page-scoped behaviour.