Repository navigation
Conversation
Contributor
|
LGTB, @lhoestq Would you mind taking a look at this PR when you get a chance? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streaming a remote TsFile currently validates its header through
xopen, then passes the URL string toTsFileReader, whose path-based constructor opens local files. Keep the fsspec seekable binary source alive and pass it to the file-object reader introduced in TsFile 2.5.0. Local files continue using the native path reader.Manage the reader and source together with context managers, including the multi-file folding path, so readers close before their sources on completion, early iterator closure, and errors. Document remote streaming and its byte-range requirement.
Validation:
pytest tests/packaged_modules/test_tsfile.py tests/packaged_modules/test_tsfile_streaming.py -q: 54 passed on each of Python 3.10.22 / PyArrow 24.0.0, Python 3.12.1 / PyArrow 26.0.0, and Python 3.14.5 / PyArrow 26.0.0, using published TsFile 2.5.0 wheels.Rangerequests /206responses, equality with local results, and source closure after complete and early iteration. Error cases cover invalid magic and reader construction failure./first-rows?dataset=tsfile/lotsa_data&config=default&split=trainrequest went through the real queue and worker. Worker logs confirm the Parquet path fell back to streaming and remote TsFile reads returned HTTP 206. The API returned 10 preview rows (truncated to the configured response limit); cache reuse,/splits, and browser CORS preflight passed.The Viewer smoke test exercises its streaming read helper. It does not establish that the public website has deployed this code or recomputed its cached preview. Hub format-tag registration is separate.
Refs: apache/tsfile#930, #8499.
Related CI cleanup: #8806 adds the stable SDK to the shared test requirements so these tests also run in the Python 3.10 jobs. This PR is independently based on main.