Skip to content

Support remote streaming reads for TsFile - #8807

Open
ColinLeeo wants to merge 1 commit into
huggingface:mainfrom
ColinLeeo:fix/tsfile-remote-streaming
Open

ColinLeeo wants to merge 1 commit into
huggingface:mainfrom
ColinLeeo:fix/tsfile-remote-streaming

Conversation

@ColinLeeo

@ColinLeeo ColinLeeo commented Oct 15, 2026 •

Copy link
Copy Markdown

Streaming a remote TsFile currently validates its header through xopen, then passes the URL string to TsFileReader, whose path-based constructor opens local files. Keep the fsspec seekable binary source alive and pass it to the file-object reader introduced in TsFile 2.5.0. Local files continue using the native path reader.

Manage the reader and source together with context managers, including the multi-file folding path, so readers close before their sources on completion, early iterator closure, and errors. Document remote streaming and its byte-range requirement.

Validation:

  • The new HTTP streaming regression cases fail on the unmodified builder because a remote URL reaches the local-path reader, and pass with this change.
  • pytest tests/packaged_modules/test_tsfile.py tests/packaged_modules/test_tsfile_streaming.py -q: 54 passed on each of Python 3.10.22 / PyArrow 24.0.0, Python 3.12.1 / PyArrow 26.0.0, and Python 3.14.5 / PyArrow 26.0.0, using published TsFile 2.5.0 wheels.
  • A real local HTTP server verifies partial Range requests / 206 responses, equality with local results, and source closure after complete and early iteration. Error cases cover invalid magic and reader construction failure.
  • Full Dataset Viewer Compose integration on Linux ARM64 / Python 3.14.5 / PyArrow 24.0.0 / TsFile 2.5.0: all seven application images built, and MongoDB, Nginx, and all seven application services started. All eight HTTP healthchecks returned 200. The local integration pinned this datasets commit and huggingface-hub 1.33.0; Debian package downloads used HTTPS to avoid HTTP mirror failures.
  • A cold /first-rows?dataset=tsfile/lotsa_data&config=default&split=train request went through the real queue and worker. Worker logs confirm the Parquet path fell back to streaming and remote TsFile reads returned HTTP 206. The API returned 10 preview rows (truncated to the configured response limit); cache reuse, /splits, and browser CORS preflight passed.
  • In the complete Linux container dependency environment, all 54 TsFile regressions passed. The API, rows, and search application test modules also passed (10 + 8 + 9 tests), for 81 passing targeted tests.
  • Anonymous Parquet publication reached the Hub branch-creation API and returned HTTP 401. Parquet-dependent pagination/search did not pass in this public read-only run; these need an authorized publishing test. This does not affect the successful streaming preview path.
  • Ruff lint and format checks passed.

The Viewer smoke test exercises its streaming read helper. It does not establish that the public website has deployed this code or recomputed its cached preview. Hub format-tag registration is separate.

Refs: apache/tsfile#930, #8499.

Related CI cleanup: #8806 adds the stable SDK to the shared test requirements so these tests also run in the Python 3.10 jobs. This PR is independently based on main.

@JackieTien97

Copy link
Copy Markdown
Contributor

LGTB, @lhoestq Would you mind taking a look at this PR when you get a chance?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants