Skip to content

fix(tsfile): guard optional Apache TsFile SDK import - #8514

Closed
AxelNoun wants to merge 1 commit into
huggingface:mainfrom
AxelNoun:fix/tsfile-optional-extra-8499
Closed

AxelNoun wants to merge 1 commit into
huggingface:mainfrom
AxelNoun:fix/tsfile-optional-extra-8499

Conversation

@AxelNoun

@AxelNoun AxelNoun commented Aug 24, 2026 •

Copy link
Copy Markdown

Summary

  • Guard every tsfile SDK import behind _require_tsfile() so missing extras raise ImportError: ... pip install 'datasets[tsfile]' instead of a bare ModuleNotFoundError.
  • Add a tsfile extra (tsfile>=2.3.0), install it in the tests extra on Python 3.14 (where it does not conflict with pyarrow>=21), and install datasets[tsfile] in the py3.14 CI job.
  • Document the extra in loading.mdx, tsfile_load.mdx, about_dataset_load.mdx, and the README.
  • Add tests/packaged_modules/test_tsfile_optional.py for the missing-SDK path. Existing round-trip tests remain behind pytest.importorskip(tsfile).

This is the datasets-side slice of #8499 (see also #8256, #8160, and the overlapping #8307). It does not register a Hub format:tsfile facet: that still needs the SDK on every dataset-viewer worker and Hub taxonomy registration.

Test plan

  • pytest tests/packaged_modules/test_tsfile_optional.py (no tsfile SDK required)
  • CI py3.14 job installs datasets[tsfile] and runs tests/packaged_modules/test_tsfile.py
  • pip install 'datasets[tsfile]' then load_dataset(tsfile, data_files=...)
  • Without the extra, load_dataset(smilegeng/pusht) raises the actionable ImportError

Refs #8499, #8160, #8256, #8307

The TsFile builder imported the third-party `tsfile` SDK inside
`_scan_metadata` with no guard, so users and dataset-viewer workers
saw a bare `ModuleNotFoundError`. That is also why no `format:tsfile`
facet is derived on the Hub (see huggingface#8499).

- raise an actionable ImportError pointing at `datasets[tsfile]`
- declare the `tsfile` optional extra and install it in the py3.14 tests extra
- document the extra in the supported-formats / TsFile loading docs
- add a missing-SDK test (existing round-trip tests stay behind importorskip)

Refs huggingface#8499, huggingface#8160, huggingface#8256, huggingface#8307
@AxelNoun

AxelNoun commented Aug 26, 2026 •

Copy link
Copy Markdown
Author

Note for the record, since this branch is closed but still linked from #8499.

Two things in it did not survive further checking:

  1. The commit message on d11450c says the unguarded SDK import is "also why no
    format:tsfile facet is derived on the Hub". That is wrong. tsfile is not a registered
    value in the Hub's format tag taxonomy at all, which accounts for the empty Format bucket
    on its own, independently of dataset-viewer job status. Shipping the SDK to the workers
    would not make the facet appear.

  2. The tsfile extra as written here only resolves on Python 3.14 (the SDK requires
    pyarrow<20 on 3.10-3.13, datasets requires pyarrow>=21). Because that is an
    environment marker, pip install 'datasets[tsfile]' returns success while installing
    nothing on 3.10-3.13, so an error message pointing at that command sends the user in a
    loop. Any guard needs to be version-aware.

Analysis of the pyarrow pins: apache/tsfile#921 (the upper bounds have since been removed upstream in apache/tsfile#922, which is not yet in a stable release). Format-facet analysis: #8499 (comment)

Edited 2026-10-07: in the last paragraph, a link to a comment that no longer exists was replaced and the status of apache/tsfile#922 was added; the rest is unchanged. The pyarrow figures in point 2 predate #8502 (datasets now requires pyarrow>=24.0.0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant