Skip to content

Python SDK: can pyarrow>=18,<20 on 3.10-3.13 be widened to overlap datasets (pyarrow>=21)? #921

Closed

Description

@AxelNoun

Python SDK: can pyarrow>=18,<20 on 3.10–3.13 be widened?

Status: hypothesis. We do not know why this pin exists; asking whether it can be relaxed.

Measured on 2026-08-25 from https://pypi.org/pypi/tsfile/json (see
pypi_tsfile_matrix.md). Latest SDK: 2.4.0. Pins identical for 2.3.0 / 2.3.1 / 2.4.0.

datasets (since 4.8.5) requires pyarrow>=21. The Apache TsFile Python SDK declares:

Python SDK pyarrow Intersection with datasets
3.9 >=16,<18 n/a (datasets needs ≥ 3.10)
3.10–3.13 >=18,<20 empty → ResolutionImpossible
3.14+ >=22,<24 [22, 24)

Empirical (2026-08-25): pip install --dry-run "datasets==5.0.1" "tsfile==2.4.0" on
Python 3.12 → ResolutionImpossible. Wheels exist for cp39–cp314; the hole is the pin.

The same SDK already accepts pyarrow 22/23 on 3.14, which suggests (does not prove)
that the <20 bound on 3.10–3.13 may be leftover rather than a live ABI/API break.

If that bound can be widened to overlap pyarrow>=21 on 3.10–3.13, then
datasets[tsfile] and dataset-viewer images on those Pythons become co-installable
with no change required in either Hugging Face repository. That is an effect
description, not an attribution of fault.

Question for upstream: is pyarrow>=18,<20 on 3.10–3.13 motivated by a known
incompatibility, or can it be relaxed?

Context: co-install with Hugging Face datasets (requires pyarrow>=21) fails on Python 3.10–3.13 with ResolutionImpossible. Related discussion: huggingface/datasets#8499

Activity

  1. ColinLeeo commented on Aug 26, 2026

    @ColinLeeo
    Contributor

    Thank you for raising this issue.

    The pyarrow>=18,<20 constraint appears to be a historical leftover rather than a current compatibility requirement. We tested TsFile 2.4.0 with PyArrow 18 through 23, including Arrow read/write paths, and confirmed that it works directly with the newer versions.

    As a temporary workaround before the next release, users can install the desired compatible PyArrow version first and then install TsFile without dependency resolution, for example:

    pip install "pyarrow>=21,<24"
    pip install --no-deps "tsfile==2.4.0"

    TsFile 2.5.0 is planned for release soon, and the PyArrow constraint will be relaxed in its published package metadata so that normal installation works without this workaround.

    Thanks again for identifying and reporting the problem. We will fix it promptly.

  2. AxelNoun commented on Oct 7, 2026

    @AxelNoun
    Author

    Follow-up on the release status, checked on 2026-10-07.

    • datasets 5.1.0, published on PyPI on 2026-10-05, requires pyarrow>=24.0.0 (Add support for datasets backed by vortex file format huggingface/datasets#8502).
    • The latest stable tsfile release on PyPI is still 2.4.0, which requires pyarrow<20 on Python 3.10 to 3.13 and pyarrow<24 on Python 3.14. As a result, datasets==5.1.0 and tsfile==2.4.0 no longer resolve together on Python 3.14 either, although the bounds still overlapped there when this issue was opened. I checked this with uv pip install --dry-run on Python 3.12 and 3.14.
    • The workaround given above first installs pyarrow>=21,<24, a range that datasets 5.1.0 no longer accepts. The Python 3.14 CI job of datasets currently installs tsfile with --no-deps alongside pyarrow>=24.0.0.
    • The pre-releases 2.5.0.dev0 and 2.5.0.dev1 carry the relaxed bounds from Fix/relax pyarrow constraint #922, and datasets==5.1.0 resolves with tsfile==2.5.0.dev1 on both Python versions. Since pip and uv do not select pre-releases by default, a requirement such as tsfile>=2.3.0 still fails to resolve against datasets 5.1.0.

    Is there a target date for the 2.5.0 release?

  3. ColinLeeo commented on Oct 8, 2026

    @ColinLeeo
    Contributor

    Thank you for following up and verifying compatibility with the pre-release.

    TsFile 2.5.0 has completed internal testing, and we plan to start the release vote within the next two days. Assuming the vote passes, we expect to publish it to PyPI this weekend or early next week.

  4. ColinLeeo commented on Oct 14, 2026

    @ColinLeeo
    Contributor

    Hi,thanks for your time and TsFile now has already release v2.5.0. You can now install tsfile 2.5.0 via pip directly.

    Pypi ref: https://pypi.org/project/tsfile/2.5.0/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions