feat: Handle all forms of range requests in fsspec

[gschulze]

Closes #259.

Adds support for all forms of range requests documented by fsspec, without changing obstore's own API.

Implemented:

• cat_file and cat_ranges accept every range form fsspec documents: either bound alone, and either counting back from the end of the object.
• cat_ranges additionally takes a scalar or None broadcast across all paths, and None elements. It also honors max_gap, batch_size, and on_error, which were previously ignored.
• Unless on_error="raise", failures now come back in the returned list.
• Suffix requests fall back to a size lookup and a bounded read on stores that reject them (Azure). A suffix larger than the object becomes a plain get.
• test_cat_ranges_mixed is no longer xfail.
• New tests were added to cover the new functionality.

Request cost: A plain start/end goes through get_range; start-only and negative-start become get with {"offset": n} and {"suffix": n}, so they stay single requests. A zero or absent start with no end sends no range header. cat_ranges splits its input the same way _get_partial_values does in the zarr PR: bounded ranges batch per object through get_ranges so nearby ones are merged, and each open-ended range is a separate get request. The object size is only needed for a negative end, or a negative start paired with an end.

Degenerate ranges: Zero-length, inverted, and start-past-the-end ranges raise an exception rather than returning empty. The first two are rejected by validate_range in obstore/src/get.rs, the third by object_store itself.

This only touches fsspec.py and its tests. get_range and get_ranges are unchanged.

[kylebarron]

This seems unnecessarily complicated when you could just add a parameter into a dict

[gschulze]

Agree on _CoalesceKwarg being ugly, it was added solely to satisfy the type checker. To make your suggestion work, I had to pass lengths explicitly, otherwise Pyright would complain.

[kylebarron]

Resolving a range should never need to know the size of the object (except on Azure, which doesn't support suffix requests).

I'd strongly recommend taking a similar approach to the Zarr-Python obstore adapter. https://github.com/zarr-developers/zarr-python/blob/d44f9f92ab4f12a8008de2553a7c9988669e3910/src/zarr/storage/_obstore.py#L440-L491

(For Azure, you can have a config parameter for the fsspec adapter for whether to avoid suffix requests, which then would make a HEAD request to always know the size)

[gschulze]

There are two cases where we need to know the size of the object: when start is negative and the backend does not support suffix requests, and when the bounds do not map to an HTTP range request, which occurs if end is negative, or when a negative start is paired with an end.

For the first case, I went with the zarr adapter's runtime fallback instead of a config parameter, which looks up the size after the store refuses the suffix.

[kylebarron]

Can you link to source code in fsspec that supports this typing? I.e. are there tests or a code path where we know that starts and ends can take None, either in isolation or as an element in a sequence?

[gschulze]

Added a comment with a permalink in _cat_ranges.

[kylebarron]

Overall this PR still seems to be overly complex for what it does (I'd say it seems to be written too much by Claude), and needs to be simplified a lot before merge

[kylebarron]

Context from a Claude review on my side, if you're interested. From a quick read of this summary, I think all of these are valid points that would need to be addressed before merging.

---

Thanks for the rework — going the adapter-only route is right, and the overall shape is what I had in mind: bounded ranges batched per object through get_ranges_async, open-ended ones through get_async with {"offset"}/{"suffix"}, mirroring _get_partial_values.

I built this branch and ran the suite against minio: 28 passed, 1 xfail, and all the new tests pass. ruff is clean; pyright shows only the pre-existing missing-stub errors for the fsspec imports.

Some follow-ups on my inline comments, including one where I was wrong.

Follow-ups on the inline comments

_CoalesceKwarg (fsspec.py:126)

Stands — and it's a one-liner. Both pyright and mypy accept the plain form:

kw: dict[str, int] = {} if max_gap is None else {"coalesce": max_gap}
store.get_ranges_async(path, starts=..., ends=..., **kw)

I see where the TypedDict came from: ebca428 refactor: Accept None for coalesce so the default lives in one place got reverted wholesale in 275e3c6 along with the get_opts change. My objection was only to get_range routing through get_opts — coalesce: int | None = None on the binding is independently reasonable and I'd take it as a follow-up if you'd rather have the default live in one place.

Needing the object size (fsspec.py:131)

I was too absolute here. A negative end genuinely has no HTTP range equivalent: data[:-10] is bytes 0..size-10, and a suffix request gives the last N bytes, not all but the last N. fsspec's own reference implementation does exactly what this PR does:

# fsspec/spec.py, AbstractFileSystem.cat_file
if end is not None:
    if end < 0:
        end = f.size + end

And test_cat_ranges_mixed — the test this issue exists to un-xfail — passes ends=[None, -10, -10]. The zarr adapter avoids the problem only because zarr's ByteRequest is a closed set (Range/Offset/Suffix) with no negative end, so it doesn't port directly. Fetching the size for a negative end is fine.

There is a real bug in the neighborhood, though. _resolve_inexpressible_bounds applies the fetched size to every row, not just the rows that needed it:

fs.cat_ranges([path, path], starts=[-5, 0], ends=[None, -10])
# range actually sent for row 0: {'offset': 9995}   ← should be {"suffix": 5}

Row 0 is an exactly-expressible suffix, but it gets rewritten into a size-dependent offset because row 1 happened to need a HEAD. So a pure suffix silently becomes size-dependent based on unrelated entries in the same call. Please gate _apply_object_size on _needs_object_size per row so rows that don't need the size are left alone.

Sequence[int | None] typing (fsspec.py:479)

You're right, and here's the source I was asking for — worth putting a pointer to one of these in a comment:

• AbstractFileSystem.cat_ranges (fsspec/spec.py) forwards each element unchanged: out.append(self.cat_file(p, s, e)). cat_file's documented contract is "start, end: int — If negative, backwards from end, like usual python slices. Either can be None for start or end of file, respectively."
• AsyncFileSystem._cat_ranges (fsspec/asyn.py) does the same: self._cat_file(p, start=s, end=e) for p, s, e in zip(paths, starts, ends).
• fsspec.utils.merge_offset_ranges normalizes starts = [s or 0 for s in starts] and guards e is not None — None elements are expected there.
• Scalar None broadcast follows from if not isinstance(starts, Iterable), since None isn't Iterable.

Other things I found

on_error diverges from upstream, and the comment justifying it is wrong. The code says "Like fsspec's own _cat_ranges, on_error is ignored, so failures propagate." Upstream ignores the on_error argument but hardcodes return_exceptions=True, so it always behaves as the default on_error="return":

upstream memory fs: [b'abc', FileNotFoundError('/nope')]
this branch:        raises FileNotFoundError

Not a regression — the old asyncio.gather also raised — but the comment should be corrected either way. If we want parity it's one kwarg plus widening output_buffers to list[bytes | BaseException].

A fully-unbounded element in cat_ranges still sends a range header. _cat_file special-cases start in (0, None), end=None into a plain get, but _cat_ranges turns it into {"offset": 0} → Range: bytes=0-, which 416s on a zero-length object where cat_file returns b"". Cheap to mirror the special case.

Degenerate ranges. Agreed with the choice to keep raising, but note upstream isn't merely inconsistent between backends — the reference cat_file does f.read(end - f.tell()), and AbstractBufferedFile.read treats a negative length as "rest of file", so an inverted range silently returns the tail. Raising is better behavior; it just deserves a line in the cat_file/cat_ranges docstring since it deviates from the documented slice semantics.

fsspec.asyn._run_coros_in_chunks is private API. It's the only way to honor batch_size and the adapter already leans on fsspec internals, so I'm fine with it — just noting the coupling.

Azure. This introduces suffix requests where the adapter previously made none, and Azure doesn't support them. Out of scope for this PR — I'll open a separate issue. (The zarr adapter catches the failure and falls back to HEAD + bounded range.)

[gschulze]

Thanks for your detailed feedback, this is all fair. Everything you raised is implemented locally. Regarding complexity, I'm afraid the PR has not gotten smaller after incorporating your points, but I think the code quality has improved. I still need some time to check whether everything is consistent now, and whether I can simplify it any further. Will ping when ready for another look.

[gschulze]

I've addressed all the points you raised and updated the PR description.

[kylebarron]

I'm catching up on my backlog. I took a quick look at it but I need more time to digest it.

I'm pasting below some opinions from Claude that I can read through next week when I come back to this PR

Claude summary

Thanks for the updates. Here's where I think this stands.

Resolved

The Rust changes are reverted, so get_range and get_ranges are untouched — thanks. _CoalesceKwarg is gone in favor of a plain dict, and the fsspec permalink is in place. That permalink does support the typing: the base _cat_ranges forwards each element straight to _cat_file, which documents negative bounds and None for either end.

On my "resolving a range should never need the object size" comment — you're right and I was overreaching. A negative end, or a negative start paired with an end, has no HTTP range-header equivalent, so a size lookup really is required there.

Two bugs

Size lookups ignore on_error. obstore/python/obstore/fsspec.py:659 resolves sizes via self._sizes, which calls _run_coros_in_chunks without return_exceptions=True. So any bound that needs a size raises instead of returning the error, and it takes the whole batch down with it. Against a memory store:

cat_ranges(["missing.txt"], [0], [10])   # -> [FileNotFoundError(...)]  correct
cat_ranges(["missing.txt"], [0], [-2])   # -> raises FileNotFoundError  wrong
cat_ranges(["a.txt", "missing.txt"], [0, 0], [5, -2])  # raises, discarding the good read

test_cat_ranges_on_error only uses bounds that skip the size lookup, so this path isn't covered.

Suffix request against an empty object. obstore/python/obstore/fsspec.py:176 sends {"suffix": n} unguarded. That's the same unsatisfiable-range hazard the start == 0 special case two lines above exists to avoid — S3 and GCS answer 416 for bytes=-n on a zero-byte object. I haven't confirmed this against a real backend; the empty-object case in test_suffix_fallback only exercises the patched-refusal branch, so it wouldn't catch it either way.

On complexity

This is my main remaining concern. The range translation is spread over nine units: four module-level helpers plus three extra methods on top of _cat_file and _cat_ranges. Roughly 70 of the 284 changed lines exist to batch size lookups across paths before dispatch, for what is a fairly rare case.

That batching pre-pass is what forces most of the structure. It's why _needs_object_size and _apply_object_size are split apart and each called twice, why _resolve_inexpressible_bounds needs a dedupe, a zip and a cast, and why error handling ended up outside the try/except that _cat_ranges already has — which is how the first bug above slipped in.

A smaller shape I'd be happy with: route only plain non-negative bounded ranges into the per-object get_ranges batch, and send everything else through _cat_file one at a time, tagged with its output index. _cat_file already resolves sizes correctly in three lines, and _info is served by the dircache. That would let you delete _needs_object_size, _apply_object_size, _resolve_inexpressible_bounds and _cat_open_ended_range, and put all the error handling in one place.

The tradeoff is that a range with a negative end no longer joins its object's coalesced batch. Suffix reads — the common case, e.g. Parquet footers — are already open-ended and dispatched individually either way, so I think that's a cheap price for the simplification.