You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Feature: chunked + concurrent get_range helper for long ranges (and CSV encoding escape hatch) #33
Enhancement request, based on field experience pulling 15 years of Schema.STATISTICS for 8 continuous symbols on GLBX.MDP3 (v0.2.0).
Motivation
Each get_range request for continuous-symbol queries carries ~25-30s of fixed server-side latency regardless of payload size. That interacts badly with both obvious client strategies:
One big request: exceeds the 100s default read timeout (separate issue) and is all-or-nothing on failure.
Sequential chunking: 8 symbols x 16 calendar years = 128 requests x ~28s = ~1 hour of accumulated fixed latency for ~66MB of data.
What actually worked: calendar-year chunks fetched concurrently (8 @async tasks gated by a Base.Semaphore) — the same 128 requests complete in a few minutes, and per-chunk failures are retryable without losing the rest. The client is already documented as safe to share across tasks, and the built-in 429 retry-with-backoff handled rate-limit brushes.
Proposal
A get_range_chunked (or kwargs on get_range: chunk = Year(1), concurrency = 8) that:
splits [start_dt, end_dt) into chunks,
fetches them concurrently under a semaphore,
concatenates records in time order,
warns-and-continues on per-chunk failure (collecting failed chunk ranges in the return for retry).
Also worth considering: an encoding = "csv" escape hatch on get_range returning a raw table — it is a useful fallback when a binary decode path has problems (it is how I worked around the StatMsg layout bug reported in DatabentoBinaryEncoding.jl), and trivially supports map_symbols/pretty_px for interactive use.
A reference implementation of the chunk+concurrency pattern is in my creditspreadsresearch repo (part1/download_statistics.jl).
The chunk+concurrency part of this is implemented in #37 (chunk/concurrency kwargs on get_range, with failed-chunk ranges reported in the new DBNStore.failed_ranges field).
The encoding="csv" escape hatch is deferred to a follow-up — it needs a decision on the return shape (raw CSV string vs. a parsed table, which would pull in a CSV.jl dependency) plus map_symbols/pretty_px plumbing, and is independent of the latency problem this issue is mainly about.
Activity
tbeason commentedon Jun 10, 2026
The chunk+concurrency part of this is implemented in #37 (
chunk/concurrencykwargs onget_range, with failed-chunk ranges reported in the newDBNStore.failed_rangesfield).The
encoding="csv"escape hatch is deferred to a follow-up — it needs a decision on the return shape (raw CSV string vs. a parsed table, which would pull in a CSV.jl dependency) plusmap_symbols/pretty_pxplumbing, and is independent of the latency problem this issue is mainly about.