Skip to content

Feature: chunked + concurrent get_range helper for long ranges (and CSV encoding escape hatch) #33

Closed

Description

@tbeason

Enhancement request, based on field experience pulling 15 years of Schema.STATISTICS for 8 continuous symbols on GLBX.MDP3 (v0.2.0).

Motivation

Each get_range request for continuous-symbol queries carries ~25-30s of fixed server-side latency regardless of payload size. That interacts badly with both obvious client strategies:

  • One big request: exceeds the 100s default read timeout (separate issue) and is all-or-nothing on failure.
  • Sequential chunking: 8 symbols x 16 calendar years = 128 requests x ~28s = ~1 hour of accumulated fixed latency for ~66MB of data.

What actually worked: calendar-year chunks fetched concurrently (8 @async tasks gated by a Base.Semaphore) — the same 128 requests complete in a few minutes, and per-chunk failures are retryable without losing the rest. The client is already documented as safe to share across tasks, and the built-in 429 retry-with-backoff handled rate-limit brushes.

Proposal

A get_range_chunked (or kwargs on get_range: chunk = Year(1), concurrency = 8) that:

  1. splits [start_dt, end_dt) into chunks,
  2. fetches them concurrently under a semaphore,
  3. concatenates records in time order,
  4. warns-and-continues on per-chunk failure (collecting failed chunk ranges in the return for retry).

Also worth considering: an encoding = "csv" escape hatch on get_range returning a raw table — it is a useful fallback when a binary decode path has problems (it is how I worked around the StatMsg layout bug reported in DatabentoBinaryEncoding.jl), and trivially supports map_symbols/pretty_px for interactive use.

A reference implementation of the chunk+concurrency pattern is in my creditspreadsresearch repo (part1/download_statistics.jl).

Activity

  1. tbeason commented on Jun 10, 2026

    @tbeason
    OwnerAuthor

    The chunk+concurrency part of this is implemented in #37 (chunk/concurrency kwargs on get_range, with failed-chunk ranges reported in the new DBNStore.failed_ranges field).

    The encoding="csv" escape hatch is deferred to a follow-up — it needs a decision on the return shape (raw CSV string vs. a parsed table, which would pull in a CSV.jl dependency) plus map_symbols/pretty_px plumbing, and is independent of the latency problem this issue is mainly about.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions