From 3995ee42cf524d46279f2ef17c207f3216013d2b Mon Sep 17 00:00:00 2001 From: Adil Date: Thu, 27 Aug 2026 20:27:38 +0500 Subject: [PATCH 1/5] docs: rewrite README as end-user documentation Drag the README back to an end-user pitch: remove coder narration (module format internals, dependency/roadmap notes) and present the tool in plain language for financial-data users and AI readers. Removes em dashes. Add a patch changeset. --- .changeset/fuzzy-readmes-resample.md | 8 + README.md | 347 ++++++++++++++------------- 2 files changed, 189 insertions(+), 166 deletions(-) create mode 100644 .changeset/fuzzy-readmes-resample.md diff --git a/.changeset/fuzzy-readmes-resample.md b/.changeset/fuzzy-readmes-resample.md new file mode 100644 index 0000000..1dc2944 --- /dev/null +++ b/.changeset/fuzzy-readmes-resample.md @@ -0,0 +1,8 @@ +--- +"ohlc-resample": patch +--- + +Rewrite the README as end-user documentation: drop internal implementation +narration (module-format internals, dependency and roadmap notes) and present +the tool in plain language for financial-data users. Also removes em dashes +for easier machine parsing. diff --git a/README.md b/README.md index db707bd..9e42765 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@

ohlc-resample ๐Ÿ•ฏ๏ธ

-Resample (inter-convert) trade, ticks or OHLCV data to different time frames +Turn trade, tick, or OHLCV data into clean candlestick charts on any time frame

@@ -18,11 +18,21 @@ Resample (inter-convert) trade, ticks or OHLCV data to different time frames

-- Typescript support -- CCXT support -- Single dependency -- Low time complexity grouping based aggregations -- Optional gap filling +## What it does + +Market data comes in many shapes: raw trades, tick streams, or ready-made +OHLCV candles at a given time frame (1m, 5m, 1h, ...). `ohlc-resample` converts +between them so you always end up with the candles you want to chart: + +- Combine raw ticks or trades into **OHLCV candles** (open, high, low, close, + volume) over a time period or a fixed number of ticks. +- Rebuild OHLCV candles from one time frame to a coarser one (for example + 1-minute candles into 5-minute candles). +- Handle data in the common formats: CCXT-style arrays, JSON objects, CSV, + JSONL, and Parquet files. +- Process **very large datasets** without running out of memory, by streaming + input line by line or row group by row group. +- Fill in missing candles so your chart has no gaps. ## Install @@ -34,7 +44,9 @@ macOS / Linux: curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh ``` -Installs the `ohlc` CLI to `~/.local/bin`. If you don't already have a recent enough Node, the installer downloads one for you and uses it. Pin a specific version with `--version 2.0.0`. +This installs the `ohlc` command to `~/.local/bin`. If you do not already have +a recent enough Node, the installer downloads one for you and uses it. Pin a +specific version with `--version 2.0.0`. To uninstall: @@ -48,15 +60,35 @@ curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh -s - npm install ohlc-resample # or pnpm add / yarn add / bun add ``` -Requires Node.js โ‰ฅ26. +Requires Node.js 26 or newer. + +## Quick start (CLI) + +```bash +# Resample 1-minute candles in data.csv into 5-minute candles +ohlc -i data.csv -b 60 -n 300 + +# Resample a Parquet file and write JSON candles to a file +ohlc -i data.parquet -f json -o candles.json + +# Pipe in JSON, get JSON out +cat data.json | ohlc +``` + +See the [CLI section](#cli) below for all options. -## Supported formats +## Supported input formats -- OHLCV (CCXT format) `[[time,open,high,low,close,volume]]` -- OHLCV JSON `[{time: number, open: number, high: number, low: number close: number, volume: number}]` -- Trade JSON `[{time: number, price: number, quantity: number}]` +- **OHLCV arrays** (CCXT-style) `[[time, open, high, low, close, volume], ...]` +- **OHLCV JSON objects** `[{ time, open, high, low, close, volume }, ...]` +- **Trade / tick JSON objects** `[{ time, price, quantity }, ...]` +- **CSV**, **JSON**, and **JSONL** files +- **Parquet** files -## Reference +Input times are epoch **milliseconds**. See the [Types](#types) section for +the exact shapes. + +## Library usage ```typescript import { @@ -65,43 +97,45 @@ import { resampleTicksByCount, } from "ohlc-resample"; -// OHLCV resampled from 1 minute to 5 minute - -resampleOhlcv(objectOhlcv as IOHLCV[], { +// OHLCV candles from 1 minute to 5 minutes +resampleOhlcv(objectOhlcv, { baseTimeframe: 60, newTimeframe: 5 * 60, -}); // return IOHLCV[] -resampleOhlcv(arrayOhlcv as OHLCV[], { +}); // returns IOHLCV[] objects +resampleOhlcv(arrayOhlcv, { baseTimeframe: 60, newTimeframe: 5 * 60, -}); // return OHLCV[] - -// Ticks grouped and resampled to 1m OHCLV -// option.includeLatestCandle is by default `true` -// options.fillGaps is by default `false` +}); // returns OHLCV[] tuples -resampleTicksByTime(tickData as TradeTick[], { +// Ticks grouped into 1-minute OHLCV candles +// includeLatestCandle is true by default, fillGaps is false by default +resampleTicksByTime(tickData, { timeframe: 60, includeLatestCandle: false, fillGaps: true, -}); // return IOHLCV[] +}); // returns IOHLCV[] -// Ticks grouped and resampled by every 5 ticks - -resampleTicksByCount(tickData as TradeTick[], { tickCount: 5 }); // return IOHLCV[] +// Ticks grouped into candles of 5 ticks each +resampleTicksByCount(tickData, { tickCount: 5 }); // returns IOHLCV[] ``` -## Streaming & large data +Each function accepts either an array or any sync iterable / generator. The +result uses the same shape as your input: pass tuples and get tuples back, +pass objects and get objects back. The return type always follows the input +shape. + +## Streaming and large datasets -The three functions above all accept a **sync iterable / generator** as well as -an array (materialized internally), and `resampleOhlcv` also accepts a binary -`Float64Array` of interleaved `[time, open, high, low, close, volume]` records. +The three functions above also accept a sync iterable or generator as input. +`resampleOhlcv` additionally accepts a binary `Float64Array` of interleaved +`[time, open, high, low, close, volume]` values, which is the fastest way to +feed in large binary data. -For **true streaming** โ€” large files, live feeds, or anything you don't want to -hold in memory โ€” use the async variants. They consume an `AsyncIterable` (a -Node `ReadableStream`, an async generator, `for await` sources) and return an -`AsyncGenerator` that emits each bucket as soon as it is safe. Memory use is -bounded by the active bucket window, never the whole input. +For **true streaming**, use the async variants. They read from any async +source (a Node `ReadableStream`, an async generator, anything you can +`for await` over) and emit each candle as soon as it is ready. Memory use stays +bounded by the active time window, never the whole input, so huge files or +live feeds are safe. ```typescript import { @@ -127,32 +161,31 @@ for await (const candle of resampleTicksByTimeAsync(tickSource, { // ... } -// Stream ticks into count buckets (O(tickCount) memory) +// Stream ticks into count buckets (memory scales with tickCount) for await (const candle of resampleTicksByCountAsync(tickSource, { tickCount: 5 })) { // ... } ``` -**Sorted input.** The array API sorts a copy for you; a streaming API cannot -buffer to sort, so pass data **ascending by time** unless you use the healing -window below. +**Sorted input.** The array functions sort a copy for you automatically. A +stream cannot buffer the whole input to sort it, so pass data **ascending by +time** unless you use the healing window below. -**Out-of-order healing.** With `outOfOrderMs > 0`, the stream keeps each bucket -open for that many milliseconds of wall-clock time, so delayed or out-of-order -records that land inside the window are folded into the correct bucket on the -fly โ€” no global sort, and no re-reading already-emitted buckets. `outOfOrderMs -= 0` (default) is exact for pre-sorted input and emits as the stream passes. +**Out-of-order data.** Set `outOfOrderMs` to a number of milliseconds. The +stream then keeps each bucket open for that much wall-clock time, so delayed +or out-of-order records that arrive inside the window are folded into the +correct bucket as they come, with no need to sort everything first. +`outOfOrderMs = 0` (the default) is exact for already-sorted input and emits +each bucket as the stream passes it. ### Parquet files -The async variants also accept a **file path** to a Parquet file. Rows are -streamed **row-group by row-group** (memory is bounded by the largest row -group, not the file), so large columnar datasets don't blow up memory. This is -the "foot in the door" before the future native engine โ€” the reader is the -pure-JS, zero-dependency `hyparquet` package. +The async variants also accept a **file path** to a Parquet file. The file is +read one row group at a time, so memory stays bounded by the largest row group +rather than the whole file. This keeps large columnar datasets usable. ```typescript -// resample a Parquet OHLCV file +// Resample a Parquet OHLCV file for await (const candle of resampleOhlcvAsync("data.parquet", { baseTimeframe: 60, newTimeframe: 300, @@ -160,7 +193,7 @@ for await (const candle of resampleOhlcvAsync("data.parquet", { // candle is an OHLCV tuple } -// resample a Parquet tick file +// Resample a Parquet tick file for await (const candle of resampleTicksByTimeAsync("ticks.parquet", { timeframe: 60, })) { @@ -168,24 +201,26 @@ for await (const candle of resampleTicksByTimeAsync("ticks.parquet", { } ``` -Column names are read by **exact canonical name** (`time`, `open`, `high`, -`low`, `close`, `volume`, and `time`/`price`/`quantity` for ticks). Timestamps -in any unit (ms / ยตs / ns / days) are converted to epoch-milliseconds. If a -required column is missing, the call throws; OHLCV `volume` is optional and -defaults to `0`. Any other column layout must be supplied with the `map` -option below. Parquet (file) input is supported **only on the async variants** -โ€” the sync functions stay array/iterable-only. +Parquet columns are read by exact canonical name: `time`, `open`, `high`, +`low`, `close`, `volume` for OHLCV, and `time` / `price` / `quantity` for +ticks. Timestamps in any unit (ms, microseconds, nanoseconds, days) are +converted to milliseconds automatically. If a required column is missing the +call throws; OHLCV `volume` is optional and defaults to `0`. For any other +column layout, use the [`map` option](#renaming-fields-with-map) below. + +Parquet input works only with the async variants. The sync functions stay +array/iterable-only. -### Per-record `map` option +### Renaming fields with `map` -The async variants take a `map` option that translates **each input record** -into canonical OHLCV. It works uniformly across any named-schema input โ€” -Parquet rows, CSV header rows, JSON objects โ€” and is how you adapt foreign -schemas (CCXT's `timestamp`/`amount`, arbitrary Parquet columns, etc.) instead -of relying on name guessing. Two shapes are accepted: +When your data uses different field names (for example CCXT's `timestamp` and +`amount`, or arbitrary Parquet columns), pass a `map` option to the async +variants. It translates each input record into canonical OHLCV and works +uniformly across Parquet rows, CSV headers, and JSON objects. Two shapes are +accepted. -1. **Record form** โ€” keys are canonical IOHLCV fields, values are the keys to - read from each input record. Fields absent from the map use the canonical +1. **Record form**, where keys are canonical OHLCV fields and values are the + keys to read from each input record. Fields you leave out use the canonical key directly: ```typescript @@ -204,7 +239,7 @@ of relying on name guessing. Two shapes are accepted: })) { } ``` -2. **Function form** โ€” a full transform `(record) => IOHLCV` for complete +2. **Function form**, a full transform `(record) => IOHLCV` for complete control: ```typescript @@ -218,28 +253,23 @@ of relying on name guessing. Two shapes are accepted: })) { } ``` -Ticks accept the same shapes over `time`/`price`/`quantity` (e.g. `map: -{ time: 'timestamp', quantity: 'amount' }`). Positional inputs โ€” `OHLCV` -tuples and `Float64Array` โ€” have no keys to map and are unaffected. The sync -functions take canonical arrays, so `map` is only relevant to the async -variants (file paths and record-shaped streams). +Ticks accept the same two shapes over `time` / `price` / `quantity` (for +example `map: { time: 'timestamp', quantity: 'amount' }`). Positional inputs +(OHLCV tuples and `Float64Array`) have no field names to remap and are +unaffected. The sync functions take canonical arrays, so `map` applies only to +the async variants. ## Module format -`ohlc-resample` is **ESM-first with a CommonJS wrapper** (`"type": "module"`). -Use `import` for the full API. `require('ohlc-resample')` also works: the build -emits `dist/index.cjs`, a one-line re-export that loads the ESM build via -Node's synchronous `require(esm)` (stable since 23.7), so both entry points -resolve to the **same module instance** โ€” there's no dual-package hazard and -no CommonJS build to maintain. The Parquet reader is ESM-native -(`hyparquet` has no CommonJS build), which is why the package is ESM-first; a -thin wrapper keeps `require()` consumers working. +The package is **ESM-first with a CommonJS wrapper** (`"type": "module"`). Use +`import` for the full API; `require('ohlc-resample')` also works and resolves +to the same module instance, so there is no dual-package confusion. ## Types ```typescript export type IOHLCV = { - time: number; + time: number; // epoch milliseconds open: number; high: number; low: number; @@ -247,20 +277,20 @@ export type IOHLCV = { volume: number; }; -export type OHLCV = [number, number, number, number, number, number]; +export type OHLCV = [number, number, number, number, number, number]; // [time, open, high, low, close, volume] export type TradeTick = { + time: number; // epoch milliseconds price: number; quantity: number; - time: number; }; ``` -**Note:** Input time for all above types must be in milliseconds +**Note:** input times for all of the above must be in milliseconds. ## Examples -**Resample CCXT (Object) OHLCV based on timeframe** +**Resample CCXT (object) OHLCV to a coarser time frame** ```typescript import { resampleOhlcv } from "ohlc-resample"; @@ -284,44 +314,25 @@ const link_btc_1m = [ }, ]; -const baseTimeframe = 60; // 60 seconds -const newTimeframe = 120; // 120 seconds - -// Candles made up of ticks within 2 minute timeframes +const baseTimeframe = 60; // 60 seconds +const newTimeframe = 120; // 120 seconds +// Candles built from the ticks within each 2-minute window const link_btc_2m = resampleOhlcv(link_btc_1m, { baseTimeframe, newTimeframe, }); ``` -**Resample ticks to OHLCV based on tick count** +**Resample ticks to OHLCV candles by tick count** ```typescript import { resampleTicksByCount, TradeTick } from "ohlc-resample"; const adabnb_trades = [ - { - time: "1564502620356", - side: "sell", - quantity: "4458", - price: "0.00224", - tradeId: "1221272", - }, - { - time: "1564503133949", - side: "sell", - quantity: "3480", - price: "0.002242", - tradeId: "1221273", - }, - { - time: "1564503134553", - side: "buy", - quantity: "51", - price: "0.002248", - tradeId: "1221274", - }, + { time: "1564502620356", side: "sell", quantity: "4458", price: "0.00224", tradeId: "1221272" }, + { time: "1564503133949", side: "sell", quantity: "3480", price: "0.002242", tradeId: "1221273" }, + { time: "1564503134553", side: "buy", quantity: "51", price: "0.002248", tradeId: "1221274" }, ]; const airbnb_ticks: TradeTick[] = adabnb_trades.map((trade: any) => ({ @@ -330,28 +341,29 @@ const airbnb_ticks: TradeTick[] = adabnb_trades.map((trade: any) => ({ price: Number(trade.price), })); -// Candles made up of two ticks - +// Candles built from two ticks each const tickChart = resampleTicksByCount(airbnb_ticks, { tickCount: 2, }); ``` -## CLI Usage +## CLI -The package includes a command-line interface for resampling OHLCV data between timeframes and file formats. +The package includes a command-line interface for resampling OHLCV data between +time frames and file formats. The installed command is `ohlc` (the npm package +binary is `ohlc-resample`). -### Basic Usage +### Basic usage ```bash -# Resample CSV file with default timeframes (1m -> 5m) -ohlc-resample -i input.csv +# Resample CSV with default time frames (1m to 5m) +ohlc -i input.csv -# Resample JSON file with custom timeframes -ohlc-resample -i input.json -b 60 -n 300 +# Resample JSON with custom time frames +ohlc -i input.json -b 60 -n 300 -# Save output to file with specific format -ohlc-resample -i input.csv -o output.json -f json +# Save output to a file in a specific format +ohlc -i input.csv -o output.json -f json ``` ### Options @@ -382,60 +394,62 @@ The CLI accepts and emits two equivalent JSON shapes: [[1609459200000, 100, 105, 95, 102, 1000]] ``` -Input shape is auto-detected. Output shape mirrors input by default; override with `-s array` or `-s object`. CSV input is always parsed as object-shape; CSV output is always rows. +Input shape is auto-detected. Output shape mirrors the input by default; +override with `-s array` or `-s object`. CSV input is always parsed as +object-shape, and CSV output is always rows. ### Large files -`.csv`, `.jsonl`, and `.ndjson` **files are read line-by-line and fed through -the async streaming resampler**, so memory never scales with file size (a -JSON array file is the exception: the whole document must be parsed to know -where the array ends, so it stays buffer-based). `.parquet` **files are read -row-group by row-group** through the same async resampler. Output is written -incrementally in CSV, JSON (a valid, parseable array), or JSONL. +`.csv`, `.jsonl`, and `.ndjson` files are read line by line and fed through +the streaming resampler, so memory never scales with file size. (A JSON array +file is the exception: the whole document must be parsed to know where the +array ends, so it stays in memory.) `.parquet` files are read row group by row +group through the same streaming resampler. Output is written incrementally in +CSV, JSON (a valid, parseable array), or JSONL. ```bash # Stream a 100MB CSV to JSONL candles -ohlc-resample -i huge.csv -f jsonl -o candles.jsonl +ohlc -i huge.csv -f jsonl -o candles.jsonl # Resample a Parquet OHLCV file (row-group streaming) -ohlc-resample -i data.parquet -f json -o candles.json +ohlc -i data.parquet -f json -o candles.json -# Pipe JSONL line-by-line (no buffering) -cat data.jsonl | ohlc-resample --input-format jsonl -f jsonl +# Pipe JSONL line by line (no buffering) +cat data.jsonl | ohlc --input-format jsonl -f jsonl ``` -### Input Formats +### Input formats -The CLI supports CSV, JSON, and JSONL input formats: +The CLI supports CSV, JSON, and JSONL input. -#### Mapping non-canonical fields (`--map`) +#### Renaming fields with `--map` -When your input uses different field names (CCXT-style `timestamp`/`amount`, a -CSV header in a foreign order, arbitrary Parquet columns), pass `--map` with -`field=sourceKey` entries separated by commas. It applies to CSV headers, -JSON object keys, and Parquet columns; tuple (array) records have no keys and +When your input uses different field names (CCXT-style `timestamp` / `amount`, +a CSV header in a foreign order, arbitrary Parquet columns), pass `--map` with +`field=sourceKey` entries separated by commas. It applies to CSV headers, JSON +object keys, and Parquet columns; tuple (array) records have no field names and are unaffected. With `--map`, a CSV's first line is always treated as the header. ```bash # CCXT-style JSON objects: timestamp + amount -ohlc-resample -i data.json --map time=timestamp,volume=amount +ohlc -i data.json --map time=timestamp,volume=amount # Foreign CSV header -ohlc-resample -i data.csv --map time=timestamp,volume=amount +ohlc -i data.csv --map time=timestamp,volume=amount # Arbitrary Parquet columns -ohlc-resample -i data.parquet --map time=mytime,open=myopen,high=myhigh,low=mylow,close=myclose,volume=myvol +ohlc -i data.parquet --map time=mytime,open=myopen,high=myhigh,low=mylow,close=myclose,volume=myvol ``` -#### CSV Format +#### CSV format ```csv time,open,high,low,close,volume 1609459200000,100,105,95,102,1000 1609459260000,102,107,101,106,1200 ``` -#### JSON Format +#### JSON format ```json [ { @@ -449,44 +463,45 @@ time,open,high,low,close,volume ] ``` -#### JSONL Format (one candle per line) +#### JSONL format (one candle per line) ```json {"time":1609459200000,"open":100,"high":105,"low":95,"close":102,"volume":1000} {"time":1609459260000,"open":102,"high":107,"low":101,"close":106,"volume":1200} ``` -### Pipe Input +### Pipe input -You can pipe data into the CLI from other commands. The format is automatically detected, or you can specify it: +You can pipe data into the CLI from other commands. The format is detected +automatically, or you can specify it: ```bash # Auto-detect format -cat data.json | ohlc-resample -cat data.csv | ohlc-resample +cat data.json | ohlc +cat data.csv | ohlc -# Force specific format -cat data.json | ohlc-resample --input-format json -cat data.csv | ohlc-resample --input-format csv +# Force a specific format +cat data.json | ohlc --input-format json +cat data.csv | ohlc --input-format csv ``` -The CLI supports two types of pipe input: -1. JSON files/strings with OHLCV objects -2. CSV files/strings with headers (time,open,high,low,close,volume) +The CLI supports two kinds of pipe input: +1. JSON files or strings with OHLCV objects +2. CSV files or strings with a header (time, open, high, low, close, volume) -### Examples +### CLI examples ```bash # Resample 1-minute data to 5-minute candles -ohlc-resample -i data.csv -b 60 -n 300 +ohlc -i data.csv -b 60 -n 300 -# Convert CSV to JSON format -ohlc-resample -i data.csv -f json +# Convert CSV to JSON +ohlc -i data.csv -f json -# Pipe data and save to file -cat data.csv | ohlc-resample -o output.json +# Pipe data and save to a file +cat data.csv | ohlc -o output.json -# Resample with custom timeframes and save as CSV -ohlc-resample -i data.json -b 300 -n 3600 -f csv -o output.csv +# Resample with custom time frames and save as CSV +ohlc -i data.json -b 300 -n 3600 -f csv -o output.csv ``` ## Contributors From 2b65e0566889e3e21d86dea5c5be8efda8d59e95 Mon Sep 17 00:00:00 2001 From: Adil Date: Thu, 27 Aug 2026 20:34:12 +0500 Subject: [PATCH 2/5] docs: add npx quick-start option and fold install one-liner into quick start --- README.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/README.md b/README.md index 9e42765..4f597a1 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,9 @@ Requires Node.js 26 or newer. ## Quick start (CLI) ```bash +# Install +curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh + # Resample 1-minute candles in data.csv into 5-minute candles ohlc -i data.csv -b 60 -n 300 @@ -75,6 +78,12 @@ ohlc -i data.parquet -f json -o candles.json cat data.json | ohlc ``` +**OR use directly** with npx, no install needed: + +```bash +npx ohlc-resample -i data.csv -b 60 -n 300 +``` + See the [CLI section](#cli) below for all options. ## Supported input formats From db994635ad58cedc3cfc1f8ee46d1bc6493337b2 Mon Sep 17 00:00:00 2001 From: Adil Date: Thu, 27 Aug 2026 20:35:29 +0500 Subject: [PATCH 3/5] docs: note arbitrary input schemas via map in supported formats --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4f597a1..088c82a 100644 --- a/README.md +++ b/README.md @@ -93,9 +93,11 @@ See the [CLI section](#cli) below for all options. - **Trade / tick JSON objects** `[{ time, price, quantity }, ...]` - **CSV**, **JSON**, and **JSONL** files - **Parquet** files +- **Arbitrary schemas**, when you supply a `map` to the canonical OHLCV fields (for example CCXT's `timestamp` / `amount`, or any foreign column names) Input times are epoch **milliseconds**. See the [Types](#types) section for -the exact shapes. +the exact shapes, and the [map option](#renaming-fields-with-map) section for +adapting arbitrary inputs. ## Library usage From 60db0c7c98ab8f565307a09f09307350d1d079ef Mon Sep 17 00:00:00 2001 From: Adil Date: Thu, 27 Aug 2026 20:43:39 +0500 Subject: [PATCH 4/5] docs: reframe map as feed-data-as-is; condense README and cut repetition --- README.md | 317 ++++++++++++++---------------------------------------- 1 file changed, 80 insertions(+), 237 deletions(-) diff --git a/README.md b/README.md index 088c82a..ab3b4cc 100644 --- a/README.md +++ b/README.md @@ -20,39 +20,26 @@ Turn trade, tick, or OHLCV data into clean candlestick charts on any time frame ## What it does -Market data comes in many shapes: raw trades, tick streams, or ready-made -OHLCV candles at a given time frame (1m, 5m, 1h, ...). `ohlc-resample` converts -between them so you always end up with the candles you want to chart: +`ohlc-resample` converts market data into the candlestick charts you want: - Combine raw ticks or trades into **OHLCV candles** (open, high, low, close, - volume) over a time period or a fixed number of ticks. -- Rebuild OHLCV candles from one time frame to a coarser one (for example - 1-minute candles into 5-minute candles). -- Handle data in the common formats: CCXT-style arrays, JSON objects, CSV, - JSONL, and Parquet files. -- Process **very large datasets** without running out of memory, by streaming - input line by line or row group by row group. + volume), by time period or by a fixed number of ticks. +- Rebuild candles from one time frame to a coarser one (1-minute to 5-minute, + for example). +- Work with the common formats (CCXT-style arrays, JSON, CSV, JSONL, Parquet) + and stream very large datasets without running out of memory. - Fill in missing candles so your chart has no gaps. ## Install ### CLI -macOS / Linux: - ```sh curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh ``` -This installs the `ohlc` command to `~/.local/bin`. If you do not already have -a recent enough Node, the installer downloads one for you and uses it. Pin a -specific version with `--version 2.0.0`. - -To uninstall: - -```sh -curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh -s -- --uninstall -``` +Installs the `ohlc` command to `~/.local/bin` (and a Node runtime if needed). +Pin a version with `--version 2.0.0`. Uninstall with `--uninstall`. ### Library @@ -73,9 +60,6 @@ ohlc -i data.csv -b 60 -n 300 # Resample a Parquet file and write JSON candles to a file ohlc -i data.parquet -f json -o candles.json - -# Pipe in JSON, get JSON out -cat data.json | ohlc ``` **OR use directly** with npx, no install needed: @@ -93,11 +77,9 @@ See the [CLI section](#cli) below for all options. - **Trade / tick JSON objects** `[{ time, price, quantity }, ...]` - **CSV**, **JSON**, and **JSONL** files - **Parquet** files -- **Arbitrary schemas**, when you supply a `map` to the canonical OHLCV fields (for example CCXT's `timestamp` / `amount`, or any foreign column names) +- **Arbitrary schemas**, when you supply a `map` (see [Feeding data as-is](#feeding-data-as-is-with-map)) -Input times are epoch **milliseconds**. See the [Types](#types) section for -the exact shapes, and the [map option](#renaming-fields-with-map) section for -adapting arbitrary inputs. +Input times are epoch **milliseconds**. See [Types](#types). ## Library usage @@ -109,44 +91,28 @@ import { } from "ohlc-resample"; // OHLCV candles from 1 minute to 5 minutes -resampleOhlcv(objectOhlcv, { - baseTimeframe: 60, - newTimeframe: 5 * 60, -}); // returns IOHLCV[] objects -resampleOhlcv(arrayOhlcv, { - baseTimeframe: 60, - newTimeframe: 5 * 60, -}); // returns OHLCV[] tuples +resampleOhlcv(objectOhlcv, { baseTimeframe: 60, newTimeframe: 5 * 60 }); // IOHLCV[] +resampleOhlcv(arrayOhlcv, { baseTimeframe: 60, newTimeframe: 5 * 60 }); // OHLCV[] // Ticks grouped into 1-minute OHLCV candles -// includeLatestCandle is true by default, fillGaps is false by default -resampleTicksByTime(tickData, { - timeframe: 60, - includeLatestCandle: false, - fillGaps: true, -}); // returns IOHLCV[] +resampleTicksByTime(tickData, { timeframe: 60, fillGaps: true }); // IOHLCV[] // Ticks grouped into candles of 5 ticks each -resampleTicksByCount(tickData, { tickCount: 5 }); // returns IOHLCV[] +resampleTicksByCount(tickData, { tickCount: 5 }); // IOHLCV[] ``` -Each function accepts either an array or any sync iterable / generator. The -result uses the same shape as your input: pass tuples and get tuples back, -pass objects and get objects back. The return type always follows the input -shape. +Each function accepts an array or any sync iterable / generator. The result +uses the same shape as your input (tuples in, tuples out; objects in, objects +out). `resampleOhlcv` also accepts a binary `Float64Array` of interleaved +`[time, open, high, low, close, volume]` values, the fastest way to feed large +binary data. ## Streaming and large datasets -The three functions above also accept a sync iterable or generator as input. -`resampleOhlcv` additionally accepts a binary `Float64Array` of interleaved -`[time, open, high, low, close, volume]` values, which is the fastest way to -feed in large binary data. - -For **true streaming**, use the async variants. They read from any async -source (a Node `ReadableStream`, an async generator, anything you can -`for await` over) and emit each candle as soon as it is ready. Memory use stays -bounded by the active time window, never the whole input, so huge files or -live feeds are safe. +The async variants read from any async source (a `ReadableStream`, an async +generator, anything you can `for await` over) and emit each candle as soon as +it is ready, so memory stays bounded by the active time window rather than the +input size. ```typescript import { @@ -159,83 +125,60 @@ import { for await (const candle of resampleOhlcvAsync(readableStream, { baseTimeframe: 60, newTimeframe: 300, -})) { - // candle is a completed OHLCV bucket -} +})) { } // Stream ticks into time buckets for await (const candle of resampleTicksByTimeAsync(tickSource, { timeframe: 60, - includeLatestCandle: false, fillGaps: true, -})) { - // ... -} +})) { } // Stream ticks into count buckets (memory scales with tickCount) -for await (const candle of resampleTicksByCountAsync(tickSource, { tickCount: 5 })) { - // ... -} +for await (const candle of resampleTicksByCountAsync(tickSource, { tickCount: 5 })) { } ``` -**Sorted input.** The array functions sort a copy for you automatically. A -stream cannot buffer the whole input to sort it, so pass data **ascending by -time** unless you use the healing window below. - -**Out-of-order data.** Set `outOfOrderMs` to a number of milliseconds. The -stream then keeps each bucket open for that much wall-clock time, so delayed -or out-of-order records that arrive inside the window are folded into the -correct bucket as they come, with no need to sort everything first. -`outOfOrderMs = 0` (the default) is exact for already-sorted input and emits -each bucket as the stream passes it. +**Sorted input.** The array functions sort a copy for you. A stream cannot +buffer the whole input, so pass data **ascending by time** unless you use the +healing window. Set `outOfOrderMs` to a number of milliseconds and the stream +keeps each bucket open that long, folding delayed or out-of-order records into +the correct bucket as they arrive. `outOfOrderMs = 0` (default) is exact for +already-sorted input. ### Parquet files -The async variants also accept a **file path** to a Parquet file. The file is -read one row group at a time, so memory stays bounded by the largest row group -rather than the whole file. This keeps large columnar datasets usable. +The async variants also accept a **file path** to a Parquet file, read one row +group at a time so memory stays bounded by the largest row group. ```typescript -// Resample a Parquet OHLCV file for await (const candle of resampleOhlcvAsync("data.parquet", { baseTimeframe: 60, newTimeframe: 300, -})) { - // candle is an OHLCV tuple -} +})) { } // OHLCV tuple -// Resample a Parquet tick file for await (const candle of resampleTicksByTimeAsync("ticks.parquet", { timeframe: 60, -})) { - // candle is an IOHLCV object -} +})) { } // IOHLCV object ``` -Parquet columns are read by exact canonical name: `time`, `open`, `high`, -`low`, `close`, `volume` for OHLCV, and `time` / `price` / `quantity` for -ticks. Timestamps in any unit (ms, microseconds, nanoseconds, days) are -converted to milliseconds automatically. If a required column is missing the -call throws; OHLCV `volume` is optional and defaults to `0`. For any other -column layout, use the [`map` option](#renaming-fields-with-map) below. - -Parquet input works only with the async variants. The sync functions stay -array/iterable-only. +Parquet columns are read by exact canonical name (`time`, `open`, `high`, +`low`, `close`, `volume` for OHLCV; `time` / `price` / `quantity` for ticks). +Timestamps in any unit (ms, microseconds, nanoseconds, days) become +milliseconds automatically. Missing required columns throw; OHLCV `volume` is +optional and defaults to `0`. Parquet works only with the async variants. -### Renaming fields with `map` +### Feeding data as-is with `map` -When your data uses different field names (for example CCXT's `timestamp` and -`amount`, or arbitrary Parquet columns), pass a `map` option to the async -variants. It translates each input record into canonical OHLCV and works -uniformly across Parquet rows, CSV headers, and JSON objects. Two shapes are -accepted. +`map` removes the need to pre-transform data before passing it in. Feed +records in whatever schema you already have (CCXT's `timestamp` / `amount`, +foreign Parquet columns, and so on) and `map` tells the resampler which keys to +read. It applies uniformly to Parquet rows, CSV headers, and JSON objects. Two +shapes are accepted: -1. **Record form**, where keys are canonical OHLCV fields and values are the - keys to read from each input record. Fields you leave out use the canonical - key directly: +1. **Record form**, mapping canonical OHLCV fields to the keys in your records. + Fields you leave out use the canonical key directly: ```typescript - // read `time` from `timestamp`, `volume` from `amount`; the rest stay canonical + // read `time` from `timestamp` and `volume` from `amount`; the rest stay canonical for await (const candle of resampleOhlcvAsync(readableStream, { baseTimeframe: 60, newTimeframe: 300, @@ -264,17 +207,16 @@ accepted. })) { } ``` -Ticks accept the same two shapes over `time` / `price` / `quantity` (for -example `map: { time: 'timestamp', quantity: 'amount' }`). Positional inputs -(OHLCV tuples and `Float64Array`) have no field names to remap and are -unaffected. The sync functions take canonical arrays, so `map` applies only to -the async variants. +Ticks accept the same shapes over `time` / `price` / `quantity` (for example +`map: { time: 'timestamp', quantity: 'amount' }`). Positional inputs (OHLCV +tuples and `Float64Array`) have no keys to read and are unaffected. The sync +functions take canonical arrays, so `map` applies only to the async variants. ## Module format The package is **ESM-first with a CommonJS wrapper** (`"type": "module"`). Use `import` for the full API; `require('ohlc-resample')` also works and resolves -to the same module instance, so there is no dual-package confusion. +to the same module instance. ## Types @@ -307,31 +249,14 @@ export type TradeTick = { import { resampleOhlcv } from "ohlc-resample"; const link_btc_1m = [ - { - time: 1563625680000, - open: 0.00024824, - high: 0.00024851, - low: 0.00024798, - close: 0.00024831, - volume: 2264, - }, - { - time: 1563625740000, - open: 0.00024817, - high: 0.00024832, - low: 0.00024795, - close: 0.00024828, - volume: 3145, - }, + { time: 1563625680000, open: 0.00024824, high: 0.00024851, low: 0.00024798, close: 0.00024831, volume: 2264 }, + { time: 1563625740000, open: 0.00024817, high: 0.00024832, low: 0.00024795, close: 0.00024828, volume: 3145 }, ]; -const baseTimeframe = 60; // 60 seconds -const newTimeframe = 120; // 120 seconds - // Candles built from the ticks within each 2-minute window const link_btc_2m = resampleOhlcv(link_btc_1m, { - baseTimeframe, - newTimeframe, + baseTimeframe: 60, + newTimeframe: 120, }); ``` @@ -353,29 +278,12 @@ const airbnb_ticks: TradeTick[] = adabnb_trades.map((trade: any) => ({ })); // Candles built from two ticks each -const tickChart = resampleTicksByCount(airbnb_ticks, { - tickCount: 2, -}); +const tickChart = resampleTicksByCount(airbnb_ticks, { tickCount: 2 }); ``` ## CLI -The package includes a command-line interface for resampling OHLCV data between -time frames and file formats. The installed command is `ohlc` (the npm package -binary is `ohlc-resample`). - -### Basic usage - -```bash -# Resample CSV with default time frames (1m to 5m) -ohlc -i input.csv - -# Resample JSON with custom time frames -ohlc -i input.json -b 60 -n 300 - -# Save output to a file in a specific format -ohlc -i input.csv -o output.json -f json -``` +The installed command is `ohlc` (the npm package binary is `ohlc-resample`). ### Options @@ -389,7 +297,7 @@ Options: -s, --shape Output shape for JSON: object, array, auto (default: "auto") -b, --base-timeframe Base timeframe in seconds (default: "60") -n, --new-timeframe New timeframe in seconds (default: "300") - --map Map record fields to canonical keys (e.g. time=timestamp,volume=amount) + --map Feed data as-is; map fields to canonical keys (e.g. time=timestamp,volume=amount) -h, --help Display help for command ``` @@ -409,112 +317,47 @@ Input shape is auto-detected. Output shape mirrors the input by default; override with `-s array` or `-s object`. CSV input is always parsed as object-shape, and CSV output is always rows. -### Large files - -`.csv`, `.jsonl`, and `.ndjson` files are read line by line and fed through -the streaming resampler, so memory never scales with file size. (A JSON array -file is the exception: the whole document must be parsed to know where the -array ends, so it stays in memory.) `.parquet` files are read row group by row -group through the same streaming resampler. Output is written incrementally in -CSV, JSON (a valid, parseable array), or JSONL. - -```bash -# Stream a 100MB CSV to JSONL candles -ohlc -i huge.csv -f jsonl -o candles.jsonl - -# Resample a Parquet OHLCV file (row-group streaming) -ohlc -i data.parquet -f json -o candles.json - -# Pipe JSONL line by line (no buffering) -cat data.jsonl | ohlc --input-format jsonl -f jsonl -``` - -### Input formats - -The CLI supports CSV, JSON, and JSONL input. - -#### Renaming fields with `--map` +### Feeding data as-is with `--map` -When your input uses different field names (CCXT-style `timestamp` / `amount`, -a CSV header in a foreign order, arbitrary Parquet columns), pass `--map` with +The CLI `--map` flag is the record-form map from the library, as `field=sourceKey` entries separated by commas. It applies to CSV headers, JSON -object keys, and Parquet columns; tuple (array) records have no field names and -are unaffected. With `--map`, a CSV's first line is always treated as the -header. +object keys, and Parquet columns; tuple (array) records have no keys and are +unaffected. With `--map`, a CSV's first line is always treated as the header. ```bash # CCXT-style JSON objects: timestamp + amount ohlc -i data.json --map time=timestamp,volume=amount -# Foreign CSV header -ohlc -i data.csv --map time=timestamp,volume=amount - # Arbitrary Parquet columns ohlc -i data.parquet --map time=mytime,open=myopen,high=myhigh,low=mylow,close=myclose,volume=myvol ``` -#### CSV format -```csv -time,open,high,low,close,volume -1609459200000,100,105,95,102,1000 -1609459260000,102,107,101,106,1200 -``` +### Large files -#### JSON format -```json -[ - { - "time": 1609459200000, - "open": 100, - "high": 105, - "low": 95, - "close": 102, - "volume": 1000 - } -] -``` +`.csv`, `.jsonl`, and `.ndjson` files stream line by line through the +resampler, so memory never scales with file size. (A JSON array file is the +exception: the whole document must be parsed to know where the array ends.) +`.parquet` files stream row group by row group. Output is written +incrementally in CSV, JSON (a valid, parseable array), or JSONL. -#### JSONL format (one candle per line) -```json -{"time":1609459200000,"open":100,"high":105,"low":95,"close":102,"volume":1000} -{"time":1609459260000,"open":102,"high":107,"low":101,"close":106,"volume":1200} +```bash +# Stream a 100MB CSV to JSONL candles +ohlc -i huge.csv -f jsonl -o candles.jsonl + +# Pipe JSONL line by line (no buffering) +cat data.jsonl | ohlc --input-format jsonl -f jsonl ``` ### Pipe input -You can pipe data into the CLI from other commands. The format is detected -automatically, or you can specify it: +Pipe data from other commands; the format is detected automatically, or force +it with `--input-format`. Supported pipe input is JSON or CSV. ```bash -# Auto-detect format cat data.json | ohlc -cat data.csv | ohlc - -# Force a specific format -cat data.json | ohlc --input-format json cat data.csv | ohlc --input-format csv ``` -The CLI supports two kinds of pipe input: -1. JSON files or strings with OHLCV objects -2. CSV files or strings with a header (time, open, high, low, close, volume) - -### CLI examples - -```bash -# Resample 1-minute data to 5-minute candles -ohlc -i data.csv -b 60 -n 300 - -# Convert CSV to JSON -ohlc -i data.csv -f json - -# Pipe data and save to a file -cat data.csv | ohlc -o output.json - -# Resample with custom time frames and save as CSV -ohlc -i data.json -b 300 -n 3600 -f csv -o output.csv -``` - ## Contributors ๐Ÿ‘ค **Adil Shaikh (https://adils.me)** From 763203bd91d85dd7fd11750574ddf6815c0570e3 Mon Sep 17 00:00:00 2001 From: Adil Date: Thu, 27 Aug 2026 20:54:57 +0500 Subject: [PATCH 5/5] docs: note Parquet pipe input is file-only --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index ab3b4cc..06fdefd 100644 --- a/README.md +++ b/README.md @@ -351,7 +351,8 @@ cat data.jsonl | ohlc --input-format jsonl -f jsonl ### Pipe input Pipe data from other commands; the format is detected automatically, or force -it with `--input-format`. Supported pipe input is JSON or CSV. +it with `--input-format`. Supported pipe input is JSON or CSV (Parquet is +file-only via `-i`). ```bash cat data.json | ohlc