-Resample (inter-convert) trade, ticks or OHLCV data to different time frames +Turn trade, tick, or OHLCV data into clean candlestick charts on any time frame
@@ -18,45 +18,70 @@ Resample (inter-convert) trade, ticks or OHLCV data to different time frames
-- Typescript support -- CCXT support -- Single dependency -- Low time complexity grouping based aggregations -- Optional gap filling +## What it does + +`ohlc-resample` converts market data into the candlestick charts you want: + +- Combine raw ticks or trades into **OHLCV candles** (open, high, low, close, + volume), by time period or by a fixed number of ticks. +- Rebuild candles from one time frame to a coarser one (1-minute to 5-minute, + for example). +- Work with the common formats (CCXT-style arrays, JSON, CSV, JSONL, Parquet) + and stream very large datasets without running out of memory. +- Fill in missing candles so your chart has no gaps. ## Install ### CLI -macOS / Linux: - ```sh curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh ``` -Installs the `ohlc` CLI to `~/.local/bin`. If you don't already have a recent enough Node, the installer downloads one for you and uses it. Pin a specific version with `--version 2.0.0`. +Installs the `ohlc` command to `~/.local/bin` (and a Node runtime if needed). +Pin a version with `--version 2.0.0`. Uninstall with `--uninstall`. -To uninstall: +### Library ```sh -curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh -s -- --uninstall +npm install ohlc-resample # or pnpm add / yarn add / bun add ``` -### Library +Requires Node.js 26 or newer. -```sh -npm install ohlc-resample # or pnpm add / yarn add / bun add +## Quick start (CLI) + +```bash +# Install +curl -fsSL https://github.com/adiled/ohlc-resample/raw/main/install.sh | sh + +# Resample 1-minute candles in data.csv into 5-minute candles +ohlc -i data.csv -b 60 -n 300 + +# Resample a Parquet file and write JSON candles to a file +ohlc -i data.parquet -f json -o candles.json +``` + +**OR use directly** with npx, no install needed: + +```bash +npx ohlc-resample -i data.csv -b 60 -n 300 ``` -Requires Node.js โฅ26. +See the [CLI section](#cli) below for all options. -## Supported formats +## Supported input formats -- OHLCV (CCXT format) `[[time,open,high,low,close,volume]]` -- OHLCV JSON `[{time: number, open: number, high: number, low: number close: number, volume: number}]` -- Trade JSON `[{time: number, price: number, quantity: number}]` +- **OHLCV arrays** (CCXT-style) `[[time, open, high, low, close, volume], ...]` +- **OHLCV JSON objects** `[{ time, open, high, low, close, volume }, ...]` +- **Trade / tick JSON objects** `[{ time, price, quantity }, ...]` +- **CSV**, **JSON**, and **JSONL** files +- **Parquet** files +- **Arbitrary schemas**, when you supply a `map` (see [Feeding data as-is](#feeding-data-as-is-with-map)) -## Reference +Input times are epoch **milliseconds**. See [Types](#types). + +## Library usage ```typescript import { @@ -65,43 +90,29 @@ import { resampleTicksByCount, } from "ohlc-resample"; -// OHLCV resampled from 1 minute to 5 minute - -resampleOhlcv(objectOhlcv as IOHLCV[], { - baseTimeframe: 60, - newTimeframe: 5 * 60, -}); // return IOHLCV[] -resampleOhlcv(arrayOhlcv as OHLCV[], { - baseTimeframe: 60, - newTimeframe: 5 * 60, -}); // return OHLCV[] - -// Ticks grouped and resampled to 1m OHCLV -// option.includeLatestCandle is by default `true` -// options.fillGaps is by default `false` - -resampleTicksByTime(tickData as TradeTick[], { - timeframe: 60, - includeLatestCandle: false, - fillGaps: true, -}); // return IOHLCV[] +// OHLCV candles from 1 minute to 5 minutes +resampleOhlcv(objectOhlcv, { baseTimeframe: 60, newTimeframe: 5 * 60 }); // IOHLCV[] +resampleOhlcv(arrayOhlcv, { baseTimeframe: 60, newTimeframe: 5 * 60 }); // OHLCV[] -// Ticks grouped and resampled by every 5 ticks +// Ticks grouped into 1-minute OHLCV candles +resampleTicksByTime(tickData, { timeframe: 60, fillGaps: true }); // IOHLCV[] -resampleTicksByCount(tickData as TradeTick[], { tickCount: 5 }); // return IOHLCV[] +// Ticks grouped into candles of 5 ticks each +resampleTicksByCount(tickData, { tickCount: 5 }); // IOHLCV[] ``` -## Streaming & large data +Each function accepts an array or any sync iterable / generator. The result +uses the same shape as your input (tuples in, tuples out; objects in, objects +out). `resampleOhlcv` also accepts a binary `Float64Array` of interleaved +`[time, open, high, low, close, volume]` values, the fastest way to feed large +binary data. -The three functions above all accept a **sync iterable / generator** as well as -an array (materialized internally), and `resampleOhlcv` also accepts a binary -`Float64Array` of interleaved `[time, open, high, low, close, volume]` records. +## Streaming and large datasets -For **true streaming** โ large files, live feeds, or anything you don't want to -hold in memory โ use the async variants. They consume an `AsyncIterable` (a -Node `ReadableStream`, an async generator, `for await` sources) and return an -`AsyncGenerator` that emits each bucket as soon as it is safe. Memory use is -bounded by the active bucket window, never the whole input. +The async variants read from any async source (a `ReadableStream`, an async +generator, anything you can `for await` over) and emit each candle as soon as +it is ready, so memory stays bounded by the active time window rather than the +input size. ```typescript import { @@ -114,82 +125,60 @@ import { for await (const candle of resampleOhlcvAsync(readableStream, { baseTimeframe: 60, newTimeframe: 300, -})) { - // candle is a completed OHLCV bucket -} +})) { } // Stream ticks into time buckets for await (const candle of resampleTicksByTimeAsync(tickSource, { timeframe: 60, - includeLatestCandle: false, fillGaps: true, -})) { - // ... -} - -// Stream ticks into count buckets (O(tickCount) memory) -for await (const candle of resampleTicksByCountAsync(tickSource, { tickCount: 5 })) { - // ... -} -``` +})) { } -**Sorted input.** The array API sorts a copy for you; a streaming API cannot -buffer to sort, so pass data **ascending by time** unless you use the healing -window below. +// Stream ticks into count buckets (memory scales with tickCount) +for await (const candle of resampleTicksByCountAsync(tickSource, { tickCount: 5 })) { } +``` -**Out-of-order healing.** With `outOfOrderMs > 0`, the stream keeps each bucket -open for that many milliseconds of wall-clock time, so delayed or out-of-order -records that land inside the window are folded into the correct bucket on the -fly โ no global sort, and no re-reading already-emitted buckets. `outOfOrderMs -= 0` (default) is exact for pre-sorted input and emits as the stream passes. +**Sorted input.** The array functions sort a copy for you. A stream cannot +buffer the whole input, so pass data **ascending by time** unless you use the +healing window. Set `outOfOrderMs` to a number of milliseconds and the stream +keeps each bucket open that long, folding delayed or out-of-order records into +the correct bucket as they arrive. `outOfOrderMs = 0` (default) is exact for +already-sorted input. ### Parquet files -The async variants also accept a **file path** to a Parquet file. Rows are -streamed **row-group by row-group** (memory is bounded by the largest row -group, not the file), so large columnar datasets don't blow up memory. This is -the "foot in the door" before the future native engine โ the reader is the -pure-JS, zero-dependency `hyparquet` package. +The async variants also accept a **file path** to a Parquet file, read one row +group at a time so memory stays bounded by the largest row group. ```typescript -// resample a Parquet OHLCV file for await (const candle of resampleOhlcvAsync("data.parquet", { baseTimeframe: 60, newTimeframe: 300, -})) { - // candle is an OHLCV tuple -} +})) { } // OHLCV tuple -// resample a Parquet tick file for await (const candle of resampleTicksByTimeAsync("ticks.parquet", { timeframe: 60, -})) { - // candle is an IOHLCV object -} +})) { } // IOHLCV object ``` -Column names are read by **exact canonical name** (`time`, `open`, `high`, -`low`, `close`, `volume`, and `time`/`price`/`quantity` for ticks). Timestamps -in any unit (ms / ยตs / ns / days) are converted to epoch-milliseconds. If a -required column is missing, the call throws; OHLCV `volume` is optional and -defaults to `0`. Any other column layout must be supplied with the `map` -option below. Parquet (file) input is supported **only on the async variants** -โ the sync functions stay array/iterable-only. +Parquet columns are read by exact canonical name (`time`, `open`, `high`, +`low`, `close`, `volume` for OHLCV; `time` / `price` / `quantity` for ticks). +Timestamps in any unit (ms, microseconds, nanoseconds, days) become +milliseconds automatically. Missing required columns throw; OHLCV `volume` is +optional and defaults to `0`. Parquet works only with the async variants. -### Per-record `map` option +### Feeding data as-is with `map` -The async variants take a `map` option that translates **each input record** -into canonical OHLCV. It works uniformly across any named-schema input โ -Parquet rows, CSV header rows, JSON objects โ and is how you adapt foreign -schemas (CCXT's `timestamp`/`amount`, arbitrary Parquet columns, etc.) instead -of relying on name guessing. Two shapes are accepted: +`map` removes the need to pre-transform data before passing it in. Feed +records in whatever schema you already have (CCXT's `timestamp` / `amount`, +foreign Parquet columns, and so on) and `map` tells the resampler which keys to +read. It applies uniformly to Parquet rows, CSV headers, and JSON objects. Two +shapes are accepted: -1. **Record form** โ keys are canonical IOHLCV fields, values are the keys to - read from each input record. Fields absent from the map use the canonical - key directly: +1. **Record form**, mapping canonical OHLCV fields to the keys in your records. + Fields you leave out use the canonical key directly: ```typescript - // read `time` from `timestamp`, `volume` from `amount`; the rest stay canonical + // read `time` from `timestamp` and `volume` from `amount`; the rest stay canonical for await (const candle of resampleOhlcvAsync(readableStream, { baseTimeframe: 60, newTimeframe: 300, @@ -204,7 +193,7 @@ of relying on name guessing. Two shapes are accepted: })) { } ``` -2. **Function form** โ a full transform `(record) => IOHLCV` for complete +2. **Function form**, a full transform `(record) => IOHLCV` for complete control: ```typescript @@ -218,28 +207,22 @@ of relying on name guessing. Two shapes are accepted: })) { } ``` -Ticks accept the same shapes over `time`/`price`/`quantity` (e.g. `map: -{ time: 'timestamp', quantity: 'amount' }`). Positional inputs โ `OHLCV` -tuples and `Float64Array` โ have no keys to map and are unaffected. The sync -functions take canonical arrays, so `map` is only relevant to the async -variants (file paths and record-shaped streams). +Ticks accept the same shapes over `time` / `price` / `quantity` (for example +`map: { time: 'timestamp', quantity: 'amount' }`). Positional inputs (OHLCV +tuples and `Float64Array`) have no keys to read and are unaffected. The sync +functions take canonical arrays, so `map` applies only to the async variants. ## Module format -`ohlc-resample` is **ESM-first with a CommonJS wrapper** (`"type": "module"`). -Use `import` for the full API. `require('ohlc-resample')` also works: the build -emits `dist/index.cjs`, a one-line re-export that loads the ESM build via -Node's synchronous `require(esm)` (stable since 23.7), so both entry points -resolve to the **same module instance** โ there's no dual-package hazard and -no CommonJS build to maintain. The Parquet reader is ESM-native -(`hyparquet` has no CommonJS build), which is why the package is ESM-first; a -thin wrapper keeps `require()` consumers working. +The package is **ESM-first with a CommonJS wrapper** (`"type": "module"`). Use +`import` for the full API; `require('ohlc-resample')` also works and resolves +to the same module instance. ## Types ```typescript export type IOHLCV = { - time: number; + time: number; // epoch milliseconds open: number; high: number; low: number; @@ -247,81 +230,45 @@ export type IOHLCV = { volume: number; }; -export type OHLCV = [number, number, number, number, number, number]; +export type OHLCV = [number, number, number, number, number, number]; // [time, open, high, low, close, volume] export type TradeTick = { + time: number; // epoch milliseconds price: number; quantity: number; - time: number; }; ``` -**Note:** Input time for all above types must be in milliseconds +**Note:** input times for all of the above must be in milliseconds. ## Examples -**Resample CCXT (Object) OHLCV based on timeframe** +**Resample CCXT (object) OHLCV to a coarser time frame** ```typescript import { resampleOhlcv } from "ohlc-resample"; const link_btc_1m = [ - { - time: 1563625680000, - open: 0.00024824, - high: 0.00024851, - low: 0.00024798, - close: 0.00024831, - volume: 2264, - }, - { - time: 1563625740000, - open: 0.00024817, - high: 0.00024832, - low: 0.00024795, - close: 0.00024828, - volume: 3145, - }, + { time: 1563625680000, open: 0.00024824, high: 0.00024851, low: 0.00024798, close: 0.00024831, volume: 2264 }, + { time: 1563625740000, open: 0.00024817, high: 0.00024832, low: 0.00024795, close: 0.00024828, volume: 3145 }, ]; -const baseTimeframe = 60; // 60 seconds -const newTimeframe = 120; // 120 seconds - -// Candles made up of ticks within 2 minute timeframes - +// Candles built from the ticks within each 2-minute window const link_btc_2m = resampleOhlcv(link_btc_1m, { - baseTimeframe, - newTimeframe, + baseTimeframe: 60, + newTimeframe: 120, }); ``` -**Resample ticks to OHLCV based on tick count** +**Resample ticks to OHLCV candles by tick count** ```typescript import { resampleTicksByCount, TradeTick } from "ohlc-resample"; const adabnb_trades = [ - { - time: "1564502620356", - side: "sell", - quantity: "4458", - price: "0.00224", - tradeId: "1221272", - }, - { - time: "1564503133949", - side: "sell", - quantity: "3480", - price: "0.002242", - tradeId: "1221273", - }, - { - time: "1564503134553", - side: "buy", - quantity: "51", - price: "0.002248", - tradeId: "1221274", - }, + { time: "1564502620356", side: "sell", quantity: "4458", price: "0.00224", tradeId: "1221272" }, + { time: "1564503133949", side: "sell", quantity: "3480", price: "0.002242", tradeId: "1221273" }, + { time: "1564503134553", side: "buy", quantity: "51", price: "0.002248", tradeId: "1221274" }, ]; const airbnb_ticks: TradeTick[] = adabnb_trades.map((trade: any) => ({ @@ -330,29 +277,13 @@ const airbnb_ticks: TradeTick[] = adabnb_trades.map((trade: any) => ({ price: Number(trade.price), })); -// Candles made up of two ticks - -const tickChart = resampleTicksByCount(airbnb_ticks, { - tickCount: 2, -}); +// Candles built from two ticks each +const tickChart = resampleTicksByCount(airbnb_ticks, { tickCount: 2 }); ``` -## CLI Usage - -The package includes a command-line interface for resampling OHLCV data between timeframes and file formats. +## CLI -### Basic Usage - -```bash -# Resample CSV file with default timeframes (1m -> 5m) -ohlc-resample -i input.csv - -# Resample JSON file with custom timeframes -ohlc-resample -i input.json -b 60 -n 300 - -# Save output to file with specific format -ohlc-resample -i input.csv -o output.json -f json -``` +The installed command is `ohlc` (the npm package binary is `ohlc-resample`). ### Options @@ -366,7 +297,7 @@ Options: -s, --shape