From 0bfb00ec2318c6b5ad1eaa3892fcd2bd68524a31 Mon Sep 17 00:00:00 2001 From: Tyler Beason Date: Tue, 8 Sep 2026 14:57:31 -0500 Subject: [PATCH 1/3] Add StatType/TradingEvent enums, InstrumentClass.INDEX, and record flag constants Aligns with Databento's CME normalization changes (2026-07) and the dbn 0.69 spec: - StatType enum (UInt16) incl. UPPER_PRICE_LIMIT=17 / LOWER_PRICE_LIMIT=18, safe_stat_type - TradingEvent enum incl. IMPLIED_MATCHING_ON=3 / IMPLIED_MATCHING_OFF=4, safe_trading_event - InstrumentClass.INDEX ('I'), previously decoded as OTHER - F_LAST/F_TOB/F_SNAPSHOT/F_MBP/F_BAD_TS_RECV/F_MAYBE_BAD_BOOK/F_PUBLISHER_SPECIFIC + has_flag - Round-trip tests for per-leg definitions, standalone F_LAST MBO records, price-limit stats, implied-matching status, FX-spot/index classes Record struct fields are unchanged (stat_type/trading_event remain raw UInt16). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01YBr6EZa1vHrJmPBoecSjRa --- CHANGELOG.md | 30 + docs/src/api/enums.md | 52 ++ src/DatabentoBinaryEncoding.jl | 268 +++--- src/types.jl | 1491 ++++++++++++++++++-------------- test/runtests.jl | 89 +- test/test_cme_2026_07.jl | 229 +++++ 6 files changed, 1315 insertions(+), 844 deletions(-) create mode 100644 test/test_cme_2026_07.jl diff --git a/CHANGELOG.md b/CHANGELOG.md index 0213233c..cc68c894 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,36 @@ All notable changes to DatabentoBinaryEncoding.jl are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [Unreleased] + +### Added + +- **`StatType` enum** (`UInt16`, matches the DBN spec) for interpreting + `StatMsg.stat_type`, including `UPPER_PRICE_LIMIT = 17` and + `LOWER_PRICE_LIMIT = 18`, which CME GLBX.MDP3 publishes since Databento's + 2026-07 normalization change, plus the block-volume, indicative-close, + market-wide-circuit-breaker, auction-collar and venue-specific codes. + `safe_stat_type(raw)` maps unrecognized codes to `StatType.UNKNOWN`. The + struct field stays a raw `UInt16` (no breaking change). +- **`TradingEvent` enum** (`UInt16`) for `StatusMsg.trading_event`: + `NONE`, `NO_CANCEL`, `CHANGE_TRADING_SESSION`, `IMPLIED_MATCHING_ON = 3`, + `IMPLIED_MATCHING_OFF = 4` (the latter two are emitted by CME since 2026-07). + `safe_trading_event(raw)` maps unrecognized codes to `TradingEvent.NONE`. +- **`InstrumentClass.INDEX`** (`'I'`), present in the DBN spec but previously + missing here; such definitions decoded as `InstrumentClass.OTHER` with a + warning. (`FX_SPOT = 'X'`, which CME now uses for FX spot instruments instead + of `FUTURE`, was already supported.) +- **Record flag constants** `F_LAST` (128), `F_TOB`, `F_SNAPSHOT`, `F_MBP`, + `F_BAD_TS_RECV`, `F_MAYBE_BAD_BOOK`, `F_PUBLISHER_SPECIFIC` and a `has_flag` + helper. Relevant to the 2026-07 CME change where `F_LAST` moved from the + final MBO book update onto a standalone `action = Action.NONE` record with + `price = UNDEF_PRICE`, `size = 0`; that record already decoded, exported + (`NaN` price) and printed correctly, and is now covered by tests. +- Test coverage (`test/test_cme_2026_07.jl`) for the 2026-07 CME normalization + shapes: one definition record per strategy leg, the standalone `F_LAST` MBO + record, price-limit statistics, implied-matching status events and FX-spot / + index definitions. + ## [0.1.6] - 2026-06-24 ### Changed diff --git a/docs/src/api/enums.md b/docs/src/api/enums.md index 50502c94..bdc1e96d 100644 --- a/docs/src/api/enums.md +++ b/docs/src/api/enums.md @@ -125,12 +125,64 @@ codes: - `CALL` (`'C'`) - Call option - `PUT` (`'P'`) - Put option - `FUTURE` (`'F'`) - Futures contract +- `INDEX` (`'I'`) - Index - `BOND` (`'B'`) - Fixed income - `FX_SPOT` (`'X'`) - Foreign exchange spot - `COMMODITY_SPOT` (`'Y'`) - Commodity spot - `MIXED_SPREAD` (`'M'`), `FUTURE_SPREAD` (`'S'`), `OPTION_SPREAD` (`'T'`) - Spreads - `OTHER` (`'?'`), `UNKNOWN_0`, `UNKNOWN_45` - Fallbacks for unclassified instruments +## Statistic Types + +```@docs +StatType +safe_stat_type +``` + +`StatMsg.stat_type` is stored as a raw `UInt16`; use `StatType` to interpret it, +e.g. `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)` or +`safe_stat_type(r.stat_type)`. Notable values: `OPENING_PRICE` (1), +`SETTLEMENT_PRICE` (3), `OPEN_INTEREST` (9), `CLOSE_PRICE` (11), +`UPPER_PRICE_LIMIT` (17) and `LOWER_PRICE_LIMIT` (18) — the last two are the +session price limits CME GLBX.MDP3 publishes since 2026-07. + +## Trading Events + +```@docs +TradingEvent +safe_trading_event +``` + +`StatusMsg.trading_event` is stored as a raw `UInt16`; `TradingEvent` names the +codes: `NONE` (0), `NO_CANCEL` (1), `CHANGE_TRADING_SESSION` (2), +`IMPLIED_MATCHING_ON` (3), `IMPLIED_MATCHING_OFF` (4). + +## Record Flags + +```@docs +F_LAST +F_TOB +F_SNAPSHOT +F_MBP +F_BAD_TS_RECV +F_MAYBE_BAD_BOOK +F_PUBLISHER_SPECIFIC +has_flag +``` + +The `flags` byte on MBO/MBP/trade/BBO records is a bit field. `F_LAST` (128) +marks the last record of a venue event. Since 2026-07, CME MBO carries it on a +standalone `action = Action.NONE` record with `price = UNDEF_PRICE` and +`size = 0`, so book builders should apply updates on every record and treat the +`F_LAST` record purely as an event boundary: + +```julia +foreach_mbo("glbx.mbo.dbn.zst") do r + r.action == Action.NONE || apply_to_book!(book, r) + has_flag(r.flags, F_LAST) && on_event_complete!(book) +end +``` + ## Usage Examples ### Working with Schemas diff --git a/src/DatabentoBinaryEncoding.jl b/src/DatabentoBinaryEncoding.jl index 655d9f9a..f7ac875f 100644 --- a/src/DatabentoBinaryEncoding.jl +++ b/src/DatabentoBinaryEncoding.jl @@ -1,133 +1,135 @@ -""" - DatabentoBinaryEncoding - -Julia implementation of the Databento Binary Encoding (DBN) format for normalized market data. - -# Overview - -DatabentoBinaryEncoding.jl provides complete support for reading and writing DBN v3 format files with: -- Efficient streaming support for large files -- Automatic Zstd compression/decompression -- All DBN v3 message types -- Timestamp utilities with nanosecond precision -- Price conversion utilities with fixed-point arithmetic - -# Main Functions - -## Reading Data -- `read_dbn(filename)`: Read entire file into memory -- `DBNStream(filename)`: Memory-efficient streaming iterator -- `DBNDecoder(filename)`: Low-level decoder with manual control - -## Writing Data -- `write_dbn(filename, metadata, records)`: Write complete file -- `DBNStreamWriter(filename, dataset, schema)`: Real-time streaming writer -- `DBNEncoder(io, metadata)`: Low-level encoder - -## Compression -- `compress_dbn_file(input, output)`: Compress single file -- `compress_daily_files(date, directory)`: Batch compress files - -## Format Conversion -- `dbn_to_csv(input, output)`: Convert DBN to CSV -- `dbn_to_json(input, output)`: Convert DBN to JSON -- `dbn_to_parquet(input, output)`: Convert DBN to Parquet -- `json_to_dbn(input, output)`: Convert JSON to DBN -- `parquet_to_dbn(input, output)`: Convert Parquet to DBN -- `csv_to_dbn(input, output)`: Convert CSV to DBN - -## Utilities -- `price_to_float(price)` / `float_to_price(value)`: Price conversions -- `ts_to_datetime(ts)` / `datetime_to_ts(dt)`: Timestamp conversions -- `DBNTimestamp(ns)`: High-precision timestamp handling - -# Example Usage - -```julia -using DatabentoBinaryEncoding -# Or, for terser internal references: `import DatabentoBinaryEncoding as DBN` - -# Reading data -records = read_dbn("data.dbn") -for record in DBNStream("large_file.dbn.zst") - process(record) -end - -# Writing data -metadata = Metadata(3, "XNAS", Schema.TRADES, start_ts, end_ts, - length(records), SType.RAW_SYMBOL, SType.RAW_SYMBOL, - false, symbols, [], [], []) -write_dbn("output.dbn", metadata, records) - -# Streaming writer -writer = DBNStreamWriter("live.dbn", "XNAS", Schema.TRADES) -write_record!(writer, trade_msg) -close_writer!(writer) -``` - -# Supported Message Types - -- Market Data: `MBOMsg`, `TradeMsg`, `MBP1Msg`, `MBP10Msg`, `OHLCVMsg` -- Consolidated: `CMBP1Msg`, `CBBO1sMsg`, `CBBO1mMsg`, `TCBBOMsg`, `BBO1sMsg`, `BBO1mMsg` -- Status: `StatusMsg`, `ImbalanceMsg`, `StatMsg` -- System: `ErrorMsg`, `SymbolMappingMsg`, `SystemMsg` -- Definition: `InstrumentDefMsg` - -See the [DBN specification](https://databento.com/docs/standards-and-conventions/databento-binary-encoding) -for complete format documentation. -""" -module DatabentoBinaryEncoding - -# All using statements at the top -using Dates -using CodecZstd -using TranscodingStreams -using EnumX -using DataFrames -using CSV -using DuckDB -using DBInterface -using JSON3 -using StructTypes - - -# Include all the component files -include("types.jl") -include("messages.jl") -include("show.jl") -include("buffered_io.jl") -include("decode.jl") -include("encode.jl") -include("streaming.jl") -include("replay.jl") -include("export.jl") -include("symbols.jl") -include("import.jl") - -# Exports -export DBNDecoder, DBNEncoder, read_dbn, read_dbn_with_metadata, read_dbn_typed, write_dbn -export read_trades, read_mbo, read_mbp1, read_mbp10, read_tbbo # Market depth readers -export read_ohlcv, read_ohlcv_1s, read_ohlcv_1m, read_ohlcv_1h, read_ohlcv_1d # OHLCV readers -export read_cmbp1, read_cbbo1s, read_cbbo1m, read_tcbbo, read_bbo1s, read_bbo1m # Consolidated/BBO readers -export Metadata, DBNHeader, RecordHeader, DBNTimestamp -export MBOMsg, TradeMsg, MBP1Msg, MBP10Msg, OHLCVMsg, StatusMsg, ImbalanceMsg, StatMsg -export CMBP1Msg, CBBO1sMsg, CBBO1mMsg, TCBBOMsg, BBO1sMsg, BBO1mMsg -export ErrorMsg, SymbolMappingMsg, SystemMsg, InstrumentDefMsg -export DBNStream, DBNStreamWriter, write_record!, close_writer! -export foreach_record, foreach_record_with_control, foreach_trade, foreach_mbo, foreach_mbp1, foreach_mbp10, foreach_tbbo # Market depth streaming -export record_type_for_dbn_schema # Schema -> concrete record type -export foreach_ohlcv, foreach_ohlcv_1s, foreach_ohlcv_1m, foreach_ohlcv_1h, foreach_ohlcv_1d # OHLCV streaming -export foreach_cmbp1, foreach_cbbo1s, foreach_cbbo1m, foreach_tcbbo, foreach_bbo1s, foreach_bbo1m # Consolidated/BBO streaming -export replay_dbn, replay_records # Timestamp-paced replay -export compress_dbn_file, compress_daily_files -export Schema, Compression, Encoding, SType, RType, Action, Side, InstrumentClass -export price_to_float, float_to_price, ts_to_datetime, datetime_to_ts, ts_to_date_time, date_time_to_ts, to_nanoseconds -export record_length_bytes -export DBN_VERSION, FIXED_PRICE_SCALE, UNDEF_PRICE, UNDEF_ORDER_SIZE, UNDEF_TIMESTAMP -export BidAskPair, VersionUpgradePolicy, DatasetCondition -export write_header, read_header!, write_record, read_record, finalize_encoder -export dbn_to_csv, dbn_to_json, dbn_to_parquet, records_to_dataframe -export symbol_map, symbol_for, add_symbol_column! -export json_to_dbn, parquet_to_dbn, csv_to_dbn - -end # module DatabentoBinaryEncoding \ No newline at end of file +""" + DatabentoBinaryEncoding + +Julia implementation of the Databento Binary Encoding (DBN) format for normalized market data. + +# Overview + +DatabentoBinaryEncoding.jl provides complete support for reading and writing DBN v3 format files with: +- Efficient streaming support for large files +- Automatic Zstd compression/decompression +- All DBN v3 message types +- Timestamp utilities with nanosecond precision +- Price conversion utilities with fixed-point arithmetic + +# Main Functions + +## Reading Data +- `read_dbn(filename)`: Read entire file into memory +- `DBNStream(filename)`: Memory-efficient streaming iterator +- `DBNDecoder(filename)`: Low-level decoder with manual control + +## Writing Data +- `write_dbn(filename, metadata, records)`: Write complete file +- `DBNStreamWriter(filename, dataset, schema)`: Real-time streaming writer +- `DBNEncoder(io, metadata)`: Low-level encoder + +## Compression +- `compress_dbn_file(input, output)`: Compress single file +- `compress_daily_files(date, directory)`: Batch compress files + +## Format Conversion +- `dbn_to_csv(input, output)`: Convert DBN to CSV +- `dbn_to_json(input, output)`: Convert DBN to JSON +- `dbn_to_parquet(input, output)`: Convert DBN to Parquet +- `json_to_dbn(input, output)`: Convert JSON to DBN +- `parquet_to_dbn(input, output)`: Convert Parquet to DBN +- `csv_to_dbn(input, output)`: Convert CSV to DBN + +## Utilities +- `price_to_float(price)` / `float_to_price(value)`: Price conversions +- `ts_to_datetime(ts)` / `datetime_to_ts(dt)`: Timestamp conversions +- `DBNTimestamp(ns)`: High-precision timestamp handling + +# Example Usage + +```julia +using DatabentoBinaryEncoding +# Or, for terser internal references: `import DatabentoBinaryEncoding as DBN` + +# Reading data +records = read_dbn("data.dbn") +for record in DBNStream("large_file.dbn.zst") + process(record) +end + +# Writing data +metadata = Metadata(3, "XNAS", Schema.TRADES, start_ts, end_ts, + length(records), SType.RAW_SYMBOL, SType.RAW_SYMBOL, + false, symbols, [], [], []) +write_dbn("output.dbn", metadata, records) + +# Streaming writer +writer = DBNStreamWriter("live.dbn", "XNAS", Schema.TRADES) +write_record!(writer, trade_msg) +close_writer!(writer) +``` + +# Supported Message Types + +- Market Data: `MBOMsg`, `TradeMsg`, `MBP1Msg`, `MBP10Msg`, `OHLCVMsg` +- Consolidated: `CMBP1Msg`, `CBBO1sMsg`, `CBBO1mMsg`, `TCBBOMsg`, `BBO1sMsg`, `BBO1mMsg` +- Status: `StatusMsg`, `ImbalanceMsg`, `StatMsg` +- System: `ErrorMsg`, `SymbolMappingMsg`, `SystemMsg` +- Definition: `InstrumentDefMsg` + +See the [DBN specification](https://databento.com/docs/standards-and-conventions/databento-binary-encoding) +for complete format documentation. +""" +module DatabentoBinaryEncoding + +# All using statements at the top +using Dates +using CodecZstd +using TranscodingStreams +using EnumX +using DataFrames +using CSV +using DuckDB +using DBInterface +using JSON3 +using StructTypes + + +# Include all the component files +include("types.jl") +include("messages.jl") +include("show.jl") +include("buffered_io.jl") +include("decode.jl") +include("encode.jl") +include("streaming.jl") +include("replay.jl") +include("export.jl") +include("symbols.jl") +include("import.jl") + +# Exports +export DBNDecoder, DBNEncoder, read_dbn, read_dbn_with_metadata, read_dbn_typed, write_dbn +export read_trades, read_mbo, read_mbp1, read_mbp10, read_tbbo # Market depth readers +export read_ohlcv, read_ohlcv_1s, read_ohlcv_1m, read_ohlcv_1h, read_ohlcv_1d # OHLCV readers +export read_cmbp1, read_cbbo1s, read_cbbo1m, read_tcbbo, read_bbo1s, read_bbo1m # Consolidated/BBO readers +export Metadata, DBNHeader, RecordHeader, DBNTimestamp +export MBOMsg, TradeMsg, MBP1Msg, MBP10Msg, OHLCVMsg, StatusMsg, ImbalanceMsg, StatMsg +export CMBP1Msg, CBBO1sMsg, CBBO1mMsg, TCBBOMsg, BBO1sMsg, BBO1mMsg +export ErrorMsg, SymbolMappingMsg, SystemMsg, InstrumentDefMsg +export DBNStream, DBNStreamWriter, write_record!, close_writer! +export foreach_record, foreach_record_with_control, foreach_trade, foreach_mbo, foreach_mbp1, foreach_mbp10, foreach_tbbo # Market depth streaming +export record_type_for_dbn_schema # Schema -> concrete record type +export foreach_ohlcv, foreach_ohlcv_1s, foreach_ohlcv_1m, foreach_ohlcv_1h, foreach_ohlcv_1d # OHLCV streaming +export foreach_cmbp1, foreach_cbbo1s, foreach_cbbo1m, foreach_tcbbo, foreach_bbo1s, foreach_bbo1m # Consolidated/BBO streaming +export replay_dbn, replay_records # Timestamp-paced replay +export compress_dbn_file, compress_daily_files +export Schema, Compression, Encoding, SType, RType, Action, Side, InstrumentClass +export StatType, TradingEvent, safe_stat_type, safe_trading_event +export F_LAST, F_TOB, F_SNAPSHOT, F_MBP, F_BAD_TS_RECV, F_MAYBE_BAD_BOOK, F_PUBLISHER_SPECIFIC, has_flag +export price_to_float, float_to_price, ts_to_datetime, datetime_to_ts, ts_to_date_time, date_time_to_ts, to_nanoseconds +export record_length_bytes +export DBN_VERSION, FIXED_PRICE_SCALE, UNDEF_PRICE, UNDEF_ORDER_SIZE, UNDEF_TIMESTAMP +export BidAskPair, VersionUpgradePolicy, DatasetCondition +export write_header, read_header!, write_record, read_record, finalize_encoder +export dbn_to_csv, dbn_to_json, dbn_to_parquet, records_to_dataframe +export symbol_map, symbol_for, add_symbol_column! +export json_to_dbn, parquet_to_dbn, csv_to_dbn + +end # module DatabentoBinaryEncoding diff --git a/src/types.jl b/src/types.jl index d5685e63..418e4e6a 100644 --- a/src/types.jl +++ b/src/types.jl @@ -1,667 +1,824 @@ -# DBN types, enums, and data structures - -# Constants -"""The current DBN format version supported by this implementation.""" -const DBN_VERSION = 3 - -"""Fixed-length symbol string size for DBN v2/v3 metadata and records.""" -const SYMBOL_CSTR_LEN = 71 - -"""Fixed-point price scaling factor for converting between integer and float prices.""" -const FIXED_PRICE_SCALE = Int32(1_000_000_000) - -"""Sentinel value indicating an undefined or missing price.""" -const UNDEF_PRICE = typemax(Int64) - -"""Sentinel value indicating an undefined or missing order size.""" -const UNDEF_ORDER_SIZE = typemax(UInt32) - -"""Sentinel value indicating an undefined or missing timestamp.""" -const UNDEF_TIMESTAMP = typemax(Int64) - -"""Multiplier for converting the length field to bytes in record headers.""" -const LENGTH_MULTIPLIER = 4 - -# Enums using EnumX for better namespace management - -""" - Schema - -DBN data schemas representing different types of market data. - -# Values -- `MBO`: Market-by-order data -- `MBP_1`: Market-by-price with book depth 1 (TBBO) -- `MBP_10`: Market-by-price with book depth 10 -- `TBBO`: Top-of-book bid/offer -- `TRADES`: Trade messages only -- `OHLCV_1S`: OHLCV data at 1-second intervals -- `OHLCV_1M`: OHLCV data at 1-minute intervals -- `OHLCV_1H`: OHLCV data at 1-hour intervals -- `OHLCV_1D`: OHLCV data at 1-day intervals -- `DEFINITION`: Instrument definition data -- `STATISTICS`: Market statistics -- `STATUS`: Exchange status messages -- `IMBALANCE`: Order imbalance data -- `CBBO`: Consolidated best bid/offer -- `CBBO_1S`: Consolidated BBO at 1-second intervals -- `CBBO_1M`: Consolidated BBO at 1-minute intervals -- `CMBP_1`: Consolidated market-by-price depth 1 -- `TCBBO`: Trade-consolidated BBO -- `BBO_1S`: BBO at 1-second intervals -- `BBO_1M`: BBO at 1-minute intervals -""" -@enumx Schema::UInt16 begin - MBO = 0 - MBP_1 = 1 - MBP_10 = 2 - TBBO = 3 - TRADES = 4 - OHLCV_1S = 5 - OHLCV_1M = 6 - OHLCV_1H = 7 - OHLCV_1D = 8 - DEFINITION = 9 - STATISTICS = 10 - STATUS = 11 - IMBALANCE = 12 - CBBO = 13 - CBBO_1S = 14 - CBBO_1M = 15 - CMBP_1 = 16 - TCBBO = 17 - BBO_1S = 18 - BBO_1M = 19 - MIX = 0xFFFF -end - -""" - Compression - -Compression algorithms supported for DBN files. - -# Values -- `NONE`: No compression -- `ZSTD`: Zstandard compression -""" -@enumx Compression::UInt8 begin - NONE = 0 - ZSTD = 1 -end - -""" - Encoding - -Output encoding formats for market data. - -# Values -- `DBN`: Databento Binary Encoding -- `CSV`: Comma-separated values -- `JSON`: JavaScript Object Notation -""" -@enumx Encoding::UInt8 begin - DBN = 0 - CSV = 1 - JSON = 2 -end - -""" - SType - -Symbol types for identifying instruments in DBN data. Numeric values match the -official Databento DBN spec — wire-encoded as a `UInt8` in metadata and in -`SymbolMappingMsg` (v2+). - -# Values -- `INSTRUMENT_ID = 0`: Numeric instrument identifier -- `RAW_SYMBOL = 1`: Raw symbol string from exchange -- `SMART = 2`: Deprecated alias (was split into `CONTINUOUS` and `PARENT`) -- `CONTINUOUS = 3`: Continuous contract symbol -- `PARENT = 4`: Parent symbol for derived instruments (e.g. `SPXW.OPT`) -- `NASDAQ_SYMBOL = 5`: Nasdaq-specific symbol -- `CMS_SYMBOL = 6`: CMS symbol -- `ISIN = 7`: ISO 6166 International Securities Identification Number -- `US_CODE = 8`: US Code (CUSIP-style) -- `BBG_COMP_ID = 9`: Bloomberg composite ID -- `BBG_COMP_TICKER = 10`: Bloomberg composite ticker -- `FIGI = 11`: OpenFIGI identifier -- `FIGI_TICKER = 12`: OpenFIGI ticker -- `UNDEF = 255`: Unset/undefined sentinel (`0xFF`). Databento wire-encodes an - unset `stype` as `0xFF`; this member lets non-nullable `stype` fields (e.g. - in `SymbolMappingMsg`) represent and round-trip that sentinel. Metadata's - nullable `stype_in` decodes `0xFF` to `nothing` instead. -""" -@enumx SType::UInt8 begin - INSTRUMENT_ID = 0 - RAW_SYMBOL = 1 - SMART = 2 # deprecated (kept for round-trip with v1 wire data) - CONTINUOUS = 3 - PARENT = 4 - NASDAQ_SYMBOL = 5 - CMS_SYMBOL = 6 - ISIN = 7 - US_CODE = 8 - BBG_COMP_ID = 9 - BBG_COMP_TICKER = 10 - FIGI = 11 - FIGI_TICKER = 12 - UNDEF = 255 # 0xFF "unset" sentinel; see docstring -end - -""" - RType - -Record types for different kinds of market data messages in DBN format. - -# Values -- `MBP_0_MSG`: Trades (book depth 0) -- `MBP_1_MSG`: TBBO/MBP-1 (book depth 1) -- `MBP_10_MSG`: MBP-10 (book depth 10) -- `STATUS_MSG`: Exchange status record -- `INSTRUMENT_DEF_MSG`: Instrument definition record -- `IMBALANCE_MSG`: Order imbalance record -- `ERROR_MSG`: Error record from live gateway -- `SYMBOL_MAPPING_MSG`: Symbol mapping record from live gateway -- `SYSTEM_MSG`: Non-error record from live gateway -- `STAT_MSG`: Statistics record from publisher -- `OHLCV_1S_MSG`: OHLCV at 1-second cadence -- `OHLCV_1M_MSG`: OHLCV at 1-minute cadence -- `OHLCV_1H_MSG`: OHLCV at hourly cadence -- `OHLCV_1D_MSG`: OHLCV at daily cadence -- `MBO_MSG`: Market-by-order record -- `CMBP_1_MSG`: Consolidated market-by-price with book depth 1 -- `CBBO_1S_MSG`: Consolidated market-by-price with book depth 1 at 1-second cadence -- `CBBO_1M_MSG`: Consolidated market-by-price with book depth 1 at 1-minute cadence -- `TCBBO_MSG`: Consolidated market-by-price with book depth 1 (trades only) -- `BBO_1S_MSG`: Market-by-price with book depth 1 at 1-second cadence -- `BBO_1M_MSG`: Market-by-price with book depth 1 at 1-minute cadence -""" -@enumx RType::UInt8 begin - MBP_0_MSG = 0x00 # Trades (book depth 0) - MBP_1_MSG = 0x01 # TBBO/MBP-1 (book depth 1) - MBP_10_MSG = 0x0A # MBP-10 (book depth 10) - STATUS_MSG = 0x12 # Exchange status record - INSTRUMENT_DEF_MSG = 0x13 # Instrument definition record - IMBALANCE_MSG = 0x14 # Order imbalance record - ERROR_MSG = 0x15 # Error record from live gateway - SYMBOL_MAPPING_MSG = 0x16 # Symbol mapping record from live gateway - SYSTEM_MSG = 0x17 # Non-error record from live gateway - STAT_MSG = 0x18 # Statistics record from publisher - OHLCV_1S_MSG = 0x20 # OHLCV at 1-second cadence - OHLCV_1M_MSG = 0x21 # OHLCV at 1-minute cadence - OHLCV_1H_MSG = 0x22 # OHLCV at hourly cadence - OHLCV_1D_MSG = 0x23 # OHLCV at daily cadence - MBO_MSG = 0xA0 # Market-by-order record - CMBP_1_MSG = 0xB1 # Consolidated market-by-price with book depth 1 - CBBO_1S_MSG = 0xC0 # Consolidated market-by-price with book depth 1 at 1-second cadence - CBBO_1M_MSG = 0xC1 # Consolidated market-by-price with book depth 1 at 1-minute cadence - TCBBO_MSG = 0xC2 # Consolidated market-by-price with book depth 1 (trades only) - BBO_1S_MSG = 0xC3 # Market-by-price with book depth 1 at 1-second cadence - BBO_1M_MSG = 0xC4 # Market-by-price with book depth 1 at 1-minute cadence -end - -""" - Action - -Market actions that can be applied to orders or trades. - -# Values -- `ADD`: Insert a new order into the book -- `MODIFY`: Change an order's price and/or size -- `CANCEL`: Fully or partially cancel an order from the book -- `CLEAR`: Remove all resting orders for the instrument -- `TRADE`: An aggressing order traded. Does not affect the book -- `FILL`: A resting order was filled. Does not affect the book -- `NONE`: No action: does not affect the book, but may carry flags or other information -""" -@enumx Action::UInt8 begin - ADD = UInt8('A') # Insert a new order into the book - MODIFY = UInt8('M') # Change an order's price and/or size - CANCEL = UInt8('C') # Fully or partially cancel an order from the book - CLEAR = UInt8('R') # Remove all resting orders for the instrument - TRADE = UInt8('T') # An aggressing order traded. Does not affect the book - FILL = UInt8('F') # A resting order was filled. Does not affect the book - NONE = UInt8('N') # No action: does not affect the book, but may carry flags or other information -end - -""" - Side - -Market sides for orders and trades. - -# Values -- `ASK`: Ask/offer side (sell orders) -- `BID`: Bid side (buy orders) -- `NONE`: No specific side or not applicable -""" -@enumx Side::UInt8 begin - ASK = UInt8('A') - BID = UInt8('B') - NONE = UInt8('N') -end - -""" - InstrumentClass - -Classification of financial instruments. Values match Databento's DBN -`InstrumentClass` enum. - -# Values -- `STOCK`: Equity instruments -- `CALL`: Call option contracts -- `PUT`: Put option contracts -- `FUTURE`: Futures contracts -- `BOND`: Fixed income securities -- `MIXED_SPREAD`: Mixed spread instruments -- `FUTURE_SPREAD`: Futures spread instruments -- `OPTION_SPREAD`: Option spread instruments -- `FX_SPOT`: Foreign exchange spot -- `COMMODITY_SPOT`: Commodity spot -- `UNKNOWN_0`, `UNKNOWN_45`: Numeric fallback values for unknown classes -""" -@enumx InstrumentClass::UInt8 begin - BOND = UInt8('B') - CALL = UInt8('C') - FUTURE = UInt8('F') - STOCK = UInt8('K') - MIXED_SPREAD = UInt8('M') - PUT = UInt8('P') - FUTURE_SPREAD = UInt8('S') - OPTION_SPREAD = UInt8('T') - FX_SPOT = UInt8('X') - COMMODITY_SPOT = UInt8('Y') - # Also support numeric values - UNKNOWN_0 = 0 - UNKNOWN_45 = 45 - OTHER = UInt8('?') -end - -# Basic structures - -""" - VersionUpgradePolicy - -Encapsulates the version upgrade policy for DBN files. - -# Fields -- `upgrade_policy::UInt8`: Policy for handling version upgrades -""" -struct VersionUpgradePolicy - upgrade_policy::UInt8 -end - -""" - DatasetCondition - -Conditions and constraints for a dataset. - -# Fields -- `last_ts_out::Int64`: Last timestamp output -- `start_ts::Int64`: Dataset start timestamp -- `end_ts::Int64`: Dataset end timestamp -- `limit::UInt64`: Record limit for the dataset -""" -struct DatasetCondition - last_ts_out::Int64 - start_ts::Int64 - end_ts::Int64 - limit::UInt64 -end - -""" - Metadata - -Metadata information for a DBN dataset. - -# Fields -- `version::UInt8`: DBN format version -- `dataset::String`: Dataset identifier -- `schema::Schema.T`: Data schema type -- `start_ts::Int64`: Start timestamp for the data -- `end_ts::Union{Int64,Nothing}`: End timestamp (can be null) -- `limit::Union{UInt64,Nothing}`: Record count limit (can be null) -- `stype_in::Union{SType.T,Nothing}`: Input symbol type (can be null) -- `stype_out::SType.T`: Output symbol type -- `ts_out::Bool`: Whether timestamps are included in output -- `symbols::Vector{String}`: List of symbols in the dataset -- `partial::Vector{String}`: Partially available symbols -- `not_found::Vector{String}`: Symbols that were not found -- `mappings::Vector{Tuple{String,String,Int64,Int64}}`: Symbol mappings as - `(raw_symbol, mapped_symbol, start_date, end_date)`, one tuple per mapping - interval. A symbol with multiple intervals (e.g. a continuous contract's roll - history) contributes consecutive tuples sharing the same raw symbol. Dates - are raw `YYYYMMDD` integers. -""" -struct Metadata - version::UInt8 - dataset::String - schema::Schema.T - start_ts::Int64 - end_ts::Union{Int64,Nothing} # Can be null - limit::Union{UInt64,Nothing} # Can be null - stype_in::Union{SType.T,Nothing} # Can be null - stype_out::SType.T - ts_out::Bool - symbols::Vector{String} - partial::Vector{String} - not_found::Vector{String} - mappings::Vector{Tuple{String,String,Int64,Int64}} -end - -""" - DBNHeader - -Complete header information for a DBN file. - -# Fields -- `version_upgrade_policy::VersionUpgradePolicy`: Version handling policy -- `dataset_condition::DatasetCondition`: Dataset conditions and constraints -- `metadata::Metadata`: Dataset metadata -""" -struct DBNHeader - version_upgrade_policy::VersionUpgradePolicy - dataset_condition::DatasetCondition - metadata::Metadata -end - -""" - RecordHeader - -Standard header present in all DBN record types. - -# Fields -- `length::UInt8`: Length of the record in 4-byte units (multiply by LENGTH_MULTIPLIER for bytes) -- `rtype::RType.T`: Record type identifier -- `publisher_id::UInt16`: Publisher/venue identifier -- `instrument_id::UInt32`: Instrument identifier -- `ts_event::Int64`: Event timestamp in nanoseconds since Unix epoch -""" -struct RecordHeader - length::UInt8 - rtype::RType.T - publisher_id::UInt16 - instrument_id::UInt32 - ts_event::Int64 -end - -""" - record_length_bytes(hd::RecordHeader) - -Get the actual record length in bytes from a RecordHeader. -The length field stores 4-byte units, so multiply by LENGTH_MULTIPLIER. -""" -record_length_bytes(hd::RecordHeader) = hd.length * LENGTH_MULTIPLIER - -""" - BidAskPair - -Bid and ask price/size information for market data. - -# Fields -- `bid_px::Int64`: Bid price (scaled by FIXED_PRICE_SCALE) -- `ask_px::Int64`: Ask price (scaled by FIXED_PRICE_SCALE) -- `bid_sz::UInt32`: Bid size/quantity -- `ask_sz::UInt32`: Ask size/quantity -- `bid_ct::UInt32`: Number of bid orders -- `ask_ct::UInt32`: Number of ask orders -""" -struct BidAskPair - bid_px::Int64 - ask_px::Int64 - bid_sz::UInt32 - ask_sz::UInt32 - bid_ct::UInt32 - ask_ct::UInt32 -end - -# Timestamp utilities - -""" - DBNTimestamp - -High-precision timestamp representation with nanosecond accuracy. - -# Fields -- `seconds::Int64`: Unix epoch seconds -- `nanoseconds::Int32`: Nanoseconds within the second (0-999_999_999) -""" -struct DBNTimestamp - seconds::Int64 # Unix epoch seconds - nanoseconds::Int32 # Nanoseconds within the second (0-999_999_999) -end - -""" - DBNTimestamp(ns::Int64) - -Construct a DBNTimestamp from nanoseconds since Unix epoch. - -# Arguments -- `ns::Int64`: Nanoseconds since Unix epoch - -# Returns -- `DBNTimestamp`: Timestamp split into seconds and nanoseconds components -""" -function DBNTimestamp(ns::Int64) - if ns == UNDEF_TIMESTAMP - return DBNTimestamp(UNDEF_TIMESTAMP, 0) - end - seconds = ns ÷ 1_000_000_000 - nanoseconds = Int32(ns % 1_000_000_000) - return DBNTimestamp(seconds, nanoseconds) -end - -""" - to_nanoseconds(ts::DBNTimestamp) - -Convert a DBNTimestamp back to nanoseconds since Unix epoch. - -# Arguments -- `ts::DBNTimestamp`: Timestamp to convert - -# Returns -- `Int64`: Nanoseconds since Unix epoch, or UNDEF_TIMESTAMP if undefined -""" -function to_nanoseconds(ts::DBNTimestamp) - if ts.seconds == UNDEF_TIMESTAMP - return UNDEF_TIMESTAMP - end - return ts.seconds * 1_000_000_000 + ts.nanoseconds -end - -""" - ts_to_datetime(ts::Int64) - -Convert a nanosecond timestamp to DateTime with nanosecond precision information. - -# Arguments -- `ts::Int64`: Nanoseconds since Unix epoch - -# Returns -- `NamedTuple`: Contains `datetime` (DateTime) and `nanoseconds` (Int32), or `nothing` if undefined -""" -function ts_to_datetime(ts::Int64) - if ts == UNDEF_TIMESTAMP - return nothing - end - # Returns DateTime with millisecond precision and separate nanosecond component - dbn_ts = DBNTimestamp(ts) - dt = unix2datetime(Float64(dbn_ts.seconds) + dbn_ts.nanoseconds / 1_000_000_000) - return (datetime=dt, nanoseconds=dbn_ts.nanoseconds) -end - -""" - datetime_to_ts(dt::DateTime, nanoseconds::Int32=0) - -Convert a DateTime with optional nanosecond precision to nanosecond timestamp. - -# Arguments -- `dt::DateTime`: DateTime to convert -- `nanoseconds::Int32`: Additional nanoseconds within the second (default: 0) - -# Returns -- `Int64`: Nanoseconds since Unix epoch -""" -function datetime_to_ts(dt::DateTime, nanoseconds::Union{Int32,Int64}=0) - # Convert DateTime to nanoseconds, preserving additional precision - seconds = Int64(round(datetime2unix(dt))) - return seconds * 1_000_000_000 + nanoseconds -end - -""" - ts_to_date_time(ts::Int64) - -Convert a nanosecond timestamp to separate Date and Time components with full nanosecond precision. - -# Arguments -- `ts::Int64`: Nanoseconds since Unix epoch - -# Returns -- `NamedTuple`: Contains `date` (Date), `time` (Time), and `timestamp` (DBNTimestamp), or `nothing` if undefined -""" -function ts_to_date_time(ts::Int64) - if ts == UNDEF_TIMESTAMP - return nothing - end - dbn_ts = DBNTimestamp(ts) - - # Get the date part - dt_seconds = unix2datetime(Float64(dbn_ts.seconds)) - date_part = Date(dt_seconds) - - # Get time within the day with nanosecond precision - seconds_in_day = dbn_ts.seconds % 86400 - time_ns = seconds_in_day * 1_000_000_000 + dbn_ts.nanoseconds - time_part = Dates.Time(Dates.Nanosecond(time_ns)) - - return (date=date_part, time=time_part, timestamp=dbn_ts) -end - -""" - date_time_to_ts(date::Date, time::Dates.Time) - -Convert separate Date and Time components to nanosecond timestamp. - -# Arguments -- `date::Date`: Date component -- `time::Dates.Time`: Time component with nanosecond precision - -# Returns -- `Int64`: Nanoseconds since Unix epoch -""" -function date_time_to_ts(date::Date, time::Dates.Time) - # Convert date to seconds since epoch - dt = DateTime(date) - date_seconds = Int64(round(datetime2unix(dt))) - - # Extract nanoseconds from time - time_ns = Dates.value(time) # Total nanoseconds since midnight - - return date_seconds * 1_000_000_000 + time_ns -end - -# Price conversion utilities - -""" - price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) - -Convert a fixed-point price to floating-point representation. - -# Arguments -- `price::Int64`: Fixed-point price value -- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) - -# Returns -- `Float64`: Floating-point price, or NaN if price is UNDEF_PRICE -""" -function price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) - if price == UNDEF_PRICE - return NaN - end - return Float64(price) / Float64(scale) -end - -""" - float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) - -Convert a floating-point price to fixed-point representation. - -# Arguments -- `value::Float64`: Floating-point price -- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) - -# Returns -- `Int64`: Fixed-point price, or UNDEF_PRICE if value is NaN or infinite -""" -function float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) - if isnan(value) || isinf(value) - return UNDEF_PRICE - end - return Int64(round(value * Float64(scale))) -end - -# Helper functions for safe enum conversion. -# All enums share one policy: an invalid byte warns (once per enum) and maps to -# a sentinel value instead of throwing, so one bad byte can't kill a decode -# stream. Lookup tables keep the hot path free of try/catch. - -function _build_enum_lookup(::Type{E}, default::E) where {E} - table = fill(default, 256) - valid = falses(256) - for inst in instances(E) - idx = Int(UInt8(inst)) + 1 - table[idx] = inst - valid[idx] = true - end - return table, valid -end - -const _ACTION_LOOKUP, _ACTION_VALID = _build_enum_lookup(Action.T, Action.NONE) -const _SIDE_LOOKUP, _SIDE_VALID = _build_enum_lookup(Side.T, Side.NONE) -const _INSTRUMENT_CLASS_LOOKUP, _INSTRUMENT_CLASS_VALID = - _build_enum_lookup(InstrumentClass.T, InstrumentClass.OTHER) - -""" - safe_action(raw_val::UInt8) - -Convert a raw byte value to an Action enum. `0x00` and invalid values map to -`Action.NONE`; invalid values additionally log a warning (once). -""" -@inline function safe_action(raw_val::UInt8) - # 0 indicates no action for certain record types - raw_val == 0x00 && return Action.NONE - idx = Int(raw_val) + 1 - @inbounds if !_ACTION_VALID[idx] - @warn "Unknown Action value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 - end - return @inbounds _ACTION_LOOKUP[idx] -end - -""" - safe_side(raw_val::UInt8) - -Convert a raw byte value to a Side enum. `0x00` and invalid values map to -`Side.NONE`; invalid values additionally log a warning (once). -""" -@inline function safe_side(raw_val::UInt8) - # 0 indicates no side for certain record types - raw_val == 0x00 && return Side.NONE - idx = Int(raw_val) + 1 - @inbounds if !_SIDE_VALID[idx] - @warn "Unknown Side value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 - end - return @inbounds _SIDE_LOOKUP[idx] -end - -""" - safe_instrument_class(raw_val::UInt8) - -Convert a raw byte value to an InstrumentClass enum. Invalid values log a -warning (once) and map to `InstrumentClass.OTHER`. -""" -@inline function safe_instrument_class(raw_val::UInt8) - idx = Int(raw_val) + 1 - @inbounds if !_INSTRUMENT_CLASS_VALID[idx] - @warn "Unknown InstrumentClass value: $raw_val (0x$(string(raw_val, base=16))), using OTHER as default" maxlog = 1 - end - return @inbounds _INSTRUMENT_CLASS_LOOKUP[idx] -end +# DBN types, enums, and data structures + +# Constants +"""The current DBN format version supported by this implementation.""" +const DBN_VERSION = 3 + +"""Fixed-length symbol string size for DBN v2/v3 metadata and records.""" +const SYMBOL_CSTR_LEN = 71 + +"""Fixed-point price scaling factor for converting between integer and float prices.""" +const FIXED_PRICE_SCALE = Int32(1_000_000_000) + +"""Sentinel value indicating an undefined or missing price.""" +const UNDEF_PRICE = typemax(Int64) + +"""Sentinel value indicating an undefined or missing order size.""" +const UNDEF_ORDER_SIZE = typemax(UInt32) + +"""Sentinel value indicating an undefined or missing timestamp.""" +const UNDEF_TIMESTAMP = typemax(Int64) + +"""Multiplier for converting the length field to bytes in record headers.""" +const LENGTH_MULTIPLIER = 4 + +# Enums using EnumX for better namespace management + +""" + Schema + +DBN data schemas representing different types of market data. + +# Values +- `MBO`: Market-by-order data +- `MBP_1`: Market-by-price with book depth 1 (TBBO) +- `MBP_10`: Market-by-price with book depth 10 +- `TBBO`: Top-of-book bid/offer +- `TRADES`: Trade messages only +- `OHLCV_1S`: OHLCV data at 1-second intervals +- `OHLCV_1M`: OHLCV data at 1-minute intervals +- `OHLCV_1H`: OHLCV data at 1-hour intervals +- `OHLCV_1D`: OHLCV data at 1-day intervals +- `DEFINITION`: Instrument definition data +- `STATISTICS`: Market statistics +- `STATUS`: Exchange status messages +- `IMBALANCE`: Order imbalance data +- `CBBO`: Consolidated best bid/offer +- `CBBO_1S`: Consolidated BBO at 1-second intervals +- `CBBO_1M`: Consolidated BBO at 1-minute intervals +- `CMBP_1`: Consolidated market-by-price depth 1 +- `TCBBO`: Trade-consolidated BBO +- `BBO_1S`: BBO at 1-second intervals +- `BBO_1M`: BBO at 1-minute intervals +""" +@enumx Schema::UInt16 begin + MBO = 0 + MBP_1 = 1 + MBP_10 = 2 + TBBO = 3 + TRADES = 4 + OHLCV_1S = 5 + OHLCV_1M = 6 + OHLCV_1H = 7 + OHLCV_1D = 8 + DEFINITION = 9 + STATISTICS = 10 + STATUS = 11 + IMBALANCE = 12 + CBBO = 13 + CBBO_1S = 14 + CBBO_1M = 15 + CMBP_1 = 16 + TCBBO = 17 + BBO_1S = 18 + BBO_1M = 19 + MIX = 0xFFFF +end + +""" + Compression + +Compression algorithms supported for DBN files. + +# Values +- `NONE`: No compression +- `ZSTD`: Zstandard compression +""" +@enumx Compression::UInt8 begin + NONE = 0 + ZSTD = 1 +end + +""" + Encoding + +Output encoding formats for market data. + +# Values +- `DBN`: Databento Binary Encoding +- `CSV`: Comma-separated values +- `JSON`: JavaScript Object Notation +""" +@enumx Encoding::UInt8 begin + DBN = 0 + CSV = 1 + JSON = 2 +end + +""" + SType + +Symbol types for identifying instruments in DBN data. Numeric values match the +official Databento DBN spec — wire-encoded as a `UInt8` in metadata and in +`SymbolMappingMsg` (v2+). + +# Values +- `INSTRUMENT_ID = 0`: Numeric instrument identifier +- `RAW_SYMBOL = 1`: Raw symbol string from exchange +- `SMART = 2`: Deprecated alias (was split into `CONTINUOUS` and `PARENT`) +- `CONTINUOUS = 3`: Continuous contract symbol +- `PARENT = 4`: Parent symbol for derived instruments (e.g. `SPXW.OPT`) +- `NASDAQ_SYMBOL = 5`: Nasdaq-specific symbol +- `CMS_SYMBOL = 6`: CMS symbol +- `ISIN = 7`: ISO 6166 International Securities Identification Number +- `US_CODE = 8`: US Code (CUSIP-style) +- `BBG_COMP_ID = 9`: Bloomberg composite ID +- `BBG_COMP_TICKER = 10`: Bloomberg composite ticker +- `FIGI = 11`: OpenFIGI identifier +- `FIGI_TICKER = 12`: OpenFIGI ticker +- `UNDEF = 255`: Unset/undefined sentinel (`0xFF`). Databento wire-encodes an + unset `stype` as `0xFF`; this member lets non-nullable `stype` fields (e.g. + in `SymbolMappingMsg`) represent and round-trip that sentinel. Metadata's + nullable `stype_in` decodes `0xFF` to `nothing` instead. +""" +@enumx SType::UInt8 begin + INSTRUMENT_ID = 0 + RAW_SYMBOL = 1 + SMART = 2 # deprecated (kept for round-trip with v1 wire data) + CONTINUOUS = 3 + PARENT = 4 + NASDAQ_SYMBOL = 5 + CMS_SYMBOL = 6 + ISIN = 7 + US_CODE = 8 + BBG_COMP_ID = 9 + BBG_COMP_TICKER = 10 + FIGI = 11 + FIGI_TICKER = 12 + UNDEF = 255 # 0xFF "unset" sentinel; see docstring +end + +""" + RType + +Record types for different kinds of market data messages in DBN format. + +# Values +- `MBP_0_MSG`: Trades (book depth 0) +- `MBP_1_MSG`: TBBO/MBP-1 (book depth 1) +- `MBP_10_MSG`: MBP-10 (book depth 10) +- `STATUS_MSG`: Exchange status record +- `INSTRUMENT_DEF_MSG`: Instrument definition record +- `IMBALANCE_MSG`: Order imbalance record +- `ERROR_MSG`: Error record from live gateway +- `SYMBOL_MAPPING_MSG`: Symbol mapping record from live gateway +- `SYSTEM_MSG`: Non-error record from live gateway +- `STAT_MSG`: Statistics record from publisher +- `OHLCV_1S_MSG`: OHLCV at 1-second cadence +- `OHLCV_1M_MSG`: OHLCV at 1-minute cadence +- `OHLCV_1H_MSG`: OHLCV at hourly cadence +- `OHLCV_1D_MSG`: OHLCV at daily cadence +- `MBO_MSG`: Market-by-order record +- `CMBP_1_MSG`: Consolidated market-by-price with book depth 1 +- `CBBO_1S_MSG`: Consolidated market-by-price with book depth 1 at 1-second cadence +- `CBBO_1M_MSG`: Consolidated market-by-price with book depth 1 at 1-minute cadence +- `TCBBO_MSG`: Consolidated market-by-price with book depth 1 (trades only) +- `BBO_1S_MSG`: Market-by-price with book depth 1 at 1-second cadence +- `BBO_1M_MSG`: Market-by-price with book depth 1 at 1-minute cadence +""" +@enumx RType::UInt8 begin + MBP_0_MSG = 0x00 # Trades (book depth 0) + MBP_1_MSG = 0x01 # TBBO/MBP-1 (book depth 1) + MBP_10_MSG = 0x0A # MBP-10 (book depth 10) + STATUS_MSG = 0x12 # Exchange status record + INSTRUMENT_DEF_MSG = 0x13 # Instrument definition record + IMBALANCE_MSG = 0x14 # Order imbalance record + ERROR_MSG = 0x15 # Error record from live gateway + SYMBOL_MAPPING_MSG = 0x16 # Symbol mapping record from live gateway + SYSTEM_MSG = 0x17 # Non-error record from live gateway + STAT_MSG = 0x18 # Statistics record from publisher + OHLCV_1S_MSG = 0x20 # OHLCV at 1-second cadence + OHLCV_1M_MSG = 0x21 # OHLCV at 1-minute cadence + OHLCV_1H_MSG = 0x22 # OHLCV at hourly cadence + OHLCV_1D_MSG = 0x23 # OHLCV at daily cadence + MBO_MSG = 0xA0 # Market-by-order record + CMBP_1_MSG = 0xB1 # Consolidated market-by-price with book depth 1 + CBBO_1S_MSG = 0xC0 # Consolidated market-by-price with book depth 1 at 1-second cadence + CBBO_1M_MSG = 0xC1 # Consolidated market-by-price with book depth 1 at 1-minute cadence + TCBBO_MSG = 0xC2 # Consolidated market-by-price with book depth 1 (trades only) + BBO_1S_MSG = 0xC3 # Market-by-price with book depth 1 at 1-second cadence + BBO_1M_MSG = 0xC4 # Market-by-price with book depth 1 at 1-minute cadence +end + +""" + Action + +Market actions that can be applied to orders or trades. + +# Values +- `ADD`: Insert a new order into the book +- `MODIFY`: Change an order's price and/or size +- `CANCEL`: Fully or partially cancel an order from the book +- `CLEAR`: Remove all resting orders for the instrument +- `TRADE`: An aggressing order traded. Does not affect the book +- `FILL`: A resting order was filled. Does not affect the book +- `NONE`: No action: does not affect the book, but may carry flags or other information +""" +@enumx Action::UInt8 begin + ADD = UInt8('A') # Insert a new order into the book + MODIFY = UInt8('M') # Change an order's price and/or size + CANCEL = UInt8('C') # Fully or partially cancel an order from the book + CLEAR = UInt8('R') # Remove all resting orders for the instrument + TRADE = UInt8('T') # An aggressing order traded. Does not affect the book + FILL = UInt8('F') # A resting order was filled. Does not affect the book + NONE = UInt8('N') # No action: does not affect the book, but may carry flags or other information +end + +""" + Side + +Market sides for orders and trades. + +# Values +- `ASK`: Ask/offer side (sell orders) +- `BID`: Bid side (buy orders) +- `NONE`: No specific side or not applicable +""" +@enumx Side::UInt8 begin + ASK = UInt8('A') + BID = UInt8('B') + NONE = UInt8('N') +end + +""" + InstrumentClass + +Classification of financial instruments. Values match Databento's DBN +`InstrumentClass` enum. + +# Values +- `STOCK`: Equity instruments +- `CALL`: Call option contracts +- `PUT`: Put option contracts +- `FUTURE`: Futures contracts +- `INDEX`: Index instruments +- `BOND`: Fixed income securities +- `MIXED_SPREAD`: Mixed spread instruments +- `FUTURE_SPREAD`: Futures spread instruments +- `OPTION_SPREAD`: Option spread instruments +- `FX_SPOT`: Foreign exchange spot +- `COMMODITY_SPOT`: Commodity spot +- `UNKNOWN_0`, `UNKNOWN_45`: Numeric fallback values for unknown classes +""" +@enumx InstrumentClass::UInt8 begin + BOND = UInt8('B') + CALL = UInt8('C') + FUTURE = UInt8('F') + INDEX = UInt8('I') + STOCK = UInt8('K') + MIXED_SPREAD = UInt8('M') + PUT = UInt8('P') + FUTURE_SPREAD = UInt8('S') + OPTION_SPREAD = UInt8('T') + FX_SPOT = UInt8('X') + COMMODITY_SPOT = UInt8('Y') + # Also support numeric values + UNKNOWN_0 = 0 + UNKNOWN_45 = 45 + OTHER = UInt8('?') +end + +# Basic structures + +""" + VersionUpgradePolicy + +Encapsulates the version upgrade policy for DBN files. + +# Fields +- `upgrade_policy::UInt8`: Policy for handling version upgrades +""" +struct VersionUpgradePolicy + upgrade_policy::UInt8 +end + +""" + DatasetCondition + +Conditions and constraints for a dataset. + +# Fields +- `last_ts_out::Int64`: Last timestamp output +- `start_ts::Int64`: Dataset start timestamp +- `end_ts::Int64`: Dataset end timestamp +- `limit::UInt64`: Record limit for the dataset +""" +struct DatasetCondition + last_ts_out::Int64 + start_ts::Int64 + end_ts::Int64 + limit::UInt64 +end + +""" + Metadata + +Metadata information for a DBN dataset. + +# Fields +- `version::UInt8`: DBN format version +- `dataset::String`: Dataset identifier +- `schema::Schema.T`: Data schema type +- `start_ts::Int64`: Start timestamp for the data +- `end_ts::Union{Int64,Nothing}`: End timestamp (can be null) +- `limit::Union{UInt64,Nothing}`: Record count limit (can be null) +- `stype_in::Union{SType.T,Nothing}`: Input symbol type (can be null) +- `stype_out::SType.T`: Output symbol type +- `ts_out::Bool`: Whether timestamps are included in output +- `symbols::Vector{String}`: List of symbols in the dataset +- `partial::Vector{String}`: Partially available symbols +- `not_found::Vector{String}`: Symbols that were not found +- `mappings::Vector{Tuple{String,String,Int64,Int64}}`: Symbol mappings as + `(raw_symbol, mapped_symbol, start_date, end_date)`, one tuple per mapping + interval. A symbol with multiple intervals (e.g. a continuous contract's roll + history) contributes consecutive tuples sharing the same raw symbol. Dates + are raw `YYYYMMDD` integers. +""" +struct Metadata + version::UInt8 + dataset::String + schema::Schema.T + start_ts::Int64 + end_ts::Union{Int64,Nothing} # Can be null + limit::Union{UInt64,Nothing} # Can be null + stype_in::Union{SType.T,Nothing} # Can be null + stype_out::SType.T + ts_out::Bool + symbols::Vector{String} + partial::Vector{String} + not_found::Vector{String} + mappings::Vector{Tuple{String,String,Int64,Int64}} +end + +""" + DBNHeader + +Complete header information for a DBN file. + +# Fields +- `version_upgrade_policy::VersionUpgradePolicy`: Version handling policy +- `dataset_condition::DatasetCondition`: Dataset conditions and constraints +- `metadata::Metadata`: Dataset metadata +""" +struct DBNHeader + version_upgrade_policy::VersionUpgradePolicy + dataset_condition::DatasetCondition + metadata::Metadata +end + +""" + RecordHeader + +Standard header present in all DBN record types. + +# Fields +- `length::UInt8`: Length of the record in 4-byte units (multiply by LENGTH_MULTIPLIER for bytes) +- `rtype::RType.T`: Record type identifier +- `publisher_id::UInt16`: Publisher/venue identifier +- `instrument_id::UInt32`: Instrument identifier +- `ts_event::Int64`: Event timestamp in nanoseconds since Unix epoch +""" +struct RecordHeader + length::UInt8 + rtype::RType.T + publisher_id::UInt16 + instrument_id::UInt32 + ts_event::Int64 +end + +""" + record_length_bytes(hd::RecordHeader) + +Get the actual record length in bytes from a RecordHeader. +The length field stores 4-byte units, so multiply by LENGTH_MULTIPLIER. +""" +record_length_bytes(hd::RecordHeader) = hd.length * LENGTH_MULTIPLIER + +""" + BidAskPair + +Bid and ask price/size information for market data. + +# Fields +- `bid_px::Int64`: Bid price (scaled by FIXED_PRICE_SCALE) +- `ask_px::Int64`: Ask price (scaled by FIXED_PRICE_SCALE) +- `bid_sz::UInt32`: Bid size/quantity +- `ask_sz::UInt32`: Ask size/quantity +- `bid_ct::UInt32`: Number of bid orders +- `ask_ct::UInt32`: Number of ask orders +""" +struct BidAskPair + bid_px::Int64 + ask_px::Int64 + bid_sz::UInt32 + ask_sz::UInt32 + bid_ct::UInt32 + ask_ct::UInt32 +end + +# Timestamp utilities + +""" + DBNTimestamp + +High-precision timestamp representation with nanosecond accuracy. + +# Fields +- `seconds::Int64`: Unix epoch seconds +- `nanoseconds::Int32`: Nanoseconds within the second (0-999_999_999) +""" +struct DBNTimestamp + seconds::Int64 # Unix epoch seconds + nanoseconds::Int32 # Nanoseconds within the second (0-999_999_999) +end + +""" + DBNTimestamp(ns::Int64) + +Construct a DBNTimestamp from nanoseconds since Unix epoch. + +# Arguments +- `ns::Int64`: Nanoseconds since Unix epoch + +# Returns +- `DBNTimestamp`: Timestamp split into seconds and nanoseconds components +""" +function DBNTimestamp(ns::Int64) + if ns == UNDEF_TIMESTAMP + return DBNTimestamp(UNDEF_TIMESTAMP, 0) + end + seconds = ns ÷ 1_000_000_000 + nanoseconds = Int32(ns % 1_000_000_000) + return DBNTimestamp(seconds, nanoseconds) +end + +""" + to_nanoseconds(ts::DBNTimestamp) + +Convert a DBNTimestamp back to nanoseconds since Unix epoch. + +# Arguments +- `ts::DBNTimestamp`: Timestamp to convert + +# Returns +- `Int64`: Nanoseconds since Unix epoch, or UNDEF_TIMESTAMP if undefined +""" +function to_nanoseconds(ts::DBNTimestamp) + if ts.seconds == UNDEF_TIMESTAMP + return UNDEF_TIMESTAMP + end + return ts.seconds * 1_000_000_000 + ts.nanoseconds +end + +""" + ts_to_datetime(ts::Int64) + +Convert a nanosecond timestamp to DateTime with nanosecond precision information. + +# Arguments +- `ts::Int64`: Nanoseconds since Unix epoch + +# Returns +- `NamedTuple`: Contains `datetime` (DateTime) and `nanoseconds` (Int32), or `nothing` if undefined +""" +function ts_to_datetime(ts::Int64) + if ts == UNDEF_TIMESTAMP + return nothing + end + # Returns DateTime with millisecond precision and separate nanosecond component + dbn_ts = DBNTimestamp(ts) + dt = unix2datetime(Float64(dbn_ts.seconds) + dbn_ts.nanoseconds / 1_000_000_000) + return (datetime=dt, nanoseconds=dbn_ts.nanoseconds) +end + +""" + datetime_to_ts(dt::DateTime, nanoseconds::Int32=0) + +Convert a DateTime with optional nanosecond precision to nanosecond timestamp. + +# Arguments +- `dt::DateTime`: DateTime to convert +- `nanoseconds::Int32`: Additional nanoseconds within the second (default: 0) + +# Returns +- `Int64`: Nanoseconds since Unix epoch +""" +function datetime_to_ts(dt::DateTime, nanoseconds::Union{Int32,Int64}=0) + # Convert DateTime to nanoseconds, preserving additional precision + seconds = Int64(round(datetime2unix(dt))) + return seconds * 1_000_000_000 + nanoseconds +end + +""" + ts_to_date_time(ts::Int64) + +Convert a nanosecond timestamp to separate Date and Time components with full nanosecond precision. + +# Arguments +- `ts::Int64`: Nanoseconds since Unix epoch + +# Returns +- `NamedTuple`: Contains `date` (Date), `time` (Time), and `timestamp` (DBNTimestamp), or `nothing` if undefined +""" +function ts_to_date_time(ts::Int64) + if ts == UNDEF_TIMESTAMP + return nothing + end + dbn_ts = DBNTimestamp(ts) + + # Get the date part + dt_seconds = unix2datetime(Float64(dbn_ts.seconds)) + date_part = Date(dt_seconds) + + # Get time within the day with nanosecond precision + seconds_in_day = dbn_ts.seconds % 86400 + time_ns = seconds_in_day * 1_000_000_000 + dbn_ts.nanoseconds + time_part = Dates.Time(Dates.Nanosecond(time_ns)) + + return (date=date_part, time=time_part, timestamp=dbn_ts) +end + +""" + date_time_to_ts(date::Date, time::Dates.Time) + +Convert separate Date and Time components to nanosecond timestamp. + +# Arguments +- `date::Date`: Date component +- `time::Dates.Time`: Time component with nanosecond precision + +# Returns +- `Int64`: Nanoseconds since Unix epoch +""" +function date_time_to_ts(date::Date, time::Dates.Time) + # Convert date to seconds since epoch + dt = DateTime(date) + date_seconds = Int64(round(datetime2unix(dt))) + + # Extract nanoseconds from time + time_ns = Dates.value(time) # Total nanoseconds since midnight + + return date_seconds * 1_000_000_000 + time_ns +end + +# Price conversion utilities + +""" + price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) + +Convert a fixed-point price to floating-point representation. + +# Arguments +- `price::Int64`: Fixed-point price value +- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) + +# Returns +- `Float64`: Floating-point price, or NaN if price is UNDEF_PRICE +""" +function price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) + if price == UNDEF_PRICE + return NaN + end + return Float64(price) / Float64(scale) +end + +""" + float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) + +Convert a floating-point price to fixed-point representation. + +# Arguments +- `value::Float64`: Floating-point price +- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) + +# Returns +- `Int64`: Fixed-point price, or UNDEF_PRICE if value is NaN or infinite +""" +function float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) + if isnan(value) || isinf(value) + return UNDEF_PRICE + end + return Int64(round(value * Float64(scale))) +end + +# Helper functions for safe enum conversion. +# All enums share one policy: an invalid byte warns (once per enum) and maps to +# a sentinel value instead of throwing, so one bad byte can't kill a decode +# stream. Lookup tables keep the hot path free of try/catch. + +function _build_enum_lookup(::Type{E}, default::E) where {E} + table = fill(default, 256) + valid = falses(256) + for inst in instances(E) + idx = Int(UInt8(inst)) + 1 + table[idx] = inst + valid[idx] = true + end + return table, valid +end + +const _ACTION_LOOKUP, _ACTION_VALID = _build_enum_lookup(Action.T, Action.NONE) +const _SIDE_LOOKUP, _SIDE_VALID = _build_enum_lookup(Side.T, Side.NONE) +const _INSTRUMENT_CLASS_LOOKUP, _INSTRUMENT_CLASS_VALID = + _build_enum_lookup(InstrumentClass.T, InstrumentClass.OTHER) + +""" + safe_action(raw_val::UInt8) + +Convert a raw byte value to an Action enum. `0x00` and invalid values map to +`Action.NONE`; invalid values additionally log a warning (once). +""" +@inline function safe_action(raw_val::UInt8) + # 0 indicates no action for certain record types + raw_val == 0x00 && return Action.NONE + idx = Int(raw_val) + 1 + @inbounds if !_ACTION_VALID[idx] + @warn "Unknown Action value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 + end + return @inbounds _ACTION_LOOKUP[idx] +end + +""" + safe_side(raw_val::UInt8) + +Convert a raw byte value to a Side enum. `0x00` and invalid values map to +`Side.NONE`; invalid values additionally log a warning (once). +""" +@inline function safe_side(raw_val::UInt8) + # 0 indicates no side for certain record types + raw_val == 0x00 && return Side.NONE + idx = Int(raw_val) + 1 + @inbounds if !_SIDE_VALID[idx] + @warn "Unknown Side value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 + end + return @inbounds _SIDE_LOOKUP[idx] +end + +""" + safe_instrument_class(raw_val::UInt8) + +Convert a raw byte value to an InstrumentClass enum. Invalid values log a +warning (once) and map to `InstrumentClass.OTHER`. +""" +@inline function safe_instrument_class(raw_val::UInt8) + idx = Int(raw_val) + 1 + @inbounds if !_INSTRUMENT_CLASS_VALID[idx] + @warn "Unknown InstrumentClass value: $raw_val (0x$(string(raw_val, base=16))), using OTHER as default" maxlog = 1 + end + return @inbounds _INSTRUMENT_CLASS_LOOKUP[idx] +end + +# --------------------------------------------------------------------------- +# Venue-code enums and record flags +# +# `StatMsg.stat_type` and `StatusMsg.trading_event` stay raw `UInt16` on the +# record structs: the wire fields are plain integers and publishers may emit +# codes outside the published enum. The enums below are for *interpreting* +# those codes. Compare with `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)` +# or convert with `safe_stat_type(r.stat_type)` / `safe_trading_event(...)`. +# --------------------------------------------------------------------------- + +""" + StatType + +Type of statistic carried by a [`StatMsg`](@ref) (`stat_type` field). Values +match the official DBN `StatType` enum (`UInt16`). `UNKNOWN = 0` is a local +sentinel returned by [`safe_stat_type`](@ref) for codes this package does not +recognize (publisher-specific, or newer than this release). + +# Values +- `OPENING_PRICE = 1`: Price of the first trade of an instrument +- `INDICATIVE_OPENING_PRICE = 2`: Probable opening price, published pre-open +- `SETTLEMENT_PRICE = 3`: Settlement price +- `TRADING_SESSION_LOW_PRICE = 4`: Lowest trade price of the session +- `TRADING_SESSION_HIGH_PRICE = 5`: Highest trade price of the session +- `CLEARED_VOLUME = 6`: Contracts cleared on the previous trading date +- `LOWEST_OFFER = 7`: Lowest offer price of the session +- `HIGHEST_BID = 8`: Highest bid price of the session +- `OPEN_INTEREST = 9`: Number of outstanding contracts +- `FIXING_PRICE = 10`: VWAP over a fixing period +- `CLOSE_PRICE = 11`: Last trade price of the session +- `NET_CHANGE = 12`: Change from the previous session's close +- `VWAP = 13`: Session volume-weighted average price +- `VOLATILITY = 14`: Implied volatility associated with the settlement price +- `DELTA = 15`: Option delta associated with the settlement price +- `UNCROSSING_PRICE = 16`: Auction uncrossing price +- `UPPER_PRICE_LIMIT = 17`: Exchange-defined upper price limit (published for + CME GLBX.MDP3 since the 2026-07 normalization change) +- `LOWER_PRICE_LIMIT = 18`: Exchange-defined lower price limit (as above) +- `BLOCK_VOLUME = 19`: Block contracts cleared on the previous trading date +- `INDICATIVE_CLOSE_PRICE = 20`: Probable closing price +- `MWCB_LEVEL_1 = 21`, `MWCB_LEVEL_2 = 22`, `MWCB_LEVEL_3 = 23`: Market-wide + circuit-breaker thresholds (7% / 13% / 20%) +- `AUCTION_COLLAR_REFERENCE_PRICE = 24`, `AUCTION_COLLAR_UPPER_PRICE = 25`, + `AUCTION_COLLAR_LOWER_PRICE = 26`: Auction collar prices +- `VENUE_SPECIFIC_VOLUME_1 = 10001`, `VENUE_SPECIFIC_PRICE_1 = 10002`: + Venue-specific statistics +- `UNKNOWN = 0`: Unrecognized code (local sentinel, not part of the DBN spec) +""" +@enumx StatType::UInt16 begin + UNKNOWN = 0 + OPENING_PRICE = 1 + INDICATIVE_OPENING_PRICE = 2 + SETTLEMENT_PRICE = 3 + TRADING_SESSION_LOW_PRICE = 4 + TRADING_SESSION_HIGH_PRICE = 5 + CLEARED_VOLUME = 6 + LOWEST_OFFER = 7 + HIGHEST_BID = 8 + OPEN_INTEREST = 9 + FIXING_PRICE = 10 + CLOSE_PRICE = 11 + NET_CHANGE = 12 + VWAP = 13 + VOLATILITY = 14 + DELTA = 15 + UNCROSSING_PRICE = 16 + UPPER_PRICE_LIMIT = 17 + LOWER_PRICE_LIMIT = 18 + BLOCK_VOLUME = 19 + INDICATIVE_CLOSE_PRICE = 20 + MWCB_LEVEL_1 = 21 + MWCB_LEVEL_2 = 22 + MWCB_LEVEL_3 = 23 + AUCTION_COLLAR_REFERENCE_PRICE = 24 + AUCTION_COLLAR_UPPER_PRICE = 25 + AUCTION_COLLAR_LOWER_PRICE = 26 + VENUE_SPECIFIC_VOLUME_1 = 10001 + VENUE_SPECIFIC_PRICE_1 = 10002 +end + +""" + TradingEvent + +Additional context for a [`StatusMsg`](@ref) (`trading_event` field). Values +match the official DBN `TradingEvent` enum (`UInt16`). + +# Values +- `NONE = 0`: No additional information given +- `NO_CANCEL = 1`: Order entry is allowed; modification and cancellation are not +- `CHANGE_TRADING_SESSION = 2`: A change of trading session occurred; daily + statistics are reset +- `IMPLIED_MATCHING_ON = 3`: Implied matching is available (CME's matching + engine is constructing implied depth) +- `IMPLIED_MATCHING_OFF = 4`: Implied matching is not available + +CME GLBX.MDP3 publishes `IMPLIED_MATCHING_ON`/`OFF` status records since the +2026-07 normalization change. +""" +@enumx TradingEvent::UInt16 begin + NONE = 0 + NO_CANCEL = 1 + CHANGE_TRADING_SESSION = 2 + IMPLIED_MATCHING_ON = 3 + IMPLIED_MATCHING_OFF = 4 +end + +const _STAT_TYPE_LOOKUP = Dict{UInt16,StatType.T}(UInt16(v) => v for v in instances(StatType.T)) +const _TRADING_EVENT_LOOKUP = Dict{UInt16,TradingEvent.T}(UInt16(v) => v for v in instances(TradingEvent.T)) + +""" + safe_stat_type(raw::Integer) -> StatType.T + +Interpret a raw `StatMsg.stat_type` code. Unrecognized codes map to +`StatType.UNKNOWN` without a warning (publisher-specific codes are legitimate). +""" +safe_stat_type(raw::Integer) = get(_STAT_TYPE_LOOKUP, UInt16(raw), StatType.UNKNOWN) + +""" + safe_trading_event(raw::Integer) -> TradingEvent.T + +Interpret a raw `StatusMsg.trading_event` code. Unrecognized codes map to +`TradingEvent.NONE` ("no additional information") without a warning. +""" +safe_trading_event(raw::Integer) = get(_TRADING_EVENT_LOOKUP, UInt16(raw), TradingEvent.NONE) + +# Record `flags` bit field (MBO / MBP / trade / BBO records). Values match the +# official DBN `flags` module. + +""" +Flag bit: last record in the event for a given `instrument_id`. Since the +2026-07 CME normalization change, GLBX.MDP3 MBO emits this on a standalone +record (`action = Action.NONE`, `price = UNDEF_PRICE`, `size = 0`) that +follows the book updates, rather than on the final book update itself. +""" +const F_LAST = 0x80 +"""Flag bit: top-of-book record, not an individual order.""" +const F_TOB = 0x40 +"""Flag bit: record sourced from a replay, such as a snapshot server.""" +const F_SNAPSHOT = 0x20 +"""Flag bit: aggregated price-level record, not an individual order.""" +const F_MBP = 0x10 +"""Flag bit: `ts_recv` is inaccurate due to clock issues or packet reordering.""" +const F_BAD_TS_RECV = 0x08 +"""Flag bit: an unrecoverable gap was detected in the channel.""" +const F_MAYBE_BAD_BOOK = 0x04 +"""Flag bit: publisher-specific event.""" +const F_PUBLISHER_SPECIFIC = 0x02 + +""" + has_flag(flags, flag) -> Bool + +`true` if the `flag` bit (e.g. [`F_LAST`](@ref)) is set in a record's `flags`. +""" +has_flag(flags::Integer, flag::Integer) = (flags & flag) != 0 diff --git a/test/runtests.jl b/test/runtests.jl index 3c1246d3..38684981 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -1,44 +1,45 @@ -using Test -using DatabentoBinaryEncoding -import DatabentoBinaryEncoding as DBN -using Dates - -# Load test utilities (safe_rm, etc.) -include("test_utils.jl") - -@testset "DBN.jl Tests" begin - include("test_phase1.jl") - include("test_phase2.jl") - include("test_phase3.jl") - include("test_phase4.jl") - include("test_phase5.jl") - include("test_phase6.jl") - include("test_phase7.jl") - include("test_phase8.jl") - include("test_phase9_working.jl") # Edge cases and error handling - include("test_phase10_complete.jl") # Integration and performance testing - include("test_convenience_functions.jl") # Test all convenience read_*/foreach_* functions - include("test_phase11_typed_with_control.jl") # foreach_record_with_control: typed data + Union control split - include("test_replay.jl") # replay_dbn / replay_records: timestamp-paced re-emission - include("test_issue23_unset_stype.jl") # regression: 0xFF unset stype in v3 SymbolMappingMsg (issue #23) - include("test_issues_32_35.jl") # regressions: pre-v3 StatMsg layout, stat_to_dataframe, mapping intervals, invalid enum bytes (issues #32-#35) - include("test_show.jl") # compact one-line Base.show for record types - include("test_symbols.jl") # symbol_map / symbol_for / add_symbol_column! / records_to_dataframe(records, metadata) - include("test_issue40_mbp_dataframe.jl") # regression: MBP-1/MBP-10/BBO records_to_dataframe via nested levels (issue #40) - - # Run compatibility tests if the Rust CLI is available - dbn_cli_path = if Sys.iswindows() - joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn.exe") - else - joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn") - end - - if isfile(dbn_cli_path) - include("test_compatibility_updated.jl") # Updated cross-implementation compatibility testing - else - @warn "Skipping compatibility tests - Rust dbn-cli not found at $dbn_cli_path" - end - - # Import/export tests (optional - uncomment if needed) - # include("test_import_simple.jl") -end \ No newline at end of file +using Test +using DatabentoBinaryEncoding +import DatabentoBinaryEncoding as DBN +using Dates + +# Load test utilities (safe_rm, etc.) +include("test_utils.jl") + +@testset "DBN.jl Tests" begin + include("test_phase1.jl") + include("test_phase2.jl") + include("test_phase3.jl") + include("test_phase4.jl") + include("test_phase5.jl") + include("test_phase6.jl") + include("test_phase7.jl") + include("test_phase8.jl") + include("test_phase9_working.jl") # Edge cases and error handling + include("test_phase10_complete.jl") # Integration and performance testing + include("test_convenience_functions.jl") # Test all convenience read_*/foreach_* functions + include("test_phase11_typed_with_control.jl") # foreach_record_with_control: typed data + Union control split + include("test_replay.jl") # replay_dbn / replay_records: timestamp-paced re-emission + include("test_issue23_unset_stype.jl") # regression: 0xFF unset stype in v3 SymbolMappingMsg (issue #23) + include("test_issues_32_35.jl") # regressions: pre-v3 StatMsg layout, stat_to_dataframe, mapping intervals, invalid enum bytes (issues #32-#35) + include("test_show.jl") # compact one-line Base.show for record types + include("test_symbols.jl") # symbol_map / symbol_for / add_symbol_column! / records_to_dataframe(records, metadata) + include("test_issue40_mbp_dataframe.jl") # regression: MBP-1/MBP-10/BBO records_to_dataframe via nested levels (issue #40) + include("test_cme_2026_07.jl") # CME 2026-07 normalization: per-leg defs, standalone F_LAST, StatType/TradingEvent enums, flags + + # Run compatibility tests if the Rust CLI is available + dbn_cli_path = if Sys.iswindows() + joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn.exe") + else + joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn") + end + + if isfile(dbn_cli_path) + include("test_compatibility_updated.jl") # Updated cross-implementation compatibility testing + else + @warn "Skipping compatibility tests - Rust dbn-cli not found at $dbn_cli_path" + end + + # Import/export tests (optional - uncomment if needed) + # include("test_import_simple.jl") +end diff --git a/test/test_cme_2026_07.jl b/test/test_cme_2026_07.jl new file mode 100644 index 00000000..e00959c5 --- /dev/null +++ b/test/test_cme_2026_07.jl @@ -0,0 +1,229 @@ +# Coverage for Databento's CME (GLBX.MDP3) normalization changes of 2026-07-07 +# (https://databento.com/blog/cme-normalization-changes-2026-07) and the DBN +# enum/flag additions that go with them: +# 1. per-leg definition records: one InstrumentDefMsg per strategy leg +# 2. standalone F_LAST MBO record: action='N', flags=F_LAST, price UNDEF, size 0 +# 3. StatType UPPER_PRICE_LIMIT (17) / LOWER_PRICE_LIMIT (18) +# 4. TradingEvent IMPLIED_MATCHING_ON (3) / IMPLIED_MATCHING_OFF (4) +# 5. InstrumentClass FX_SPOT ('X') for FX spot, and INDEX ('I') from the spec +# +# Everything is round-tripped through the real encoder/decoder so the tests +# exercise the wire layout, not just the Julia constructors. + +@testset "CME 2026-07 normalization / enum coverage" begin + ts0 = Int64(1_783_411_200_000_000_000) # 2026-07-07T00:00:00Z + + mk_meta(schema) = Metadata(UInt8(3), "GLBX.MDP3", schema, ts0, + ts0 + 3_600_000_000_000, nothing, + SType.RAW_SYMBOL, SType.INSTRUMENT_ID, false, + String[], String[], String[], + Tuple{String,String,Int64,Int64}[]) + + function roundtrip(schema, records) + tmp = tempname() * ".dbn" + try + write_dbn(tmp, mk_meta(schema), records) + return read_dbn(tmp) + finally + safe_rm(tmp) + end + end + + @testset "enum and flag values match the DBN spec" begin + # StatType (UInt16) + @test UInt16(StatType.OPENING_PRICE) == 1 + @test UInt16(StatType.SETTLEMENT_PRICE) == 3 + @test UInt16(StatType.OPEN_INTEREST) == 9 + @test UInt16(StatType.UPPER_PRICE_LIMIT) == 17 + @test UInt16(StatType.LOWER_PRICE_LIMIT) == 18 + @test UInt16(StatType.AUCTION_COLLAR_LOWER_PRICE) == 26 + @test UInt16(StatType.VENUE_SPECIFIC_VOLUME_1) == 10001 + @test UInt16(StatType.VENUE_SPECIFIC_PRICE_1) == 10002 + @test safe_stat_type(17) == StatType.UPPER_PRICE_LIMIT + @test safe_stat_type(UInt16(18)) == StatType.LOWER_PRICE_LIMIT + @test safe_stat_type(0x7FFF) == StatType.UNKNOWN + @test safe_stat_type(0) == StatType.UNKNOWN + + # TradingEvent (UInt16) + @test UInt16(TradingEvent.NONE) == 0 + @test UInt16(TradingEvent.NO_CANCEL) == 1 + @test UInt16(TradingEvent.CHANGE_TRADING_SESSION) == 2 + @test UInt16(TradingEvent.IMPLIED_MATCHING_ON) == 3 + @test UInt16(TradingEvent.IMPLIED_MATCHING_OFF) == 4 + @test safe_trading_event(3) == TradingEvent.IMPLIED_MATCHING_ON + @test safe_trading_event(UInt16(4)) == TradingEvent.IMPLIED_MATCHING_OFF + @test safe_trading_event(999) == TradingEvent.NONE + + # InstrumentClass single-char codes + @test InstrumentClass.FX_SPOT == InstrumentClass.T(UInt8('X')) + @test InstrumentClass.INDEX == InstrumentClass.T(UInt8('I')) + @test DBN.safe_instrument_class(UInt8('X')) == InstrumentClass.FX_SPOT + @test DBN.safe_instrument_class(UInt8('I')) == InstrumentClass.INDEX + + # Record flags bit field + @test F_LAST == 0x80 + @test F_TOB == 0x40 + @test F_SNAPSHOT == 0x20 + @test F_MBP == 0x10 + @test F_BAD_TS_RECV == 0x08 + @test F_MAYBE_BAD_BOOK == 0x04 + @test F_PUBLISHER_SPECIFIC == 0x02 + @test has_flag(0x80, F_LAST) + @test has_flag(F_LAST | F_TOB, F_TOB) + @test has_flag(UInt8(0xA0), F_SNAPSHOT) + @test !has_flag(0x00, F_LAST) + @test !has_flag(F_TOB, F_LAST) + end + + @testset "standalone F_LAST MBO record round-trips" begin + # MBOMsg is 56 bytes -> hd.length 14 + hd(iid) = RecordHeader(UInt8(14), RType.MBO_MSG, UInt16(1), UInt32(iid), ts0) + add = MBOMsg(hd(42), UInt64(1001), Int64(5_000_000_000_000), UInt32(3), + 0x00, 0x00, Action.ADD, Side.BID, ts0 + 10, Int32(100), UInt32(1)) + # Event terminator as CME now publishes it: no book effect, only the + # F_LAST flag (and its own ts_recv) carry information. + last = MBOMsg(hd(42), UInt64(0), UNDEF_PRICE, UInt32(0), + F_LAST, 0x00, Action.NONE, Side.NONE, ts0 + 20, Int32(100), UInt32(2)) + + recs = roundtrip(Schema.MBO, [add, last]) + @test length(recs) == 2 + @test all(r -> r isa MBOMsg, recs) + + # The book update itself no longer carries F_LAST + @test !has_flag(recs[1].flags, F_LAST) + @test recs[1].action == Action.ADD + + r = recs[2] + @test r.action == Action.NONE + @test r.flags == F_LAST + @test has_flag(r.flags, F_LAST) + @test r.price == UNDEF_PRICE + @test r.size == 0 + @test r.side == Side.NONE + @test r.ts_recv == ts0 + 20 + @test r.hd.instrument_id == 42 + + # Export renders the undefined price as NaN, and keeps the flag byte + mbos = MBOMsg[x for x in recs] + df = records_to_dataframe(mbos) + @test size(df, 1) == 2 + @test isnan(df.price[2]) + @test df.flags[2] == F_LAST + @test df.size[2] == 0 + @test !isnan(df.price[1]) + + # Compact printing must not choke on the UNDEF_PRICE sentinel + s = sprint(show, r) + @test !isempty(s) + end + + @testset "price-limit statistics (stat_type 17/18) round-trip" begin + # v3 StatMsg is 80 bytes -> hd.length 20 + hd = RecordHeader(UInt8(20), RType.STAT_MSG, UInt16(1), UInt32(7), ts0) + mk_stat(stype, px, seq) = StatMsg(hd, UInt64(ts0 + 1), UInt64(ts0), Int64(px), + typemax(Int64), UInt32(seq), Int32(0), + UInt16(stype), UInt16(0), UInt8(1), UInt8(0)) + upper = mk_stat(StatType.UPPER_PRICE_LIMIT, 5_100_000_000_000, 1) + lower = mk_stat(StatType.LOWER_PRICE_LIMIT, 4_900_000_000_000, 2) + + recs = roundtrip(Schema.STATISTICS, [upper, lower]) + @test length(recs) == 2 + @test all(r -> r isa StatMsg, recs) + @test recs[1].stat_type == 17 + @test recs[2].stat_type == 18 + @test safe_stat_type(recs[1].stat_type) == StatType.UPPER_PRICE_LIMIT + @test safe_stat_type(recs[2].stat_type) == StatType.LOWER_PRICE_LIMIT + @test recs[1].stat_type == UInt16(StatType.UPPER_PRICE_LIMIT) + @test recs[1].price == 5_100_000_000_000 + @test recs[2].price == 4_900_000_000_000 + # quantity is unset for price statistics + @test recs[1].quantity == typemax(Int64) + + df = records_to_dataframe(StatMsg[x for x in recs]) + @test df.stat_type == [17, 18] + end + + @testset "implied-matching status events round-trip" begin + # StatusMsg is 40 bytes -> hd.length 10 + hd = RecordHeader(UInt8(10), RType.STATUS_MSG, UInt16(1), UInt32(7), ts0) + mk_status(ev) = StatusMsg(hd, UInt64(ts0 + 1), UInt16(1), UInt16(0), UInt16(ev), + UInt8('Y'), UInt8('Y'), UInt8('~')) + on = mk_status(TradingEvent.IMPLIED_MATCHING_ON) + off = mk_status(TradingEvent.IMPLIED_MATCHING_OFF) + + recs = roundtrip(Schema.STATUS, [on, off]) + @test length(recs) == 2 + @test all(r -> r isa StatusMsg, recs) + @test recs[1].trading_event == 3 + @test recs[2].trading_event == 4 + @test safe_trading_event(recs[1].trading_event) == TradingEvent.IMPLIED_MATCHING_ON + @test safe_trading_event(recs[2].trading_event) == TradingEvent.IMPLIED_MATCHING_OFF + @test recs[1].is_trading == UInt8('Y') + + df = records_to_dataframe(StatusMsg[x for x in recs]) + @test df.trading_event == [3, 4] + end + + @testset "per-leg definition records and FX-spot / index classes" begin + # v3 InstrumentDefMsg is 520 bytes -> hd.length 130 + function mk_def(iid, sym, cls; leg_count = 0, leg_index = 0, leg_iid = 0, + leg_sym = "", leg_side = Side.NONE, leg_cls = InstrumentClass.OTHER) + hd = RecordHeader(UInt8(130), RType.INSTRUMENT_DEF_MSG, UInt16(1), UInt32(iid), ts0) + InstrumentDefMsg(hd, ts0 + 1, + # prices / qty + 25_000_000, 1_000_000_000, + ts0 + 86_400_000_000_000, ts0 - 86_400_000_000_000, + UNDEF_PRICE, UNDEF_PRICE, UNDEF_PRICE, 0, 0, 0, 0, + # ints + 0, 0, UInt64(iid), 0, 10, 1, 1000, 1, 1, 1, 1, 1, 0, 1, 0, 1, 2026, 0, 1, + # strings + "USD", "USD", "", sym, "ES", "XCME", "ES", "FFICSX", "FUT", "Index", "", "", + # class / misc + cls, UNDEF_PRICE, 'F', 0, 0, 0, 0, 0, 0, 'A', 12, 0, 0, false, 0, 0, 0, + # legs + leg_count, leg_index, leg_iid, leg_sym, leg_side, 0, leg_cls, + 0, 0, 0, 0, 0, 0) + end + + # A straddle now produces one definition record per leg, sharing the + # strategy's instrument_id. + leg1 = mk_def(100, "UD:1V: VT 0625K5000", InstrumentClass.OPTION_SPREAD; + leg_count = 2, leg_index = 0, leg_iid = 201, leg_sym = "ESM6 C5000", + leg_side = Side.BID, leg_cls = InstrumentClass.CALL) + leg2 = mk_def(100, "UD:1V: VT 0625K5000", InstrumentClass.OPTION_SPREAD; + leg_count = 2, leg_index = 1, leg_iid = 202, leg_sym = "ESM6 P5000", + leg_side = Side.ASK, leg_cls = InstrumentClass.PUT) + # FX spot now reports instrument_class = 'X' instead of 'F' + fx = mk_def(300, "EURUSD", InstrumentClass.FX_SPOT) + # 'I' is in the DBN spec; make sure it survives instead of falling to OTHER + idx = mk_def(400, "SPX", InstrumentClass.INDEX) + + recs = roundtrip(Schema.DEFINITION, [leg1, leg2, fx, idx]) + @test length(recs) == 4 + @test all(r -> r isa InstrumentDefMsg, recs) + + legs = [r for r in recs if r.hd.instrument_id == 100] + @test length(legs) == 2 # one record per leg, not per strategy + @test all(r -> r.leg_count == 2, legs) + @test [r.leg_index for r in legs] == [0, 1] + @test [r.leg_instrument_id for r in legs] == [201, 202] + @test [r.leg_raw_symbol for r in legs] == ["ESM6 C5000", "ESM6 P5000"] + @test [r.leg_side for r in legs] == [Side.BID, Side.ASK] + @test [r.leg_instrument_class for r in legs] == [InstrumentClass.CALL, InstrumentClass.PUT] + @test all(r -> r.instrument_class == InstrumentClass.OPTION_SPREAD, legs) + @test all(r -> r.raw_symbol == "UD:1V: VT 0625K5000", legs) + + @test recs[3].instrument_class == InstrumentClass.FX_SPOT + @test recs[3].raw_symbol == "EURUSD" + @test recs[3].leg_count == 0 + + @test recs[4].instrument_class == InstrumentClass.INDEX + + # DataFrame export keeps one row per leg and the class names + df = records_to_dataframe(InstrumentDefMsg[x for x in recs]) + @test size(df, 1) == 4 + @test count(==(100), df.instrument_id) == 2 + @test occursin("FX_SPOT", df.instrument_class[3]) + @test occursin("INDEX", df.instrument_class[4]) + end +end From 514790818cdd93cef84b3df0fa09e0d76d370f28 Mon Sep 17 00:00:00 2001 From: Tyler Beason Date: Tue, 8 Sep 2026 15:21:58 -0500 Subject: [PATCH 2/3] Restore CRLF line endings on types.jl, DatabentoBinaryEncoding.jl, runtests.jl These three files are CRLF in main; the previous commit accidentally rewrote them as LF, turning the PR diff into a whole-file change. Content is unchanged. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01YBr6EZa1vHrJmPBoecSjRa --- src/DatabentoBinaryEncoding.jl | 270 +++--- src/types.jl | 1648 ++++++++++++++++---------------- test/runtests.jl | 90 +- 3 files changed, 1004 insertions(+), 1004 deletions(-) diff --git a/src/DatabentoBinaryEncoding.jl b/src/DatabentoBinaryEncoding.jl index f7ac875f..aeda549b 100644 --- a/src/DatabentoBinaryEncoding.jl +++ b/src/DatabentoBinaryEncoding.jl @@ -1,135 +1,135 @@ -""" - DatabentoBinaryEncoding - -Julia implementation of the Databento Binary Encoding (DBN) format for normalized market data. - -# Overview - -DatabentoBinaryEncoding.jl provides complete support for reading and writing DBN v3 format files with: -- Efficient streaming support for large files -- Automatic Zstd compression/decompression -- All DBN v3 message types -- Timestamp utilities with nanosecond precision -- Price conversion utilities with fixed-point arithmetic - -# Main Functions - -## Reading Data -- `read_dbn(filename)`: Read entire file into memory -- `DBNStream(filename)`: Memory-efficient streaming iterator -- `DBNDecoder(filename)`: Low-level decoder with manual control - -## Writing Data -- `write_dbn(filename, metadata, records)`: Write complete file -- `DBNStreamWriter(filename, dataset, schema)`: Real-time streaming writer -- `DBNEncoder(io, metadata)`: Low-level encoder - -## Compression -- `compress_dbn_file(input, output)`: Compress single file -- `compress_daily_files(date, directory)`: Batch compress files - -## Format Conversion -- `dbn_to_csv(input, output)`: Convert DBN to CSV -- `dbn_to_json(input, output)`: Convert DBN to JSON -- `dbn_to_parquet(input, output)`: Convert DBN to Parquet -- `json_to_dbn(input, output)`: Convert JSON to DBN -- `parquet_to_dbn(input, output)`: Convert Parquet to DBN -- `csv_to_dbn(input, output)`: Convert CSV to DBN - -## Utilities -- `price_to_float(price)` / `float_to_price(value)`: Price conversions -- `ts_to_datetime(ts)` / `datetime_to_ts(dt)`: Timestamp conversions -- `DBNTimestamp(ns)`: High-precision timestamp handling - -# Example Usage - -```julia -using DatabentoBinaryEncoding -# Or, for terser internal references: `import DatabentoBinaryEncoding as DBN` - -# Reading data -records = read_dbn("data.dbn") -for record in DBNStream("large_file.dbn.zst") - process(record) -end - -# Writing data -metadata = Metadata(3, "XNAS", Schema.TRADES, start_ts, end_ts, - length(records), SType.RAW_SYMBOL, SType.RAW_SYMBOL, - false, symbols, [], [], []) -write_dbn("output.dbn", metadata, records) - -# Streaming writer -writer = DBNStreamWriter("live.dbn", "XNAS", Schema.TRADES) -write_record!(writer, trade_msg) -close_writer!(writer) -``` - -# Supported Message Types - -- Market Data: `MBOMsg`, `TradeMsg`, `MBP1Msg`, `MBP10Msg`, `OHLCVMsg` -- Consolidated: `CMBP1Msg`, `CBBO1sMsg`, `CBBO1mMsg`, `TCBBOMsg`, `BBO1sMsg`, `BBO1mMsg` -- Status: `StatusMsg`, `ImbalanceMsg`, `StatMsg` -- System: `ErrorMsg`, `SymbolMappingMsg`, `SystemMsg` -- Definition: `InstrumentDefMsg` - -See the [DBN specification](https://databento.com/docs/standards-and-conventions/databento-binary-encoding) -for complete format documentation. -""" -module DatabentoBinaryEncoding - -# All using statements at the top -using Dates -using CodecZstd -using TranscodingStreams -using EnumX -using DataFrames -using CSV -using DuckDB -using DBInterface -using JSON3 -using StructTypes - - -# Include all the component files -include("types.jl") -include("messages.jl") -include("show.jl") -include("buffered_io.jl") -include("decode.jl") -include("encode.jl") -include("streaming.jl") -include("replay.jl") -include("export.jl") -include("symbols.jl") -include("import.jl") - -# Exports -export DBNDecoder, DBNEncoder, read_dbn, read_dbn_with_metadata, read_dbn_typed, write_dbn -export read_trades, read_mbo, read_mbp1, read_mbp10, read_tbbo # Market depth readers -export read_ohlcv, read_ohlcv_1s, read_ohlcv_1m, read_ohlcv_1h, read_ohlcv_1d # OHLCV readers -export read_cmbp1, read_cbbo1s, read_cbbo1m, read_tcbbo, read_bbo1s, read_bbo1m # Consolidated/BBO readers -export Metadata, DBNHeader, RecordHeader, DBNTimestamp -export MBOMsg, TradeMsg, MBP1Msg, MBP10Msg, OHLCVMsg, StatusMsg, ImbalanceMsg, StatMsg -export CMBP1Msg, CBBO1sMsg, CBBO1mMsg, TCBBOMsg, BBO1sMsg, BBO1mMsg -export ErrorMsg, SymbolMappingMsg, SystemMsg, InstrumentDefMsg -export DBNStream, DBNStreamWriter, write_record!, close_writer! -export foreach_record, foreach_record_with_control, foreach_trade, foreach_mbo, foreach_mbp1, foreach_mbp10, foreach_tbbo # Market depth streaming -export record_type_for_dbn_schema # Schema -> concrete record type -export foreach_ohlcv, foreach_ohlcv_1s, foreach_ohlcv_1m, foreach_ohlcv_1h, foreach_ohlcv_1d # OHLCV streaming -export foreach_cmbp1, foreach_cbbo1s, foreach_cbbo1m, foreach_tcbbo, foreach_bbo1s, foreach_bbo1m # Consolidated/BBO streaming -export replay_dbn, replay_records # Timestamp-paced replay -export compress_dbn_file, compress_daily_files -export Schema, Compression, Encoding, SType, RType, Action, Side, InstrumentClass -export StatType, TradingEvent, safe_stat_type, safe_trading_event -export F_LAST, F_TOB, F_SNAPSHOT, F_MBP, F_BAD_TS_RECV, F_MAYBE_BAD_BOOK, F_PUBLISHER_SPECIFIC, has_flag -export price_to_float, float_to_price, ts_to_datetime, datetime_to_ts, ts_to_date_time, date_time_to_ts, to_nanoseconds -export record_length_bytes -export DBN_VERSION, FIXED_PRICE_SCALE, UNDEF_PRICE, UNDEF_ORDER_SIZE, UNDEF_TIMESTAMP -export BidAskPair, VersionUpgradePolicy, DatasetCondition -export write_header, read_header!, write_record, read_record, finalize_encoder -export dbn_to_csv, dbn_to_json, dbn_to_parquet, records_to_dataframe -export symbol_map, symbol_for, add_symbol_column! -export json_to_dbn, parquet_to_dbn, csv_to_dbn - -end # module DatabentoBinaryEncoding +""" + DatabentoBinaryEncoding + +Julia implementation of the Databento Binary Encoding (DBN) format for normalized market data. + +# Overview + +DatabentoBinaryEncoding.jl provides complete support for reading and writing DBN v3 format files with: +- Efficient streaming support for large files +- Automatic Zstd compression/decompression +- All DBN v3 message types +- Timestamp utilities with nanosecond precision +- Price conversion utilities with fixed-point arithmetic + +# Main Functions + +## Reading Data +- `read_dbn(filename)`: Read entire file into memory +- `DBNStream(filename)`: Memory-efficient streaming iterator +- `DBNDecoder(filename)`: Low-level decoder with manual control + +## Writing Data +- `write_dbn(filename, metadata, records)`: Write complete file +- `DBNStreamWriter(filename, dataset, schema)`: Real-time streaming writer +- `DBNEncoder(io, metadata)`: Low-level encoder + +## Compression +- `compress_dbn_file(input, output)`: Compress single file +- `compress_daily_files(date, directory)`: Batch compress files + +## Format Conversion +- `dbn_to_csv(input, output)`: Convert DBN to CSV +- `dbn_to_json(input, output)`: Convert DBN to JSON +- `dbn_to_parquet(input, output)`: Convert DBN to Parquet +- `json_to_dbn(input, output)`: Convert JSON to DBN +- `parquet_to_dbn(input, output)`: Convert Parquet to DBN +- `csv_to_dbn(input, output)`: Convert CSV to DBN + +## Utilities +- `price_to_float(price)` / `float_to_price(value)`: Price conversions +- `ts_to_datetime(ts)` / `datetime_to_ts(dt)`: Timestamp conversions +- `DBNTimestamp(ns)`: High-precision timestamp handling + +# Example Usage + +```julia +using DatabentoBinaryEncoding +# Or, for terser internal references: `import DatabentoBinaryEncoding as DBN` + +# Reading data +records = read_dbn("data.dbn") +for record in DBNStream("large_file.dbn.zst") + process(record) +end + +# Writing data +metadata = Metadata(3, "XNAS", Schema.TRADES, start_ts, end_ts, + length(records), SType.RAW_SYMBOL, SType.RAW_SYMBOL, + false, symbols, [], [], []) +write_dbn("output.dbn", metadata, records) + +# Streaming writer +writer = DBNStreamWriter("live.dbn", "XNAS", Schema.TRADES) +write_record!(writer, trade_msg) +close_writer!(writer) +``` + +# Supported Message Types + +- Market Data: `MBOMsg`, `TradeMsg`, `MBP1Msg`, `MBP10Msg`, `OHLCVMsg` +- Consolidated: `CMBP1Msg`, `CBBO1sMsg`, `CBBO1mMsg`, `TCBBOMsg`, `BBO1sMsg`, `BBO1mMsg` +- Status: `StatusMsg`, `ImbalanceMsg`, `StatMsg` +- System: `ErrorMsg`, `SymbolMappingMsg`, `SystemMsg` +- Definition: `InstrumentDefMsg` + +See the [DBN specification](https://databento.com/docs/standards-and-conventions/databento-binary-encoding) +for complete format documentation. +""" +module DatabentoBinaryEncoding + +# All using statements at the top +using Dates +using CodecZstd +using TranscodingStreams +using EnumX +using DataFrames +using CSV +using DuckDB +using DBInterface +using JSON3 +using StructTypes + + +# Include all the component files +include("types.jl") +include("messages.jl") +include("show.jl") +include("buffered_io.jl") +include("decode.jl") +include("encode.jl") +include("streaming.jl") +include("replay.jl") +include("export.jl") +include("symbols.jl") +include("import.jl") + +# Exports +export DBNDecoder, DBNEncoder, read_dbn, read_dbn_with_metadata, read_dbn_typed, write_dbn +export read_trades, read_mbo, read_mbp1, read_mbp10, read_tbbo # Market depth readers +export read_ohlcv, read_ohlcv_1s, read_ohlcv_1m, read_ohlcv_1h, read_ohlcv_1d # OHLCV readers +export read_cmbp1, read_cbbo1s, read_cbbo1m, read_tcbbo, read_bbo1s, read_bbo1m # Consolidated/BBO readers +export Metadata, DBNHeader, RecordHeader, DBNTimestamp +export MBOMsg, TradeMsg, MBP1Msg, MBP10Msg, OHLCVMsg, StatusMsg, ImbalanceMsg, StatMsg +export CMBP1Msg, CBBO1sMsg, CBBO1mMsg, TCBBOMsg, BBO1sMsg, BBO1mMsg +export ErrorMsg, SymbolMappingMsg, SystemMsg, InstrumentDefMsg +export DBNStream, DBNStreamWriter, write_record!, close_writer! +export foreach_record, foreach_record_with_control, foreach_trade, foreach_mbo, foreach_mbp1, foreach_mbp10, foreach_tbbo # Market depth streaming +export record_type_for_dbn_schema # Schema -> concrete record type +export foreach_ohlcv, foreach_ohlcv_1s, foreach_ohlcv_1m, foreach_ohlcv_1h, foreach_ohlcv_1d # OHLCV streaming +export foreach_cmbp1, foreach_cbbo1s, foreach_cbbo1m, foreach_tcbbo, foreach_bbo1s, foreach_bbo1m # Consolidated/BBO streaming +export replay_dbn, replay_records # Timestamp-paced replay +export compress_dbn_file, compress_daily_files +export Schema, Compression, Encoding, SType, RType, Action, Side, InstrumentClass +export StatType, TradingEvent, safe_stat_type, safe_trading_event +export F_LAST, F_TOB, F_SNAPSHOT, F_MBP, F_BAD_TS_RECV, F_MAYBE_BAD_BOOK, F_PUBLISHER_SPECIFIC, has_flag +export price_to_float, float_to_price, ts_to_datetime, datetime_to_ts, ts_to_date_time, date_time_to_ts, to_nanoseconds +export record_length_bytes +export DBN_VERSION, FIXED_PRICE_SCALE, UNDEF_PRICE, UNDEF_ORDER_SIZE, UNDEF_TIMESTAMP +export BidAskPair, VersionUpgradePolicy, DatasetCondition +export write_header, read_header!, write_record, read_record, finalize_encoder +export dbn_to_csv, dbn_to_json, dbn_to_parquet, records_to_dataframe +export symbol_map, symbol_for, add_symbol_column! +export json_to_dbn, parquet_to_dbn, csv_to_dbn + +end # module DatabentoBinaryEncoding diff --git a/src/types.jl b/src/types.jl index 418e4e6a..c7507e54 100644 --- a/src/types.jl +++ b/src/types.jl @@ -1,824 +1,824 @@ -# DBN types, enums, and data structures - -# Constants -"""The current DBN format version supported by this implementation.""" -const DBN_VERSION = 3 - -"""Fixed-length symbol string size for DBN v2/v3 metadata and records.""" -const SYMBOL_CSTR_LEN = 71 - -"""Fixed-point price scaling factor for converting between integer and float prices.""" -const FIXED_PRICE_SCALE = Int32(1_000_000_000) - -"""Sentinel value indicating an undefined or missing price.""" -const UNDEF_PRICE = typemax(Int64) - -"""Sentinel value indicating an undefined or missing order size.""" -const UNDEF_ORDER_SIZE = typemax(UInt32) - -"""Sentinel value indicating an undefined or missing timestamp.""" -const UNDEF_TIMESTAMP = typemax(Int64) - -"""Multiplier for converting the length field to bytes in record headers.""" -const LENGTH_MULTIPLIER = 4 - -# Enums using EnumX for better namespace management - -""" - Schema - -DBN data schemas representing different types of market data. - -# Values -- `MBO`: Market-by-order data -- `MBP_1`: Market-by-price with book depth 1 (TBBO) -- `MBP_10`: Market-by-price with book depth 10 -- `TBBO`: Top-of-book bid/offer -- `TRADES`: Trade messages only -- `OHLCV_1S`: OHLCV data at 1-second intervals -- `OHLCV_1M`: OHLCV data at 1-minute intervals -- `OHLCV_1H`: OHLCV data at 1-hour intervals -- `OHLCV_1D`: OHLCV data at 1-day intervals -- `DEFINITION`: Instrument definition data -- `STATISTICS`: Market statistics -- `STATUS`: Exchange status messages -- `IMBALANCE`: Order imbalance data -- `CBBO`: Consolidated best bid/offer -- `CBBO_1S`: Consolidated BBO at 1-second intervals -- `CBBO_1M`: Consolidated BBO at 1-minute intervals -- `CMBP_1`: Consolidated market-by-price depth 1 -- `TCBBO`: Trade-consolidated BBO -- `BBO_1S`: BBO at 1-second intervals -- `BBO_1M`: BBO at 1-minute intervals -""" -@enumx Schema::UInt16 begin - MBO = 0 - MBP_1 = 1 - MBP_10 = 2 - TBBO = 3 - TRADES = 4 - OHLCV_1S = 5 - OHLCV_1M = 6 - OHLCV_1H = 7 - OHLCV_1D = 8 - DEFINITION = 9 - STATISTICS = 10 - STATUS = 11 - IMBALANCE = 12 - CBBO = 13 - CBBO_1S = 14 - CBBO_1M = 15 - CMBP_1 = 16 - TCBBO = 17 - BBO_1S = 18 - BBO_1M = 19 - MIX = 0xFFFF -end - -""" - Compression - -Compression algorithms supported for DBN files. - -# Values -- `NONE`: No compression -- `ZSTD`: Zstandard compression -""" -@enumx Compression::UInt8 begin - NONE = 0 - ZSTD = 1 -end - -""" - Encoding - -Output encoding formats for market data. - -# Values -- `DBN`: Databento Binary Encoding -- `CSV`: Comma-separated values -- `JSON`: JavaScript Object Notation -""" -@enumx Encoding::UInt8 begin - DBN = 0 - CSV = 1 - JSON = 2 -end - -""" - SType - -Symbol types for identifying instruments in DBN data. Numeric values match the -official Databento DBN spec — wire-encoded as a `UInt8` in metadata and in -`SymbolMappingMsg` (v2+). - -# Values -- `INSTRUMENT_ID = 0`: Numeric instrument identifier -- `RAW_SYMBOL = 1`: Raw symbol string from exchange -- `SMART = 2`: Deprecated alias (was split into `CONTINUOUS` and `PARENT`) -- `CONTINUOUS = 3`: Continuous contract symbol -- `PARENT = 4`: Parent symbol for derived instruments (e.g. `SPXW.OPT`) -- `NASDAQ_SYMBOL = 5`: Nasdaq-specific symbol -- `CMS_SYMBOL = 6`: CMS symbol -- `ISIN = 7`: ISO 6166 International Securities Identification Number -- `US_CODE = 8`: US Code (CUSIP-style) -- `BBG_COMP_ID = 9`: Bloomberg composite ID -- `BBG_COMP_TICKER = 10`: Bloomberg composite ticker -- `FIGI = 11`: OpenFIGI identifier -- `FIGI_TICKER = 12`: OpenFIGI ticker -- `UNDEF = 255`: Unset/undefined sentinel (`0xFF`). Databento wire-encodes an - unset `stype` as `0xFF`; this member lets non-nullable `stype` fields (e.g. - in `SymbolMappingMsg`) represent and round-trip that sentinel. Metadata's - nullable `stype_in` decodes `0xFF` to `nothing` instead. -""" -@enumx SType::UInt8 begin - INSTRUMENT_ID = 0 - RAW_SYMBOL = 1 - SMART = 2 # deprecated (kept for round-trip with v1 wire data) - CONTINUOUS = 3 - PARENT = 4 - NASDAQ_SYMBOL = 5 - CMS_SYMBOL = 6 - ISIN = 7 - US_CODE = 8 - BBG_COMP_ID = 9 - BBG_COMP_TICKER = 10 - FIGI = 11 - FIGI_TICKER = 12 - UNDEF = 255 # 0xFF "unset" sentinel; see docstring -end - -""" - RType - -Record types for different kinds of market data messages in DBN format. - -# Values -- `MBP_0_MSG`: Trades (book depth 0) -- `MBP_1_MSG`: TBBO/MBP-1 (book depth 1) -- `MBP_10_MSG`: MBP-10 (book depth 10) -- `STATUS_MSG`: Exchange status record -- `INSTRUMENT_DEF_MSG`: Instrument definition record -- `IMBALANCE_MSG`: Order imbalance record -- `ERROR_MSG`: Error record from live gateway -- `SYMBOL_MAPPING_MSG`: Symbol mapping record from live gateway -- `SYSTEM_MSG`: Non-error record from live gateway -- `STAT_MSG`: Statistics record from publisher -- `OHLCV_1S_MSG`: OHLCV at 1-second cadence -- `OHLCV_1M_MSG`: OHLCV at 1-minute cadence -- `OHLCV_1H_MSG`: OHLCV at hourly cadence -- `OHLCV_1D_MSG`: OHLCV at daily cadence -- `MBO_MSG`: Market-by-order record -- `CMBP_1_MSG`: Consolidated market-by-price with book depth 1 -- `CBBO_1S_MSG`: Consolidated market-by-price with book depth 1 at 1-second cadence -- `CBBO_1M_MSG`: Consolidated market-by-price with book depth 1 at 1-minute cadence -- `TCBBO_MSG`: Consolidated market-by-price with book depth 1 (trades only) -- `BBO_1S_MSG`: Market-by-price with book depth 1 at 1-second cadence -- `BBO_1M_MSG`: Market-by-price with book depth 1 at 1-minute cadence -""" -@enumx RType::UInt8 begin - MBP_0_MSG = 0x00 # Trades (book depth 0) - MBP_1_MSG = 0x01 # TBBO/MBP-1 (book depth 1) - MBP_10_MSG = 0x0A # MBP-10 (book depth 10) - STATUS_MSG = 0x12 # Exchange status record - INSTRUMENT_DEF_MSG = 0x13 # Instrument definition record - IMBALANCE_MSG = 0x14 # Order imbalance record - ERROR_MSG = 0x15 # Error record from live gateway - SYMBOL_MAPPING_MSG = 0x16 # Symbol mapping record from live gateway - SYSTEM_MSG = 0x17 # Non-error record from live gateway - STAT_MSG = 0x18 # Statistics record from publisher - OHLCV_1S_MSG = 0x20 # OHLCV at 1-second cadence - OHLCV_1M_MSG = 0x21 # OHLCV at 1-minute cadence - OHLCV_1H_MSG = 0x22 # OHLCV at hourly cadence - OHLCV_1D_MSG = 0x23 # OHLCV at daily cadence - MBO_MSG = 0xA0 # Market-by-order record - CMBP_1_MSG = 0xB1 # Consolidated market-by-price with book depth 1 - CBBO_1S_MSG = 0xC0 # Consolidated market-by-price with book depth 1 at 1-second cadence - CBBO_1M_MSG = 0xC1 # Consolidated market-by-price with book depth 1 at 1-minute cadence - TCBBO_MSG = 0xC2 # Consolidated market-by-price with book depth 1 (trades only) - BBO_1S_MSG = 0xC3 # Market-by-price with book depth 1 at 1-second cadence - BBO_1M_MSG = 0xC4 # Market-by-price with book depth 1 at 1-minute cadence -end - -""" - Action - -Market actions that can be applied to orders or trades. - -# Values -- `ADD`: Insert a new order into the book -- `MODIFY`: Change an order's price and/or size -- `CANCEL`: Fully or partially cancel an order from the book -- `CLEAR`: Remove all resting orders for the instrument -- `TRADE`: An aggressing order traded. Does not affect the book -- `FILL`: A resting order was filled. Does not affect the book -- `NONE`: No action: does not affect the book, but may carry flags or other information -""" -@enumx Action::UInt8 begin - ADD = UInt8('A') # Insert a new order into the book - MODIFY = UInt8('M') # Change an order's price and/or size - CANCEL = UInt8('C') # Fully or partially cancel an order from the book - CLEAR = UInt8('R') # Remove all resting orders for the instrument - TRADE = UInt8('T') # An aggressing order traded. Does not affect the book - FILL = UInt8('F') # A resting order was filled. Does not affect the book - NONE = UInt8('N') # No action: does not affect the book, but may carry flags or other information -end - -""" - Side - -Market sides for orders and trades. - -# Values -- `ASK`: Ask/offer side (sell orders) -- `BID`: Bid side (buy orders) -- `NONE`: No specific side or not applicable -""" -@enumx Side::UInt8 begin - ASK = UInt8('A') - BID = UInt8('B') - NONE = UInt8('N') -end - -""" - InstrumentClass - -Classification of financial instruments. Values match Databento's DBN -`InstrumentClass` enum. - -# Values -- `STOCK`: Equity instruments -- `CALL`: Call option contracts -- `PUT`: Put option contracts -- `FUTURE`: Futures contracts -- `INDEX`: Index instruments -- `BOND`: Fixed income securities -- `MIXED_SPREAD`: Mixed spread instruments -- `FUTURE_SPREAD`: Futures spread instruments -- `OPTION_SPREAD`: Option spread instruments -- `FX_SPOT`: Foreign exchange spot -- `COMMODITY_SPOT`: Commodity spot -- `UNKNOWN_0`, `UNKNOWN_45`: Numeric fallback values for unknown classes -""" -@enumx InstrumentClass::UInt8 begin - BOND = UInt8('B') - CALL = UInt8('C') - FUTURE = UInt8('F') - INDEX = UInt8('I') - STOCK = UInt8('K') - MIXED_SPREAD = UInt8('M') - PUT = UInt8('P') - FUTURE_SPREAD = UInt8('S') - OPTION_SPREAD = UInt8('T') - FX_SPOT = UInt8('X') - COMMODITY_SPOT = UInt8('Y') - # Also support numeric values - UNKNOWN_0 = 0 - UNKNOWN_45 = 45 - OTHER = UInt8('?') -end - -# Basic structures - -""" - VersionUpgradePolicy - -Encapsulates the version upgrade policy for DBN files. - -# Fields -- `upgrade_policy::UInt8`: Policy for handling version upgrades -""" -struct VersionUpgradePolicy - upgrade_policy::UInt8 -end - -""" - DatasetCondition - -Conditions and constraints for a dataset. - -# Fields -- `last_ts_out::Int64`: Last timestamp output -- `start_ts::Int64`: Dataset start timestamp -- `end_ts::Int64`: Dataset end timestamp -- `limit::UInt64`: Record limit for the dataset -""" -struct DatasetCondition - last_ts_out::Int64 - start_ts::Int64 - end_ts::Int64 - limit::UInt64 -end - -""" - Metadata - -Metadata information for a DBN dataset. - -# Fields -- `version::UInt8`: DBN format version -- `dataset::String`: Dataset identifier -- `schema::Schema.T`: Data schema type -- `start_ts::Int64`: Start timestamp for the data -- `end_ts::Union{Int64,Nothing}`: End timestamp (can be null) -- `limit::Union{UInt64,Nothing}`: Record count limit (can be null) -- `stype_in::Union{SType.T,Nothing}`: Input symbol type (can be null) -- `stype_out::SType.T`: Output symbol type -- `ts_out::Bool`: Whether timestamps are included in output -- `symbols::Vector{String}`: List of symbols in the dataset -- `partial::Vector{String}`: Partially available symbols -- `not_found::Vector{String}`: Symbols that were not found -- `mappings::Vector{Tuple{String,String,Int64,Int64}}`: Symbol mappings as - `(raw_symbol, mapped_symbol, start_date, end_date)`, one tuple per mapping - interval. A symbol with multiple intervals (e.g. a continuous contract's roll - history) contributes consecutive tuples sharing the same raw symbol. Dates - are raw `YYYYMMDD` integers. -""" -struct Metadata - version::UInt8 - dataset::String - schema::Schema.T - start_ts::Int64 - end_ts::Union{Int64,Nothing} # Can be null - limit::Union{UInt64,Nothing} # Can be null - stype_in::Union{SType.T,Nothing} # Can be null - stype_out::SType.T - ts_out::Bool - symbols::Vector{String} - partial::Vector{String} - not_found::Vector{String} - mappings::Vector{Tuple{String,String,Int64,Int64}} -end - -""" - DBNHeader - -Complete header information for a DBN file. - -# Fields -- `version_upgrade_policy::VersionUpgradePolicy`: Version handling policy -- `dataset_condition::DatasetCondition`: Dataset conditions and constraints -- `metadata::Metadata`: Dataset metadata -""" -struct DBNHeader - version_upgrade_policy::VersionUpgradePolicy - dataset_condition::DatasetCondition - metadata::Metadata -end - -""" - RecordHeader - -Standard header present in all DBN record types. - -# Fields -- `length::UInt8`: Length of the record in 4-byte units (multiply by LENGTH_MULTIPLIER for bytes) -- `rtype::RType.T`: Record type identifier -- `publisher_id::UInt16`: Publisher/venue identifier -- `instrument_id::UInt32`: Instrument identifier -- `ts_event::Int64`: Event timestamp in nanoseconds since Unix epoch -""" -struct RecordHeader - length::UInt8 - rtype::RType.T - publisher_id::UInt16 - instrument_id::UInt32 - ts_event::Int64 -end - -""" - record_length_bytes(hd::RecordHeader) - -Get the actual record length in bytes from a RecordHeader. -The length field stores 4-byte units, so multiply by LENGTH_MULTIPLIER. -""" -record_length_bytes(hd::RecordHeader) = hd.length * LENGTH_MULTIPLIER - -""" - BidAskPair - -Bid and ask price/size information for market data. - -# Fields -- `bid_px::Int64`: Bid price (scaled by FIXED_PRICE_SCALE) -- `ask_px::Int64`: Ask price (scaled by FIXED_PRICE_SCALE) -- `bid_sz::UInt32`: Bid size/quantity -- `ask_sz::UInt32`: Ask size/quantity -- `bid_ct::UInt32`: Number of bid orders -- `ask_ct::UInt32`: Number of ask orders -""" -struct BidAskPair - bid_px::Int64 - ask_px::Int64 - bid_sz::UInt32 - ask_sz::UInt32 - bid_ct::UInt32 - ask_ct::UInt32 -end - -# Timestamp utilities - -""" - DBNTimestamp - -High-precision timestamp representation with nanosecond accuracy. - -# Fields -- `seconds::Int64`: Unix epoch seconds -- `nanoseconds::Int32`: Nanoseconds within the second (0-999_999_999) -""" -struct DBNTimestamp - seconds::Int64 # Unix epoch seconds - nanoseconds::Int32 # Nanoseconds within the second (0-999_999_999) -end - -""" - DBNTimestamp(ns::Int64) - -Construct a DBNTimestamp from nanoseconds since Unix epoch. - -# Arguments -- `ns::Int64`: Nanoseconds since Unix epoch - -# Returns -- `DBNTimestamp`: Timestamp split into seconds and nanoseconds components -""" -function DBNTimestamp(ns::Int64) - if ns == UNDEF_TIMESTAMP - return DBNTimestamp(UNDEF_TIMESTAMP, 0) - end - seconds = ns ÷ 1_000_000_000 - nanoseconds = Int32(ns % 1_000_000_000) - return DBNTimestamp(seconds, nanoseconds) -end - -""" - to_nanoseconds(ts::DBNTimestamp) - -Convert a DBNTimestamp back to nanoseconds since Unix epoch. - -# Arguments -- `ts::DBNTimestamp`: Timestamp to convert - -# Returns -- `Int64`: Nanoseconds since Unix epoch, or UNDEF_TIMESTAMP if undefined -""" -function to_nanoseconds(ts::DBNTimestamp) - if ts.seconds == UNDEF_TIMESTAMP - return UNDEF_TIMESTAMP - end - return ts.seconds * 1_000_000_000 + ts.nanoseconds -end - -""" - ts_to_datetime(ts::Int64) - -Convert a nanosecond timestamp to DateTime with nanosecond precision information. - -# Arguments -- `ts::Int64`: Nanoseconds since Unix epoch - -# Returns -- `NamedTuple`: Contains `datetime` (DateTime) and `nanoseconds` (Int32), or `nothing` if undefined -""" -function ts_to_datetime(ts::Int64) - if ts == UNDEF_TIMESTAMP - return nothing - end - # Returns DateTime with millisecond precision and separate nanosecond component - dbn_ts = DBNTimestamp(ts) - dt = unix2datetime(Float64(dbn_ts.seconds) + dbn_ts.nanoseconds / 1_000_000_000) - return (datetime=dt, nanoseconds=dbn_ts.nanoseconds) -end - -""" - datetime_to_ts(dt::DateTime, nanoseconds::Int32=0) - -Convert a DateTime with optional nanosecond precision to nanosecond timestamp. - -# Arguments -- `dt::DateTime`: DateTime to convert -- `nanoseconds::Int32`: Additional nanoseconds within the second (default: 0) - -# Returns -- `Int64`: Nanoseconds since Unix epoch -""" -function datetime_to_ts(dt::DateTime, nanoseconds::Union{Int32,Int64}=0) - # Convert DateTime to nanoseconds, preserving additional precision - seconds = Int64(round(datetime2unix(dt))) - return seconds * 1_000_000_000 + nanoseconds -end - -""" - ts_to_date_time(ts::Int64) - -Convert a nanosecond timestamp to separate Date and Time components with full nanosecond precision. - -# Arguments -- `ts::Int64`: Nanoseconds since Unix epoch - -# Returns -- `NamedTuple`: Contains `date` (Date), `time` (Time), and `timestamp` (DBNTimestamp), or `nothing` if undefined -""" -function ts_to_date_time(ts::Int64) - if ts == UNDEF_TIMESTAMP - return nothing - end - dbn_ts = DBNTimestamp(ts) - - # Get the date part - dt_seconds = unix2datetime(Float64(dbn_ts.seconds)) - date_part = Date(dt_seconds) - - # Get time within the day with nanosecond precision - seconds_in_day = dbn_ts.seconds % 86400 - time_ns = seconds_in_day * 1_000_000_000 + dbn_ts.nanoseconds - time_part = Dates.Time(Dates.Nanosecond(time_ns)) - - return (date=date_part, time=time_part, timestamp=dbn_ts) -end - -""" - date_time_to_ts(date::Date, time::Dates.Time) - -Convert separate Date and Time components to nanosecond timestamp. - -# Arguments -- `date::Date`: Date component -- `time::Dates.Time`: Time component with nanosecond precision - -# Returns -- `Int64`: Nanoseconds since Unix epoch -""" -function date_time_to_ts(date::Date, time::Dates.Time) - # Convert date to seconds since epoch - dt = DateTime(date) - date_seconds = Int64(round(datetime2unix(dt))) - - # Extract nanoseconds from time - time_ns = Dates.value(time) # Total nanoseconds since midnight - - return date_seconds * 1_000_000_000 + time_ns -end - -# Price conversion utilities - -""" - price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) - -Convert a fixed-point price to floating-point representation. - -# Arguments -- `price::Int64`: Fixed-point price value -- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) - -# Returns -- `Float64`: Floating-point price, or NaN if price is UNDEF_PRICE -""" -function price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) - if price == UNDEF_PRICE - return NaN - end - return Float64(price) / Float64(scale) -end - -""" - float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) - -Convert a floating-point price to fixed-point representation. - -# Arguments -- `value::Float64`: Floating-point price -- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) - -# Returns -- `Int64`: Fixed-point price, or UNDEF_PRICE if value is NaN or infinite -""" -function float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) - if isnan(value) || isinf(value) - return UNDEF_PRICE - end - return Int64(round(value * Float64(scale))) -end - -# Helper functions for safe enum conversion. -# All enums share one policy: an invalid byte warns (once per enum) and maps to -# a sentinel value instead of throwing, so one bad byte can't kill a decode -# stream. Lookup tables keep the hot path free of try/catch. - -function _build_enum_lookup(::Type{E}, default::E) where {E} - table = fill(default, 256) - valid = falses(256) - for inst in instances(E) - idx = Int(UInt8(inst)) + 1 - table[idx] = inst - valid[idx] = true - end - return table, valid -end - -const _ACTION_LOOKUP, _ACTION_VALID = _build_enum_lookup(Action.T, Action.NONE) -const _SIDE_LOOKUP, _SIDE_VALID = _build_enum_lookup(Side.T, Side.NONE) -const _INSTRUMENT_CLASS_LOOKUP, _INSTRUMENT_CLASS_VALID = - _build_enum_lookup(InstrumentClass.T, InstrumentClass.OTHER) - -""" - safe_action(raw_val::UInt8) - -Convert a raw byte value to an Action enum. `0x00` and invalid values map to -`Action.NONE`; invalid values additionally log a warning (once). -""" -@inline function safe_action(raw_val::UInt8) - # 0 indicates no action for certain record types - raw_val == 0x00 && return Action.NONE - idx = Int(raw_val) + 1 - @inbounds if !_ACTION_VALID[idx] - @warn "Unknown Action value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 - end - return @inbounds _ACTION_LOOKUP[idx] -end - -""" - safe_side(raw_val::UInt8) - -Convert a raw byte value to a Side enum. `0x00` and invalid values map to -`Side.NONE`; invalid values additionally log a warning (once). -""" -@inline function safe_side(raw_val::UInt8) - # 0 indicates no side for certain record types - raw_val == 0x00 && return Side.NONE - idx = Int(raw_val) + 1 - @inbounds if !_SIDE_VALID[idx] - @warn "Unknown Side value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 - end - return @inbounds _SIDE_LOOKUP[idx] -end - -""" - safe_instrument_class(raw_val::UInt8) - -Convert a raw byte value to an InstrumentClass enum. Invalid values log a -warning (once) and map to `InstrumentClass.OTHER`. -""" -@inline function safe_instrument_class(raw_val::UInt8) - idx = Int(raw_val) + 1 - @inbounds if !_INSTRUMENT_CLASS_VALID[idx] - @warn "Unknown InstrumentClass value: $raw_val (0x$(string(raw_val, base=16))), using OTHER as default" maxlog = 1 - end - return @inbounds _INSTRUMENT_CLASS_LOOKUP[idx] -end - -# --------------------------------------------------------------------------- -# Venue-code enums and record flags -# -# `StatMsg.stat_type` and `StatusMsg.trading_event` stay raw `UInt16` on the -# record structs: the wire fields are plain integers and publishers may emit -# codes outside the published enum. The enums below are for *interpreting* -# those codes. Compare with `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)` -# or convert with `safe_stat_type(r.stat_type)` / `safe_trading_event(...)`. -# --------------------------------------------------------------------------- - -""" - StatType - -Type of statistic carried by a [`StatMsg`](@ref) (`stat_type` field). Values -match the official DBN `StatType` enum (`UInt16`). `UNKNOWN = 0` is a local -sentinel returned by [`safe_stat_type`](@ref) for codes this package does not -recognize (publisher-specific, or newer than this release). - -# Values -- `OPENING_PRICE = 1`: Price of the first trade of an instrument -- `INDICATIVE_OPENING_PRICE = 2`: Probable opening price, published pre-open -- `SETTLEMENT_PRICE = 3`: Settlement price -- `TRADING_SESSION_LOW_PRICE = 4`: Lowest trade price of the session -- `TRADING_SESSION_HIGH_PRICE = 5`: Highest trade price of the session -- `CLEARED_VOLUME = 6`: Contracts cleared on the previous trading date -- `LOWEST_OFFER = 7`: Lowest offer price of the session -- `HIGHEST_BID = 8`: Highest bid price of the session -- `OPEN_INTEREST = 9`: Number of outstanding contracts -- `FIXING_PRICE = 10`: VWAP over a fixing period -- `CLOSE_PRICE = 11`: Last trade price of the session -- `NET_CHANGE = 12`: Change from the previous session's close -- `VWAP = 13`: Session volume-weighted average price -- `VOLATILITY = 14`: Implied volatility associated with the settlement price -- `DELTA = 15`: Option delta associated with the settlement price -- `UNCROSSING_PRICE = 16`: Auction uncrossing price -- `UPPER_PRICE_LIMIT = 17`: Exchange-defined upper price limit (published for - CME GLBX.MDP3 since the 2026-07 normalization change) -- `LOWER_PRICE_LIMIT = 18`: Exchange-defined lower price limit (as above) -- `BLOCK_VOLUME = 19`: Block contracts cleared on the previous trading date -- `INDICATIVE_CLOSE_PRICE = 20`: Probable closing price -- `MWCB_LEVEL_1 = 21`, `MWCB_LEVEL_2 = 22`, `MWCB_LEVEL_3 = 23`: Market-wide - circuit-breaker thresholds (7% / 13% / 20%) -- `AUCTION_COLLAR_REFERENCE_PRICE = 24`, `AUCTION_COLLAR_UPPER_PRICE = 25`, - `AUCTION_COLLAR_LOWER_PRICE = 26`: Auction collar prices -- `VENUE_SPECIFIC_VOLUME_1 = 10001`, `VENUE_SPECIFIC_PRICE_1 = 10002`: - Venue-specific statistics -- `UNKNOWN = 0`: Unrecognized code (local sentinel, not part of the DBN spec) -""" -@enumx StatType::UInt16 begin - UNKNOWN = 0 - OPENING_PRICE = 1 - INDICATIVE_OPENING_PRICE = 2 - SETTLEMENT_PRICE = 3 - TRADING_SESSION_LOW_PRICE = 4 - TRADING_SESSION_HIGH_PRICE = 5 - CLEARED_VOLUME = 6 - LOWEST_OFFER = 7 - HIGHEST_BID = 8 - OPEN_INTEREST = 9 - FIXING_PRICE = 10 - CLOSE_PRICE = 11 - NET_CHANGE = 12 - VWAP = 13 - VOLATILITY = 14 - DELTA = 15 - UNCROSSING_PRICE = 16 - UPPER_PRICE_LIMIT = 17 - LOWER_PRICE_LIMIT = 18 - BLOCK_VOLUME = 19 - INDICATIVE_CLOSE_PRICE = 20 - MWCB_LEVEL_1 = 21 - MWCB_LEVEL_2 = 22 - MWCB_LEVEL_3 = 23 - AUCTION_COLLAR_REFERENCE_PRICE = 24 - AUCTION_COLLAR_UPPER_PRICE = 25 - AUCTION_COLLAR_LOWER_PRICE = 26 - VENUE_SPECIFIC_VOLUME_1 = 10001 - VENUE_SPECIFIC_PRICE_1 = 10002 -end - -""" - TradingEvent - -Additional context for a [`StatusMsg`](@ref) (`trading_event` field). Values -match the official DBN `TradingEvent` enum (`UInt16`). - -# Values -- `NONE = 0`: No additional information given -- `NO_CANCEL = 1`: Order entry is allowed; modification and cancellation are not -- `CHANGE_TRADING_SESSION = 2`: A change of trading session occurred; daily - statistics are reset -- `IMPLIED_MATCHING_ON = 3`: Implied matching is available (CME's matching - engine is constructing implied depth) -- `IMPLIED_MATCHING_OFF = 4`: Implied matching is not available - -CME GLBX.MDP3 publishes `IMPLIED_MATCHING_ON`/`OFF` status records since the -2026-07 normalization change. -""" -@enumx TradingEvent::UInt16 begin - NONE = 0 - NO_CANCEL = 1 - CHANGE_TRADING_SESSION = 2 - IMPLIED_MATCHING_ON = 3 - IMPLIED_MATCHING_OFF = 4 -end - -const _STAT_TYPE_LOOKUP = Dict{UInt16,StatType.T}(UInt16(v) => v for v in instances(StatType.T)) -const _TRADING_EVENT_LOOKUP = Dict{UInt16,TradingEvent.T}(UInt16(v) => v for v in instances(TradingEvent.T)) - -""" - safe_stat_type(raw::Integer) -> StatType.T - -Interpret a raw `StatMsg.stat_type` code. Unrecognized codes map to -`StatType.UNKNOWN` without a warning (publisher-specific codes are legitimate). -""" -safe_stat_type(raw::Integer) = get(_STAT_TYPE_LOOKUP, UInt16(raw), StatType.UNKNOWN) - -""" - safe_trading_event(raw::Integer) -> TradingEvent.T - -Interpret a raw `StatusMsg.trading_event` code. Unrecognized codes map to -`TradingEvent.NONE` ("no additional information") without a warning. -""" -safe_trading_event(raw::Integer) = get(_TRADING_EVENT_LOOKUP, UInt16(raw), TradingEvent.NONE) - -# Record `flags` bit field (MBO / MBP / trade / BBO records). Values match the -# official DBN `flags` module. - -""" -Flag bit: last record in the event for a given `instrument_id`. Since the -2026-07 CME normalization change, GLBX.MDP3 MBO emits this on a standalone -record (`action = Action.NONE`, `price = UNDEF_PRICE`, `size = 0`) that -follows the book updates, rather than on the final book update itself. -""" -const F_LAST = 0x80 -"""Flag bit: top-of-book record, not an individual order.""" -const F_TOB = 0x40 -"""Flag bit: record sourced from a replay, such as a snapshot server.""" -const F_SNAPSHOT = 0x20 -"""Flag bit: aggregated price-level record, not an individual order.""" -const F_MBP = 0x10 -"""Flag bit: `ts_recv` is inaccurate due to clock issues or packet reordering.""" -const F_BAD_TS_RECV = 0x08 -"""Flag bit: an unrecoverable gap was detected in the channel.""" -const F_MAYBE_BAD_BOOK = 0x04 -"""Flag bit: publisher-specific event.""" -const F_PUBLISHER_SPECIFIC = 0x02 - -""" - has_flag(flags, flag) -> Bool - -`true` if the `flag` bit (e.g. [`F_LAST`](@ref)) is set in a record's `flags`. -""" -has_flag(flags::Integer, flag::Integer) = (flags & flag) != 0 +# DBN types, enums, and data structures + +# Constants +"""The current DBN format version supported by this implementation.""" +const DBN_VERSION = 3 + +"""Fixed-length symbol string size for DBN v2/v3 metadata and records.""" +const SYMBOL_CSTR_LEN = 71 + +"""Fixed-point price scaling factor for converting between integer and float prices.""" +const FIXED_PRICE_SCALE = Int32(1_000_000_000) + +"""Sentinel value indicating an undefined or missing price.""" +const UNDEF_PRICE = typemax(Int64) + +"""Sentinel value indicating an undefined or missing order size.""" +const UNDEF_ORDER_SIZE = typemax(UInt32) + +"""Sentinel value indicating an undefined or missing timestamp.""" +const UNDEF_TIMESTAMP = typemax(Int64) + +"""Multiplier for converting the length field to bytes in record headers.""" +const LENGTH_MULTIPLIER = 4 + +# Enums using EnumX for better namespace management + +""" + Schema + +DBN data schemas representing different types of market data. + +# Values +- `MBO`: Market-by-order data +- `MBP_1`: Market-by-price with book depth 1 (TBBO) +- `MBP_10`: Market-by-price with book depth 10 +- `TBBO`: Top-of-book bid/offer +- `TRADES`: Trade messages only +- `OHLCV_1S`: OHLCV data at 1-second intervals +- `OHLCV_1M`: OHLCV data at 1-minute intervals +- `OHLCV_1H`: OHLCV data at 1-hour intervals +- `OHLCV_1D`: OHLCV data at 1-day intervals +- `DEFINITION`: Instrument definition data +- `STATISTICS`: Market statistics +- `STATUS`: Exchange status messages +- `IMBALANCE`: Order imbalance data +- `CBBO`: Consolidated best bid/offer +- `CBBO_1S`: Consolidated BBO at 1-second intervals +- `CBBO_1M`: Consolidated BBO at 1-minute intervals +- `CMBP_1`: Consolidated market-by-price depth 1 +- `TCBBO`: Trade-consolidated BBO +- `BBO_1S`: BBO at 1-second intervals +- `BBO_1M`: BBO at 1-minute intervals +""" +@enumx Schema::UInt16 begin + MBO = 0 + MBP_1 = 1 + MBP_10 = 2 + TBBO = 3 + TRADES = 4 + OHLCV_1S = 5 + OHLCV_1M = 6 + OHLCV_1H = 7 + OHLCV_1D = 8 + DEFINITION = 9 + STATISTICS = 10 + STATUS = 11 + IMBALANCE = 12 + CBBO = 13 + CBBO_1S = 14 + CBBO_1M = 15 + CMBP_1 = 16 + TCBBO = 17 + BBO_1S = 18 + BBO_1M = 19 + MIX = 0xFFFF +end + +""" + Compression + +Compression algorithms supported for DBN files. + +# Values +- `NONE`: No compression +- `ZSTD`: Zstandard compression +""" +@enumx Compression::UInt8 begin + NONE = 0 + ZSTD = 1 +end + +""" + Encoding + +Output encoding formats for market data. + +# Values +- `DBN`: Databento Binary Encoding +- `CSV`: Comma-separated values +- `JSON`: JavaScript Object Notation +""" +@enumx Encoding::UInt8 begin + DBN = 0 + CSV = 1 + JSON = 2 +end + +""" + SType + +Symbol types for identifying instruments in DBN data. Numeric values match the +official Databento DBN spec — wire-encoded as a `UInt8` in metadata and in +`SymbolMappingMsg` (v2+). + +# Values +- `INSTRUMENT_ID = 0`: Numeric instrument identifier +- `RAW_SYMBOL = 1`: Raw symbol string from exchange +- `SMART = 2`: Deprecated alias (was split into `CONTINUOUS` and `PARENT`) +- `CONTINUOUS = 3`: Continuous contract symbol +- `PARENT = 4`: Parent symbol for derived instruments (e.g. `SPXW.OPT`) +- `NASDAQ_SYMBOL = 5`: Nasdaq-specific symbol +- `CMS_SYMBOL = 6`: CMS symbol +- `ISIN = 7`: ISO 6166 International Securities Identification Number +- `US_CODE = 8`: US Code (CUSIP-style) +- `BBG_COMP_ID = 9`: Bloomberg composite ID +- `BBG_COMP_TICKER = 10`: Bloomberg composite ticker +- `FIGI = 11`: OpenFIGI identifier +- `FIGI_TICKER = 12`: OpenFIGI ticker +- `UNDEF = 255`: Unset/undefined sentinel (`0xFF`). Databento wire-encodes an + unset `stype` as `0xFF`; this member lets non-nullable `stype` fields (e.g. + in `SymbolMappingMsg`) represent and round-trip that sentinel. Metadata's + nullable `stype_in` decodes `0xFF` to `nothing` instead. +""" +@enumx SType::UInt8 begin + INSTRUMENT_ID = 0 + RAW_SYMBOL = 1 + SMART = 2 # deprecated (kept for round-trip with v1 wire data) + CONTINUOUS = 3 + PARENT = 4 + NASDAQ_SYMBOL = 5 + CMS_SYMBOL = 6 + ISIN = 7 + US_CODE = 8 + BBG_COMP_ID = 9 + BBG_COMP_TICKER = 10 + FIGI = 11 + FIGI_TICKER = 12 + UNDEF = 255 # 0xFF "unset" sentinel; see docstring +end + +""" + RType + +Record types for different kinds of market data messages in DBN format. + +# Values +- `MBP_0_MSG`: Trades (book depth 0) +- `MBP_1_MSG`: TBBO/MBP-1 (book depth 1) +- `MBP_10_MSG`: MBP-10 (book depth 10) +- `STATUS_MSG`: Exchange status record +- `INSTRUMENT_DEF_MSG`: Instrument definition record +- `IMBALANCE_MSG`: Order imbalance record +- `ERROR_MSG`: Error record from live gateway +- `SYMBOL_MAPPING_MSG`: Symbol mapping record from live gateway +- `SYSTEM_MSG`: Non-error record from live gateway +- `STAT_MSG`: Statistics record from publisher +- `OHLCV_1S_MSG`: OHLCV at 1-second cadence +- `OHLCV_1M_MSG`: OHLCV at 1-minute cadence +- `OHLCV_1H_MSG`: OHLCV at hourly cadence +- `OHLCV_1D_MSG`: OHLCV at daily cadence +- `MBO_MSG`: Market-by-order record +- `CMBP_1_MSG`: Consolidated market-by-price with book depth 1 +- `CBBO_1S_MSG`: Consolidated market-by-price with book depth 1 at 1-second cadence +- `CBBO_1M_MSG`: Consolidated market-by-price with book depth 1 at 1-minute cadence +- `TCBBO_MSG`: Consolidated market-by-price with book depth 1 (trades only) +- `BBO_1S_MSG`: Market-by-price with book depth 1 at 1-second cadence +- `BBO_1M_MSG`: Market-by-price with book depth 1 at 1-minute cadence +""" +@enumx RType::UInt8 begin + MBP_0_MSG = 0x00 # Trades (book depth 0) + MBP_1_MSG = 0x01 # TBBO/MBP-1 (book depth 1) + MBP_10_MSG = 0x0A # MBP-10 (book depth 10) + STATUS_MSG = 0x12 # Exchange status record + INSTRUMENT_DEF_MSG = 0x13 # Instrument definition record + IMBALANCE_MSG = 0x14 # Order imbalance record + ERROR_MSG = 0x15 # Error record from live gateway + SYMBOL_MAPPING_MSG = 0x16 # Symbol mapping record from live gateway + SYSTEM_MSG = 0x17 # Non-error record from live gateway + STAT_MSG = 0x18 # Statistics record from publisher + OHLCV_1S_MSG = 0x20 # OHLCV at 1-second cadence + OHLCV_1M_MSG = 0x21 # OHLCV at 1-minute cadence + OHLCV_1H_MSG = 0x22 # OHLCV at hourly cadence + OHLCV_1D_MSG = 0x23 # OHLCV at daily cadence + MBO_MSG = 0xA0 # Market-by-order record + CMBP_1_MSG = 0xB1 # Consolidated market-by-price with book depth 1 + CBBO_1S_MSG = 0xC0 # Consolidated market-by-price with book depth 1 at 1-second cadence + CBBO_1M_MSG = 0xC1 # Consolidated market-by-price with book depth 1 at 1-minute cadence + TCBBO_MSG = 0xC2 # Consolidated market-by-price with book depth 1 (trades only) + BBO_1S_MSG = 0xC3 # Market-by-price with book depth 1 at 1-second cadence + BBO_1M_MSG = 0xC4 # Market-by-price with book depth 1 at 1-minute cadence +end + +""" + Action + +Market actions that can be applied to orders or trades. + +# Values +- `ADD`: Insert a new order into the book +- `MODIFY`: Change an order's price and/or size +- `CANCEL`: Fully or partially cancel an order from the book +- `CLEAR`: Remove all resting orders for the instrument +- `TRADE`: An aggressing order traded. Does not affect the book +- `FILL`: A resting order was filled. Does not affect the book +- `NONE`: No action: does not affect the book, but may carry flags or other information +""" +@enumx Action::UInt8 begin + ADD = UInt8('A') # Insert a new order into the book + MODIFY = UInt8('M') # Change an order's price and/or size + CANCEL = UInt8('C') # Fully or partially cancel an order from the book + CLEAR = UInt8('R') # Remove all resting orders for the instrument + TRADE = UInt8('T') # An aggressing order traded. Does not affect the book + FILL = UInt8('F') # A resting order was filled. Does not affect the book + NONE = UInt8('N') # No action: does not affect the book, but may carry flags or other information +end + +""" + Side + +Market sides for orders and trades. + +# Values +- `ASK`: Ask/offer side (sell orders) +- `BID`: Bid side (buy orders) +- `NONE`: No specific side or not applicable +""" +@enumx Side::UInt8 begin + ASK = UInt8('A') + BID = UInt8('B') + NONE = UInt8('N') +end + +""" + InstrumentClass + +Classification of financial instruments. Values match Databento's DBN +`InstrumentClass` enum. + +# Values +- `STOCK`: Equity instruments +- `CALL`: Call option contracts +- `PUT`: Put option contracts +- `FUTURE`: Futures contracts +- `INDEX`: Index instruments +- `BOND`: Fixed income securities +- `MIXED_SPREAD`: Mixed spread instruments +- `FUTURE_SPREAD`: Futures spread instruments +- `OPTION_SPREAD`: Option spread instruments +- `FX_SPOT`: Foreign exchange spot +- `COMMODITY_SPOT`: Commodity spot +- `UNKNOWN_0`, `UNKNOWN_45`: Numeric fallback values for unknown classes +""" +@enumx InstrumentClass::UInt8 begin + BOND = UInt8('B') + CALL = UInt8('C') + FUTURE = UInt8('F') + INDEX = UInt8('I') + STOCK = UInt8('K') + MIXED_SPREAD = UInt8('M') + PUT = UInt8('P') + FUTURE_SPREAD = UInt8('S') + OPTION_SPREAD = UInt8('T') + FX_SPOT = UInt8('X') + COMMODITY_SPOT = UInt8('Y') + # Also support numeric values + UNKNOWN_0 = 0 + UNKNOWN_45 = 45 + OTHER = UInt8('?') +end + +# Basic structures + +""" + VersionUpgradePolicy + +Encapsulates the version upgrade policy for DBN files. + +# Fields +- `upgrade_policy::UInt8`: Policy for handling version upgrades +""" +struct VersionUpgradePolicy + upgrade_policy::UInt8 +end + +""" + DatasetCondition + +Conditions and constraints for a dataset. + +# Fields +- `last_ts_out::Int64`: Last timestamp output +- `start_ts::Int64`: Dataset start timestamp +- `end_ts::Int64`: Dataset end timestamp +- `limit::UInt64`: Record limit for the dataset +""" +struct DatasetCondition + last_ts_out::Int64 + start_ts::Int64 + end_ts::Int64 + limit::UInt64 +end + +""" + Metadata + +Metadata information for a DBN dataset. + +# Fields +- `version::UInt8`: DBN format version +- `dataset::String`: Dataset identifier +- `schema::Schema.T`: Data schema type +- `start_ts::Int64`: Start timestamp for the data +- `end_ts::Union{Int64,Nothing}`: End timestamp (can be null) +- `limit::Union{UInt64,Nothing}`: Record count limit (can be null) +- `stype_in::Union{SType.T,Nothing}`: Input symbol type (can be null) +- `stype_out::SType.T`: Output symbol type +- `ts_out::Bool`: Whether timestamps are included in output +- `symbols::Vector{String}`: List of symbols in the dataset +- `partial::Vector{String}`: Partially available symbols +- `not_found::Vector{String}`: Symbols that were not found +- `mappings::Vector{Tuple{String,String,Int64,Int64}}`: Symbol mappings as + `(raw_symbol, mapped_symbol, start_date, end_date)`, one tuple per mapping + interval. A symbol with multiple intervals (e.g. a continuous contract's roll + history) contributes consecutive tuples sharing the same raw symbol. Dates + are raw `YYYYMMDD` integers. +""" +struct Metadata + version::UInt8 + dataset::String + schema::Schema.T + start_ts::Int64 + end_ts::Union{Int64,Nothing} # Can be null + limit::Union{UInt64,Nothing} # Can be null + stype_in::Union{SType.T,Nothing} # Can be null + stype_out::SType.T + ts_out::Bool + symbols::Vector{String} + partial::Vector{String} + not_found::Vector{String} + mappings::Vector{Tuple{String,String,Int64,Int64}} +end + +""" + DBNHeader + +Complete header information for a DBN file. + +# Fields +- `version_upgrade_policy::VersionUpgradePolicy`: Version handling policy +- `dataset_condition::DatasetCondition`: Dataset conditions and constraints +- `metadata::Metadata`: Dataset metadata +""" +struct DBNHeader + version_upgrade_policy::VersionUpgradePolicy + dataset_condition::DatasetCondition + metadata::Metadata +end + +""" + RecordHeader + +Standard header present in all DBN record types. + +# Fields +- `length::UInt8`: Length of the record in 4-byte units (multiply by LENGTH_MULTIPLIER for bytes) +- `rtype::RType.T`: Record type identifier +- `publisher_id::UInt16`: Publisher/venue identifier +- `instrument_id::UInt32`: Instrument identifier +- `ts_event::Int64`: Event timestamp in nanoseconds since Unix epoch +""" +struct RecordHeader + length::UInt8 + rtype::RType.T + publisher_id::UInt16 + instrument_id::UInt32 + ts_event::Int64 +end + +""" + record_length_bytes(hd::RecordHeader) + +Get the actual record length in bytes from a RecordHeader. +The length field stores 4-byte units, so multiply by LENGTH_MULTIPLIER. +""" +record_length_bytes(hd::RecordHeader) = hd.length * LENGTH_MULTIPLIER + +""" + BidAskPair + +Bid and ask price/size information for market data. + +# Fields +- `bid_px::Int64`: Bid price (scaled by FIXED_PRICE_SCALE) +- `ask_px::Int64`: Ask price (scaled by FIXED_PRICE_SCALE) +- `bid_sz::UInt32`: Bid size/quantity +- `ask_sz::UInt32`: Ask size/quantity +- `bid_ct::UInt32`: Number of bid orders +- `ask_ct::UInt32`: Number of ask orders +""" +struct BidAskPair + bid_px::Int64 + ask_px::Int64 + bid_sz::UInt32 + ask_sz::UInt32 + bid_ct::UInt32 + ask_ct::UInt32 +end + +# Timestamp utilities + +""" + DBNTimestamp + +High-precision timestamp representation with nanosecond accuracy. + +# Fields +- `seconds::Int64`: Unix epoch seconds +- `nanoseconds::Int32`: Nanoseconds within the second (0-999_999_999) +""" +struct DBNTimestamp + seconds::Int64 # Unix epoch seconds + nanoseconds::Int32 # Nanoseconds within the second (0-999_999_999) +end + +""" + DBNTimestamp(ns::Int64) + +Construct a DBNTimestamp from nanoseconds since Unix epoch. + +# Arguments +- `ns::Int64`: Nanoseconds since Unix epoch + +# Returns +- `DBNTimestamp`: Timestamp split into seconds and nanoseconds components +""" +function DBNTimestamp(ns::Int64) + if ns == UNDEF_TIMESTAMP + return DBNTimestamp(UNDEF_TIMESTAMP, 0) + end + seconds = ns ÷ 1_000_000_000 + nanoseconds = Int32(ns % 1_000_000_000) + return DBNTimestamp(seconds, nanoseconds) +end + +""" + to_nanoseconds(ts::DBNTimestamp) + +Convert a DBNTimestamp back to nanoseconds since Unix epoch. + +# Arguments +- `ts::DBNTimestamp`: Timestamp to convert + +# Returns +- `Int64`: Nanoseconds since Unix epoch, or UNDEF_TIMESTAMP if undefined +""" +function to_nanoseconds(ts::DBNTimestamp) + if ts.seconds == UNDEF_TIMESTAMP + return UNDEF_TIMESTAMP + end + return ts.seconds * 1_000_000_000 + ts.nanoseconds +end + +""" + ts_to_datetime(ts::Int64) + +Convert a nanosecond timestamp to DateTime with nanosecond precision information. + +# Arguments +- `ts::Int64`: Nanoseconds since Unix epoch + +# Returns +- `NamedTuple`: Contains `datetime` (DateTime) and `nanoseconds` (Int32), or `nothing` if undefined +""" +function ts_to_datetime(ts::Int64) + if ts == UNDEF_TIMESTAMP + return nothing + end + # Returns DateTime with millisecond precision and separate nanosecond component + dbn_ts = DBNTimestamp(ts) + dt = unix2datetime(Float64(dbn_ts.seconds) + dbn_ts.nanoseconds / 1_000_000_000) + return (datetime=dt, nanoseconds=dbn_ts.nanoseconds) +end + +""" + datetime_to_ts(dt::DateTime, nanoseconds::Int32=0) + +Convert a DateTime with optional nanosecond precision to nanosecond timestamp. + +# Arguments +- `dt::DateTime`: DateTime to convert +- `nanoseconds::Int32`: Additional nanoseconds within the second (default: 0) + +# Returns +- `Int64`: Nanoseconds since Unix epoch +""" +function datetime_to_ts(dt::DateTime, nanoseconds::Union{Int32,Int64}=0) + # Convert DateTime to nanoseconds, preserving additional precision + seconds = Int64(round(datetime2unix(dt))) + return seconds * 1_000_000_000 + nanoseconds +end + +""" + ts_to_date_time(ts::Int64) + +Convert a nanosecond timestamp to separate Date and Time components with full nanosecond precision. + +# Arguments +- `ts::Int64`: Nanoseconds since Unix epoch + +# Returns +- `NamedTuple`: Contains `date` (Date), `time` (Time), and `timestamp` (DBNTimestamp), or `nothing` if undefined +""" +function ts_to_date_time(ts::Int64) + if ts == UNDEF_TIMESTAMP + return nothing + end + dbn_ts = DBNTimestamp(ts) + + # Get the date part + dt_seconds = unix2datetime(Float64(dbn_ts.seconds)) + date_part = Date(dt_seconds) + + # Get time within the day with nanosecond precision + seconds_in_day = dbn_ts.seconds % 86400 + time_ns = seconds_in_day * 1_000_000_000 + dbn_ts.nanoseconds + time_part = Dates.Time(Dates.Nanosecond(time_ns)) + + return (date=date_part, time=time_part, timestamp=dbn_ts) +end + +""" + date_time_to_ts(date::Date, time::Dates.Time) + +Convert separate Date and Time components to nanosecond timestamp. + +# Arguments +- `date::Date`: Date component +- `time::Dates.Time`: Time component with nanosecond precision + +# Returns +- `Int64`: Nanoseconds since Unix epoch +""" +function date_time_to_ts(date::Date, time::Dates.Time) + # Convert date to seconds since epoch + dt = DateTime(date) + date_seconds = Int64(round(datetime2unix(dt))) + + # Extract nanoseconds from time + time_ns = Dates.value(time) # Total nanoseconds since midnight + + return date_seconds * 1_000_000_000 + time_ns +end + +# Price conversion utilities + +""" + price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) + +Convert a fixed-point price to floating-point representation. + +# Arguments +- `price::Int64`: Fixed-point price value +- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) + +# Returns +- `Float64`: Floating-point price, or NaN if price is UNDEF_PRICE +""" +function price_to_float(price::Int64, scale::Int32=FIXED_PRICE_SCALE) + if price == UNDEF_PRICE + return NaN + end + return Float64(price) / Float64(scale) +end + +""" + float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) + +Convert a floating-point price to fixed-point representation. + +# Arguments +- `value::Float64`: Floating-point price +- `scale::Int32`: Scaling factor (default: FIXED_PRICE_SCALE) + +# Returns +- `Int64`: Fixed-point price, or UNDEF_PRICE if value is NaN or infinite +""" +function float_to_price(value::Float64, scale::Int32=FIXED_PRICE_SCALE) + if isnan(value) || isinf(value) + return UNDEF_PRICE + end + return Int64(round(value * Float64(scale))) +end + +# Helper functions for safe enum conversion. +# All enums share one policy: an invalid byte warns (once per enum) and maps to +# a sentinel value instead of throwing, so one bad byte can't kill a decode +# stream. Lookup tables keep the hot path free of try/catch. + +function _build_enum_lookup(::Type{E}, default::E) where {E} + table = fill(default, 256) + valid = falses(256) + for inst in instances(E) + idx = Int(UInt8(inst)) + 1 + table[idx] = inst + valid[idx] = true + end + return table, valid +end + +const _ACTION_LOOKUP, _ACTION_VALID = _build_enum_lookup(Action.T, Action.NONE) +const _SIDE_LOOKUP, _SIDE_VALID = _build_enum_lookup(Side.T, Side.NONE) +const _INSTRUMENT_CLASS_LOOKUP, _INSTRUMENT_CLASS_VALID = + _build_enum_lookup(InstrumentClass.T, InstrumentClass.OTHER) + +""" + safe_action(raw_val::UInt8) + +Convert a raw byte value to an Action enum. `0x00` and invalid values map to +`Action.NONE`; invalid values additionally log a warning (once). +""" +@inline function safe_action(raw_val::UInt8) + # 0 indicates no action for certain record types + raw_val == 0x00 && return Action.NONE + idx = Int(raw_val) + 1 + @inbounds if !_ACTION_VALID[idx] + @warn "Unknown Action value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 + end + return @inbounds _ACTION_LOOKUP[idx] +end + +""" + safe_side(raw_val::UInt8) + +Convert a raw byte value to a Side enum. `0x00` and invalid values map to +`Side.NONE`; invalid values additionally log a warning (once). +""" +@inline function safe_side(raw_val::UInt8) + # 0 indicates no side for certain record types + raw_val == 0x00 && return Side.NONE + idx = Int(raw_val) + 1 + @inbounds if !_SIDE_VALID[idx] + @warn "Unknown Side value: $raw_val (0x$(string(raw_val, base=16))), using NONE as default" maxlog = 1 + end + return @inbounds _SIDE_LOOKUP[idx] +end + +""" + safe_instrument_class(raw_val::UInt8) + +Convert a raw byte value to an InstrumentClass enum. Invalid values log a +warning (once) and map to `InstrumentClass.OTHER`. +""" +@inline function safe_instrument_class(raw_val::UInt8) + idx = Int(raw_val) + 1 + @inbounds if !_INSTRUMENT_CLASS_VALID[idx] + @warn "Unknown InstrumentClass value: $raw_val (0x$(string(raw_val, base=16))), using OTHER as default" maxlog = 1 + end + return @inbounds _INSTRUMENT_CLASS_LOOKUP[idx] +end + +# --------------------------------------------------------------------------- +# Venue-code enums and record flags +# +# `StatMsg.stat_type` and `StatusMsg.trading_event` stay raw `UInt16` on the +# record structs: the wire fields are plain integers and publishers may emit +# codes outside the published enum. The enums below are for *interpreting* +# those codes. Compare with `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)` +# or convert with `safe_stat_type(r.stat_type)` / `safe_trading_event(...)`. +# --------------------------------------------------------------------------- + +""" + StatType + +Type of statistic carried by a [`StatMsg`](@ref) (`stat_type` field). Values +match the official DBN `StatType` enum (`UInt16`). `UNKNOWN = 0` is a local +sentinel returned by [`safe_stat_type`](@ref) for codes this package does not +recognize (publisher-specific, or newer than this release). + +# Values +- `OPENING_PRICE = 1`: Price of the first trade of an instrument +- `INDICATIVE_OPENING_PRICE = 2`: Probable opening price, published pre-open +- `SETTLEMENT_PRICE = 3`: Settlement price +- `TRADING_SESSION_LOW_PRICE = 4`: Lowest trade price of the session +- `TRADING_SESSION_HIGH_PRICE = 5`: Highest trade price of the session +- `CLEARED_VOLUME = 6`: Contracts cleared on the previous trading date +- `LOWEST_OFFER = 7`: Lowest offer price of the session +- `HIGHEST_BID = 8`: Highest bid price of the session +- `OPEN_INTEREST = 9`: Number of outstanding contracts +- `FIXING_PRICE = 10`: VWAP over a fixing period +- `CLOSE_PRICE = 11`: Last trade price of the session +- `NET_CHANGE = 12`: Change from the previous session's close +- `VWAP = 13`: Session volume-weighted average price +- `VOLATILITY = 14`: Implied volatility associated with the settlement price +- `DELTA = 15`: Option delta associated with the settlement price +- `UNCROSSING_PRICE = 16`: Auction uncrossing price +- `UPPER_PRICE_LIMIT = 17`: Exchange-defined upper price limit (published for + CME GLBX.MDP3 since the 2026-07 normalization change) +- `LOWER_PRICE_LIMIT = 18`: Exchange-defined lower price limit (as above) +- `BLOCK_VOLUME = 19`: Block contracts cleared on the previous trading date +- `INDICATIVE_CLOSE_PRICE = 20`: Probable closing price +- `MWCB_LEVEL_1 = 21`, `MWCB_LEVEL_2 = 22`, `MWCB_LEVEL_3 = 23`: Market-wide + circuit-breaker thresholds (7% / 13% / 20%) +- `AUCTION_COLLAR_REFERENCE_PRICE = 24`, `AUCTION_COLLAR_UPPER_PRICE = 25`, + `AUCTION_COLLAR_LOWER_PRICE = 26`: Auction collar prices +- `VENUE_SPECIFIC_VOLUME_1 = 10001`, `VENUE_SPECIFIC_PRICE_1 = 10002`: + Venue-specific statistics +- `UNKNOWN = 0`: Unrecognized code (local sentinel, not part of the DBN spec) +""" +@enumx StatType::UInt16 begin + UNKNOWN = 0 + OPENING_PRICE = 1 + INDICATIVE_OPENING_PRICE = 2 + SETTLEMENT_PRICE = 3 + TRADING_SESSION_LOW_PRICE = 4 + TRADING_SESSION_HIGH_PRICE = 5 + CLEARED_VOLUME = 6 + LOWEST_OFFER = 7 + HIGHEST_BID = 8 + OPEN_INTEREST = 9 + FIXING_PRICE = 10 + CLOSE_PRICE = 11 + NET_CHANGE = 12 + VWAP = 13 + VOLATILITY = 14 + DELTA = 15 + UNCROSSING_PRICE = 16 + UPPER_PRICE_LIMIT = 17 + LOWER_PRICE_LIMIT = 18 + BLOCK_VOLUME = 19 + INDICATIVE_CLOSE_PRICE = 20 + MWCB_LEVEL_1 = 21 + MWCB_LEVEL_2 = 22 + MWCB_LEVEL_3 = 23 + AUCTION_COLLAR_REFERENCE_PRICE = 24 + AUCTION_COLLAR_UPPER_PRICE = 25 + AUCTION_COLLAR_LOWER_PRICE = 26 + VENUE_SPECIFIC_VOLUME_1 = 10001 + VENUE_SPECIFIC_PRICE_1 = 10002 +end + +""" + TradingEvent + +Additional context for a [`StatusMsg`](@ref) (`trading_event` field). Values +match the official DBN `TradingEvent` enum (`UInt16`). + +# Values +- `NONE = 0`: No additional information given +- `NO_CANCEL = 1`: Order entry is allowed; modification and cancellation are not +- `CHANGE_TRADING_SESSION = 2`: A change of trading session occurred; daily + statistics are reset +- `IMPLIED_MATCHING_ON = 3`: Implied matching is available (CME's matching + engine is constructing implied depth) +- `IMPLIED_MATCHING_OFF = 4`: Implied matching is not available + +CME GLBX.MDP3 publishes `IMPLIED_MATCHING_ON`/`OFF` status records since the +2026-07 normalization change. +""" +@enumx TradingEvent::UInt16 begin + NONE = 0 + NO_CANCEL = 1 + CHANGE_TRADING_SESSION = 2 + IMPLIED_MATCHING_ON = 3 + IMPLIED_MATCHING_OFF = 4 +end + +const _STAT_TYPE_LOOKUP = Dict{UInt16,StatType.T}(UInt16(v) => v for v in instances(StatType.T)) +const _TRADING_EVENT_LOOKUP = Dict{UInt16,TradingEvent.T}(UInt16(v) => v for v in instances(TradingEvent.T)) + +""" + safe_stat_type(raw::Integer) -> StatType.T + +Interpret a raw `StatMsg.stat_type` code. Unrecognized codes map to +`StatType.UNKNOWN` without a warning (publisher-specific codes are legitimate). +""" +safe_stat_type(raw::Integer) = get(_STAT_TYPE_LOOKUP, UInt16(raw), StatType.UNKNOWN) + +""" + safe_trading_event(raw::Integer) -> TradingEvent.T + +Interpret a raw `StatusMsg.trading_event` code. Unrecognized codes map to +`TradingEvent.NONE` ("no additional information") without a warning. +""" +safe_trading_event(raw::Integer) = get(_TRADING_EVENT_LOOKUP, UInt16(raw), TradingEvent.NONE) + +# Record `flags` bit field (MBO / MBP / trade / BBO records). Values match the +# official DBN `flags` module. + +""" +Flag bit: last record in the event for a given `instrument_id`. Since the +2026-07 CME normalization change, GLBX.MDP3 MBO emits this on a standalone +record (`action = Action.NONE`, `price = UNDEF_PRICE`, `size = 0`) that +follows the book updates, rather than on the final book update itself. +""" +const F_LAST = 0x80 +"""Flag bit: top-of-book record, not an individual order.""" +const F_TOB = 0x40 +"""Flag bit: record sourced from a replay, such as a snapshot server.""" +const F_SNAPSHOT = 0x20 +"""Flag bit: aggregated price-level record, not an individual order.""" +const F_MBP = 0x10 +"""Flag bit: `ts_recv` is inaccurate due to clock issues or packet reordering.""" +const F_BAD_TS_RECV = 0x08 +"""Flag bit: an unrecoverable gap was detected in the channel.""" +const F_MAYBE_BAD_BOOK = 0x04 +"""Flag bit: publisher-specific event.""" +const F_PUBLISHER_SPECIFIC = 0x02 + +""" + has_flag(flags, flag) -> Bool + +`true` if the `flag` bit (e.g. [`F_LAST`](@ref)) is set in a record's `flags`. +""" +has_flag(flags::Integer, flag::Integer) = (flags & flag) != 0 diff --git a/test/runtests.jl b/test/runtests.jl index 38684981..9c4d019e 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -1,45 +1,45 @@ -using Test -using DatabentoBinaryEncoding -import DatabentoBinaryEncoding as DBN -using Dates - -# Load test utilities (safe_rm, etc.) -include("test_utils.jl") - -@testset "DBN.jl Tests" begin - include("test_phase1.jl") - include("test_phase2.jl") - include("test_phase3.jl") - include("test_phase4.jl") - include("test_phase5.jl") - include("test_phase6.jl") - include("test_phase7.jl") - include("test_phase8.jl") - include("test_phase9_working.jl") # Edge cases and error handling - include("test_phase10_complete.jl") # Integration and performance testing - include("test_convenience_functions.jl") # Test all convenience read_*/foreach_* functions - include("test_phase11_typed_with_control.jl") # foreach_record_with_control: typed data + Union control split - include("test_replay.jl") # replay_dbn / replay_records: timestamp-paced re-emission - include("test_issue23_unset_stype.jl") # regression: 0xFF unset stype in v3 SymbolMappingMsg (issue #23) - include("test_issues_32_35.jl") # regressions: pre-v3 StatMsg layout, stat_to_dataframe, mapping intervals, invalid enum bytes (issues #32-#35) - include("test_show.jl") # compact one-line Base.show for record types - include("test_symbols.jl") # symbol_map / symbol_for / add_symbol_column! / records_to_dataframe(records, metadata) - include("test_issue40_mbp_dataframe.jl") # regression: MBP-1/MBP-10/BBO records_to_dataframe via nested levels (issue #40) - include("test_cme_2026_07.jl") # CME 2026-07 normalization: per-leg defs, standalone F_LAST, StatType/TradingEvent enums, flags - - # Run compatibility tests if the Rust CLI is available - dbn_cli_path = if Sys.iswindows() - joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn.exe") - else - joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn") - end - - if isfile(dbn_cli_path) - include("test_compatibility_updated.jl") # Updated cross-implementation compatibility testing - else - @warn "Skipping compatibility tests - Rust dbn-cli not found at $dbn_cli_path" - end - - # Import/export tests (optional - uncomment if needed) - # include("test_import_simple.jl") -end +using Test +using DatabentoBinaryEncoding +import DatabentoBinaryEncoding as DBN +using Dates + +# Load test utilities (safe_rm, etc.) +include("test_utils.jl") + +@testset "DBN.jl Tests" begin + include("test_phase1.jl") + include("test_phase2.jl") + include("test_phase3.jl") + include("test_phase4.jl") + include("test_phase5.jl") + include("test_phase6.jl") + include("test_phase7.jl") + include("test_phase8.jl") + include("test_phase9_working.jl") # Edge cases and error handling + include("test_phase10_complete.jl") # Integration and performance testing + include("test_convenience_functions.jl") # Test all convenience read_*/foreach_* functions + include("test_phase11_typed_with_control.jl") # foreach_record_with_control: typed data + Union control split + include("test_replay.jl") # replay_dbn / replay_records: timestamp-paced re-emission + include("test_issue23_unset_stype.jl") # regression: 0xFF unset stype in v3 SymbolMappingMsg (issue #23) + include("test_issues_32_35.jl") # regressions: pre-v3 StatMsg layout, stat_to_dataframe, mapping intervals, invalid enum bytes (issues #32-#35) + include("test_show.jl") # compact one-line Base.show for record types + include("test_symbols.jl") # symbol_map / symbol_for / add_symbol_column! / records_to_dataframe(records, metadata) + include("test_issue40_mbp_dataframe.jl") # regression: MBP-1/MBP-10/BBO records_to_dataframe via nested levels (issue #40) + include("test_cme_2026_07.jl") # CME 2026-07 normalization: per-leg defs, standalone F_LAST, StatType/TradingEvent enums, flags + + # Run compatibility tests if the Rust CLI is available + dbn_cli_path = if Sys.iswindows() + joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn.exe") + else + joinpath(homedir(), "dbn-workspace", "dbn", "target", "release", "dbn") + end + + if isfile(dbn_cli_path) + include("test_compatibility_updated.jl") # Updated cross-implementation compatibility testing + else + @warn "Skipping compatibility tests - Rust dbn-cli not found at $dbn_cli_path" + end + + # Import/export tests (optional - uncomment if needed) + # include("test_import_simple.jl") +end From 40f2c6058155a2bb504a8dd495ed96e8a29d0bf7 Mon Sep 17 00:00:00 2001 From: Tyler Beason Date: Thu, 17 Sep 2026 09:41:22 -0500 Subject: [PATCH 3/3] Fix mislabeled CHANGELOG history for v0.1.4 / v0.1.5 v0.1.4 (2026-06-22) and v0.1.5 (2026-06-24) were both tagged and registered while their entries still sat under `## [Unreleased]`, so neither ever got a version heading. Commit 7c10ea9 then retitled that whole accumulated block `## [0.1.6] - 2026-06-24` and bumped to 0.1.6 -- but that 0.1.6 was never registered. When #49 later renumbered the September release from 0.1.7 back down to 0.1.6, it renamed only the new section, leaving two `## [0.1.6]` headings and no 0.1.4/0.1.5 at all. Split the June block back along the real tag boundaries, determined by diffing CHANGELOG.md as it stood at each tag: - `[0.1.5] - 2026-06-24` -- DuckDB Parquet migration, ZSTD default, and the OHLCV/STATUS/IMBALANCE/DEFINITION, empty-record-set and create_metadata_from_dataframe fixes. - `[0.1.4] - 2026-06-22` -- symbol resolution helpers, plus the MBP-1/MBP-10 FieldError, read_stat_msg sizing, stat_to_dataframe, symbol-mapping-interval and invalid-enum-byte fixes (#32-#35, #40). The replay feature moves up into `[0.1.6] - 2026-09-10`: it was written in June but first reached the registry in that release. Also adds the missing [0.1.7]/[0.1.5]/[0.1.4] compare links. No entry text was changed -- only the headings it sits under. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 58 ++++++++++++++++++++++++++++++------------------ test/runtests.jl | 2 +- 2 files changed, 37 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3ea7435d..2f12fbd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Replay pacing now uses a monotonic clock by default, so NTP or manual wall-clock adjustments cannot change replay timing. +### Added + +- Timestamp-paced **replay**: `replay_dbn(f, filename; ...)` re-emits a DBN file's + records in time order, pacing each `f(record)` callback by the gap between record + timestamps to simulate a live feed (backtesting, demos, driving real-time + consumers). `replay_records(f, records; ...)` does the same over an already-loaded + collection. Zstd-aware and skips unknown record types like `DBNStream`. Pacing is + anchored to absolute wall-clock targets so callback execution time is absorbed + instead of accumulating as drift. Options: `speed` (time-compression multiplier; + `Inf` = no waiting), `timestamp` (`:ts_event` or `:ts_recv`, with fallback), + `max_sleep` (cap large gaps; re-anchors after a clamp), and `precise` (busy-wait + sub-millisecond gaps, since `Base.sleep` only resolves ~1 ms on Unix / ~15 ms on + Windows). `clock`/`sleep_fn` are injectable for deterministic testing. + ### Fixed - Parquet compression names are validated before being interpolated into the @@ -76,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 the bug). Files written by earlier versions with undefined statistics quantities contain `-1` on disk. -## [0.1.6] - 2026-06-24 +## [0.1.5] - 2026-06-24 ### Changed @@ -89,29 +103,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Timestamp-paced **replay**: `replay_dbn(f, filename; ...)` re-emits a DBN file's - records in time order, pacing each `f(record)` callback by the gap between record - timestamps to simulate a live feed (backtesting, demos, driving real-time - consumers). `replay_records(f, records; ...)` does the same over an already-loaded - collection. Zstd-aware and skips unknown record types like `DBNStream`. Pacing is - anchored to absolute wall-clock targets so callback execution time is absorbed - instead of accumulating as drift. Options: `speed` (time-compression multiplier; - `Inf` = no waiting), `timestamp` (`:ts_event` or `:ts_recv`, with fallback), - `max_sleep` (cap large gaps; re-anchors after a clamp), and `precise` (busy-wait - sub-millisecond gaps, since `Base.sleep` only resolves ~1 ms on Unix / ~15 ms on - Windows). `clock`/`sleep_fn` are injectable for deterministic testing. - `dbn_to_parquet` now compresses with **ZSTD by default** and accepts a `compression` keyword — one of `"zstd"` (default), `"snappy"`, `"gzip"`, or `"uncompressed"`. -- Symbol resolution helpers that join the human-readable `raw_symbol` back onto - records using `Metadata.mappings`: `symbol_map(metadata)` builds an - `instrument_id -> [(start_date, end_date, raw_symbol)]` lookup, - `symbol_for(map_or_metadata, instrument_id, ts_event)` resolves the symbol - valid at a record's timestamp, `add_symbol_column!(df, metadata)` joins a - `:symbol` column onto a records DataFrame in place, and - `records_to_dataframe(records, metadata; symbols=true)` does the conversion + - join in one call. The join keys on `instrument_id`/`ts_event` (so it also works - for the row-expanded MBP-10 frame) and only applies when the query was resolved - with `stype_out = SType.INSTRUMENT_ID`. ### Fixed @@ -134,6 +127,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `create_metadata_from_dataframe` (used by `parquet_to_dbn` / `csv_to_dbn`) no longer throws from `minimum`/`maximum` on a zero-row DataFrame; an empty frame yields `start_ts = end_ts = 0`, so a fully-empty Parquet/CSV round-trips back to DBN. + +## [0.1.4] - 2026-06-22 + +### Added + +- Symbol resolution helpers that join the human-readable `raw_symbol` back onto + records using `Metadata.mappings`: `symbol_map(metadata)` builds an + `instrument_id -> [(start_date, end_date, raw_symbol)]` lookup, + `symbol_for(map_or_metadata, instrument_id, ts_event)` resolves the symbol + valid at a record's timestamp, `add_symbol_column!(df, metadata)` joins a + `:symbol` column onto a records DataFrame in place, and + `records_to_dataframe(records, metadata; symbols=true)` does the conversion + + join in one call. The join keys on `instrument_id`/`ts_event` (so it also works + for the row-expanded MBP-10 frame) and only applies when the query was resolved + with `stype_out = SType.INSTRUMENT_ID`. + +### Fixed + - `records_to_dataframe` (and therefore `to_dataframe` / `dbn_to_parquet`) no longer throws `FieldError` on **TBBO / MBP-1** and **MBP-10**. The converters read non-existent flat `bid_px_00…` fields; the bid/ask data actually lives in @@ -212,7 +223,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 conversion to JSON/Parquet/CSV; byte-for-byte compatibility with the official Rust implementation ([#18], [#19]). +[0.1.7]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.6...v0.1.7 [0.1.6]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.5...v0.1.6 +[0.1.5]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.4...v0.1.5 +[0.1.4]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.3...v0.1.4 [0.1.3]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.2...v0.1.3 [0.1.2]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.1...v0.1.2 [0.1.1]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.0...v0.1.1 diff --git a/test/runtests.jl b/test/runtests.jl index a13f3009..195c66af 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -44,4 +44,4 @@ include("test_utils.jl") # Import/export tests (optional - uncomment if needed) # include("test_import_simple.jl") -end +end \ No newline at end of file