diff --git a/CHANGELOG.md b/CHANGELOG.md index 92c1be78..2f12fbd8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,36 @@ All notable changes to DatabentoBinaryEncoding.jl are documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [0.1.7] - 2026-09-17 + +### Added + +- **`StatType` enum** (`UInt16`, matches the DBN spec) for interpreting + `StatMsg.stat_type`, including `UPPER_PRICE_LIMIT = 17` and + `LOWER_PRICE_LIMIT = 18`, which CME GLBX.MDP3 publishes since Databento's + 2026-07 normalization change, plus the block-volume, indicative-close, + market-wide-circuit-breaker, auction-collar and venue-specific codes. + `safe_stat_type(raw)` maps unrecognized codes to `StatType.UNKNOWN`. The + struct field stays a raw `UInt16` (no breaking change). +- **`TradingEvent` enum** (`UInt16`) for `StatusMsg.trading_event`: + `NONE`, `NO_CANCEL`, `CHANGE_TRADING_SESSION`, `IMPLIED_MATCHING_ON = 3`, + `IMPLIED_MATCHING_OFF = 4` (the latter two are emitted by CME since 2026-07). + `safe_trading_event(raw)` maps unrecognized codes to `TradingEvent.NONE`. +- **`InstrumentClass.INDEX`** (`'I'`), present in the DBN spec but previously + missing here; such definitions decoded as `InstrumentClass.OTHER` with a + warning. (`FX_SPOT = 'X'`, which CME now uses for FX spot instruments instead + of `FUTURE`, was already supported.) +- **Record flag constants** `F_LAST` (128), `F_TOB`, `F_SNAPSHOT`, `F_MBP`, + `F_BAD_TS_RECV`, `F_MAYBE_BAD_BOOK`, `F_PUBLISHER_SPECIFIC` and a `has_flag` + helper. Relevant to the 2026-07 CME change where `F_LAST` moved from the + final MBO book update onto a standalone `action = Action.NONE` record with + `price = UNDEF_PRICE`, `size = 0`; that record already decoded, exported + (`NaN` price) and printed correctly, and is now covered by tests. +- Test coverage (`test/test_cme_2026_07.jl`) for the 2026-07 CME normalization + shapes: one definition record per strategy leg, the standalone `F_LAST` MBO + record, price-limit statistics, implied-matching status events and FX-spot / + index definitions. + ## [0.1.6] - 2026-09-10 ### Changed @@ -12,6 +42,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Replay pacing now uses a monotonic clock by default, so NTP or manual wall-clock adjustments cannot change replay timing. +### Added + +- Timestamp-paced **replay**: `replay_dbn(f, filename; ...)` re-emits a DBN file's + records in time order, pacing each `f(record)` callback by the gap between record + timestamps to simulate a live feed (backtesting, demos, driving real-time + consumers). `replay_records(f, records; ...)` does the same over an already-loaded + collection. Zstd-aware and skips unknown record types like `DBNStream`. Pacing is + anchored to absolute wall-clock targets so callback execution time is absorbed + instead of accumulating as drift. Options: `speed` (time-compression multiplier; + `Inf` = no waiting), `timestamp` (`:ts_event` or `:ts_recv`, with fallback), + `max_sleep` (cap large gaps; re-anchors after a clamp), and `precise` (busy-wait + sub-millisecond gaps, since `Base.sleep` only resolves ~1 ms on Unix / ~15 ms on + Windows). `clock`/`sleep_fn` are injectable for deterministic testing. + ### Fixed - Parquet compression names are validated before being interpolated into the @@ -46,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 the bug). Files written by earlier versions with undefined statistics quantities contain `-1` on disk. -## [0.1.6] - 2026-06-24 +## [0.1.5] - 2026-06-24 ### Changed @@ -59,29 +103,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Timestamp-paced **replay**: `replay_dbn(f, filename; ...)` re-emits a DBN file's - records in time order, pacing each `f(record)` callback by the gap between record - timestamps to simulate a live feed (backtesting, demos, driving real-time - consumers). `replay_records(f, records; ...)` does the same over an already-loaded - collection. Zstd-aware and skips unknown record types like `DBNStream`. Pacing is - anchored to absolute wall-clock targets so callback execution time is absorbed - instead of accumulating as drift. Options: `speed` (time-compression multiplier; - `Inf` = no waiting), `timestamp` (`:ts_event` or `:ts_recv`, with fallback), - `max_sleep` (cap large gaps; re-anchors after a clamp), and `precise` (busy-wait - sub-millisecond gaps, since `Base.sleep` only resolves ~1 ms on Unix / ~15 ms on - Windows). `clock`/`sleep_fn` are injectable for deterministic testing. - `dbn_to_parquet` now compresses with **ZSTD by default** and accepts a `compression` keyword — one of `"zstd"` (default), `"snappy"`, `"gzip"`, or `"uncompressed"`. -- Symbol resolution helpers that join the human-readable `raw_symbol` back onto - records using `Metadata.mappings`: `symbol_map(metadata)` builds an - `instrument_id -> [(start_date, end_date, raw_symbol)]` lookup, - `symbol_for(map_or_metadata, instrument_id, ts_event)` resolves the symbol - valid at a record's timestamp, `add_symbol_column!(df, metadata)` joins a - `:symbol` column onto a records DataFrame in place, and - `records_to_dataframe(records, metadata; symbols=true)` does the conversion + - join in one call. The join keys on `instrument_id`/`ts_event` (so it also works - for the row-expanded MBP-10 frame) and only applies when the query was resolved - with `stype_out = SType.INSTRUMENT_ID`. ### Fixed @@ -104,6 +127,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `create_metadata_from_dataframe` (used by `parquet_to_dbn` / `csv_to_dbn`) no longer throws from `minimum`/`maximum` on a zero-row DataFrame; an empty frame yields `start_ts = end_ts = 0`, so a fully-empty Parquet/CSV round-trips back to DBN. + +## [0.1.4] - 2026-06-22 + +### Added + +- Symbol resolution helpers that join the human-readable `raw_symbol` back onto + records using `Metadata.mappings`: `symbol_map(metadata)` builds an + `instrument_id -> [(start_date, end_date, raw_symbol)]` lookup, + `symbol_for(map_or_metadata, instrument_id, ts_event)` resolves the symbol + valid at a record's timestamp, `add_symbol_column!(df, metadata)` joins a + `:symbol` column onto a records DataFrame in place, and + `records_to_dataframe(records, metadata; symbols=true)` does the conversion + + join in one call. The join keys on `instrument_id`/`ts_event` (so it also works + for the row-expanded MBP-10 frame) and only applies when the query was resolved + with `stype_out = SType.INSTRUMENT_ID`. + +### Fixed + - `records_to_dataframe` (and therefore `to_dataframe` / `dbn_to_parquet`) no longer throws `FieldError` on **TBBO / MBP-1** and **MBP-10**. The converters read non-existent flat `bid_px_00…` fields; the bid/ask data actually lives in @@ -182,7 +223,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 conversion to JSON/Parquet/CSV; byte-for-byte compatibility with the official Rust implementation ([#18], [#19]). +[0.1.7]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.6...v0.1.7 [0.1.6]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.5...v0.1.6 +[0.1.5]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.4...v0.1.5 +[0.1.4]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.3...v0.1.4 [0.1.3]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.2...v0.1.3 [0.1.2]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.1...v0.1.2 [0.1.1]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.0...v0.1.1 diff --git a/Project.toml b/Project.toml index 3d60e3c6..7d2aa985 100644 --- a/Project.toml +++ b/Project.toml @@ -1,6 +1,6 @@ name = "DatabentoBinaryEncoding" uuid = "90689371-c8cb-40d1-831f-18033db90f74" -version = "0.1.6" +version = "0.1.7" authors = ["Tyler Beason "] [deps] diff --git a/docs/src/api/enums.md b/docs/src/api/enums.md index 50502c94..bdc1e96d 100644 --- a/docs/src/api/enums.md +++ b/docs/src/api/enums.md @@ -125,12 +125,64 @@ codes: - `CALL` (`'C'`) - Call option - `PUT` (`'P'`) - Put option - `FUTURE` (`'F'`) - Futures contract +- `INDEX` (`'I'`) - Index - `BOND` (`'B'`) - Fixed income - `FX_SPOT` (`'X'`) - Foreign exchange spot - `COMMODITY_SPOT` (`'Y'`) - Commodity spot - `MIXED_SPREAD` (`'M'`), `FUTURE_SPREAD` (`'S'`), `OPTION_SPREAD` (`'T'`) - Spreads - `OTHER` (`'?'`), `UNKNOWN_0`, `UNKNOWN_45` - Fallbacks for unclassified instruments +## Statistic Types + +```@docs +StatType +safe_stat_type +``` + +`StatMsg.stat_type` is stored as a raw `UInt16`; use `StatType` to interpret it, +e.g. `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)` or +`safe_stat_type(r.stat_type)`. Notable values: `OPENING_PRICE` (1), +`SETTLEMENT_PRICE` (3), `OPEN_INTEREST` (9), `CLOSE_PRICE` (11), +`UPPER_PRICE_LIMIT` (17) and `LOWER_PRICE_LIMIT` (18) — the last two are the +session price limits CME GLBX.MDP3 publishes since 2026-07. + +## Trading Events + +```@docs +TradingEvent +safe_trading_event +``` + +`StatusMsg.trading_event` is stored as a raw `UInt16`; `TradingEvent` names the +codes: `NONE` (0), `NO_CANCEL` (1), `CHANGE_TRADING_SESSION` (2), +`IMPLIED_MATCHING_ON` (3), `IMPLIED_MATCHING_OFF` (4). + +## Record Flags + +```@docs +F_LAST +F_TOB +F_SNAPSHOT +F_MBP +F_BAD_TS_RECV +F_MAYBE_BAD_BOOK +F_PUBLISHER_SPECIFIC +has_flag +``` + +The `flags` byte on MBO/MBP/trade/BBO records is a bit field. `F_LAST` (128) +marks the last record of a venue event. Since 2026-07, CME MBO carries it on a +standalone `action = Action.NONE` record with `price = UNDEF_PRICE` and +`size = 0`, so book builders should apply updates on every record and treat the +`F_LAST` record purely as an event boundary: + +```julia +foreach_mbo("glbx.mbo.dbn.zst") do r + r.action == Action.NONE || apply_to_book!(book, r) + has_flag(r.flags, F_LAST) && on_event_complete!(book) +end +``` + ## Usage Examples ### Working with Schemas diff --git a/src/DatabentoBinaryEncoding.jl b/src/DatabentoBinaryEncoding.jl index 655d9f9a..aeda549b 100644 --- a/src/DatabentoBinaryEncoding.jl +++ b/src/DatabentoBinaryEncoding.jl @@ -121,6 +121,8 @@ export foreach_cmbp1, foreach_cbbo1s, foreach_cbbo1m, foreach_tcbbo, foreach_bbo export replay_dbn, replay_records # Timestamp-paced replay export compress_dbn_file, compress_daily_files export Schema, Compression, Encoding, SType, RType, Action, Side, InstrumentClass +export StatType, TradingEvent, safe_stat_type, safe_trading_event +export F_LAST, F_TOB, F_SNAPSHOT, F_MBP, F_BAD_TS_RECV, F_MAYBE_BAD_BOOK, F_PUBLISHER_SPECIFIC, has_flag export price_to_float, float_to_price, ts_to_datetime, datetime_to_ts, ts_to_date_time, date_time_to_ts, to_nanoseconds export record_length_bytes export DBN_VERSION, FIXED_PRICE_SCALE, UNDEF_PRICE, UNDEF_ORDER_SIZE, UNDEF_TIMESTAMP @@ -130,4 +132,4 @@ export dbn_to_csv, dbn_to_json, dbn_to_parquet, records_to_dataframe export symbol_map, symbol_for, add_symbol_column! export json_to_dbn, parquet_to_dbn, csv_to_dbn -end # module DatabentoBinaryEncoding \ No newline at end of file +end # module DatabentoBinaryEncoding diff --git a/src/types.jl b/src/types.jl index d5685e63..c7507e54 100644 --- a/src/types.jl +++ b/src/types.jl @@ -251,6 +251,7 @@ Classification of financial instruments. Values match Databento's DBN - `CALL`: Call option contracts - `PUT`: Put option contracts - `FUTURE`: Futures contracts +- `INDEX`: Index instruments - `BOND`: Fixed income securities - `MIXED_SPREAD`: Mixed spread instruments - `FUTURE_SPREAD`: Futures spread instruments @@ -263,6 +264,7 @@ Classification of financial instruments. Values match Databento's DBN BOND = UInt8('B') CALL = UInt8('C') FUTURE = UInt8('F') + INDEX = UInt8('I') STOCK = UInt8('K') MIXED_SPREAD = UInt8('M') PUT = UInt8('P') @@ -665,3 +667,158 @@ warning (once) and map to `InstrumentClass.OTHER`. end return @inbounds _INSTRUMENT_CLASS_LOOKUP[idx] end + +# --------------------------------------------------------------------------- +# Venue-code enums and record flags +# +# `StatMsg.stat_type` and `StatusMsg.trading_event` stay raw `UInt16` on the +# record structs: the wire fields are plain integers and publishers may emit +# codes outside the published enum. The enums below are for *interpreting* +# those codes. Compare with `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)` +# or convert with `safe_stat_type(r.stat_type)` / `safe_trading_event(...)`. +# --------------------------------------------------------------------------- + +""" + StatType + +Type of statistic carried by a [`StatMsg`](@ref) (`stat_type` field). Values +match the official DBN `StatType` enum (`UInt16`). `UNKNOWN = 0` is a local +sentinel returned by [`safe_stat_type`](@ref) for codes this package does not +recognize (publisher-specific, or newer than this release). + +# Values +- `OPENING_PRICE = 1`: Price of the first trade of an instrument +- `INDICATIVE_OPENING_PRICE = 2`: Probable opening price, published pre-open +- `SETTLEMENT_PRICE = 3`: Settlement price +- `TRADING_SESSION_LOW_PRICE = 4`: Lowest trade price of the session +- `TRADING_SESSION_HIGH_PRICE = 5`: Highest trade price of the session +- `CLEARED_VOLUME = 6`: Contracts cleared on the previous trading date +- `LOWEST_OFFER = 7`: Lowest offer price of the session +- `HIGHEST_BID = 8`: Highest bid price of the session +- `OPEN_INTEREST = 9`: Number of outstanding contracts +- `FIXING_PRICE = 10`: VWAP over a fixing period +- `CLOSE_PRICE = 11`: Last trade price of the session +- `NET_CHANGE = 12`: Change from the previous session's close +- `VWAP = 13`: Session volume-weighted average price +- `VOLATILITY = 14`: Implied volatility associated with the settlement price +- `DELTA = 15`: Option delta associated with the settlement price +- `UNCROSSING_PRICE = 16`: Auction uncrossing price +- `UPPER_PRICE_LIMIT = 17`: Exchange-defined upper price limit (published for + CME GLBX.MDP3 since the 2026-07 normalization change) +- `LOWER_PRICE_LIMIT = 18`: Exchange-defined lower price limit (as above) +- `BLOCK_VOLUME = 19`: Block contracts cleared on the previous trading date +- `INDICATIVE_CLOSE_PRICE = 20`: Probable closing price +- `MWCB_LEVEL_1 = 21`, `MWCB_LEVEL_2 = 22`, `MWCB_LEVEL_3 = 23`: Market-wide + circuit-breaker thresholds (7% / 13% / 20%) +- `AUCTION_COLLAR_REFERENCE_PRICE = 24`, `AUCTION_COLLAR_UPPER_PRICE = 25`, + `AUCTION_COLLAR_LOWER_PRICE = 26`: Auction collar prices +- `VENUE_SPECIFIC_VOLUME_1 = 10001`, `VENUE_SPECIFIC_PRICE_1 = 10002`: + Venue-specific statistics +- `UNKNOWN = 0`: Unrecognized code (local sentinel, not part of the DBN spec) +""" +@enumx StatType::UInt16 begin + UNKNOWN = 0 + OPENING_PRICE = 1 + INDICATIVE_OPENING_PRICE = 2 + SETTLEMENT_PRICE = 3 + TRADING_SESSION_LOW_PRICE = 4 + TRADING_SESSION_HIGH_PRICE = 5 + CLEARED_VOLUME = 6 + LOWEST_OFFER = 7 + HIGHEST_BID = 8 + OPEN_INTEREST = 9 + FIXING_PRICE = 10 + CLOSE_PRICE = 11 + NET_CHANGE = 12 + VWAP = 13 + VOLATILITY = 14 + DELTA = 15 + UNCROSSING_PRICE = 16 + UPPER_PRICE_LIMIT = 17 + LOWER_PRICE_LIMIT = 18 + BLOCK_VOLUME = 19 + INDICATIVE_CLOSE_PRICE = 20 + MWCB_LEVEL_1 = 21 + MWCB_LEVEL_2 = 22 + MWCB_LEVEL_3 = 23 + AUCTION_COLLAR_REFERENCE_PRICE = 24 + AUCTION_COLLAR_UPPER_PRICE = 25 + AUCTION_COLLAR_LOWER_PRICE = 26 + VENUE_SPECIFIC_VOLUME_1 = 10001 + VENUE_SPECIFIC_PRICE_1 = 10002 +end + +""" + TradingEvent + +Additional context for a [`StatusMsg`](@ref) (`trading_event` field). Values +match the official DBN `TradingEvent` enum (`UInt16`). + +# Values +- `NONE = 0`: No additional information given +- `NO_CANCEL = 1`: Order entry is allowed; modification and cancellation are not +- `CHANGE_TRADING_SESSION = 2`: A change of trading session occurred; daily + statistics are reset +- `IMPLIED_MATCHING_ON = 3`: Implied matching is available (CME's matching + engine is constructing implied depth) +- `IMPLIED_MATCHING_OFF = 4`: Implied matching is not available + +CME GLBX.MDP3 publishes `IMPLIED_MATCHING_ON`/`OFF` status records since the +2026-07 normalization change. +""" +@enumx TradingEvent::UInt16 begin + NONE = 0 + NO_CANCEL = 1 + CHANGE_TRADING_SESSION = 2 + IMPLIED_MATCHING_ON = 3 + IMPLIED_MATCHING_OFF = 4 +end + +const _STAT_TYPE_LOOKUP = Dict{UInt16,StatType.T}(UInt16(v) => v for v in instances(StatType.T)) +const _TRADING_EVENT_LOOKUP = Dict{UInt16,TradingEvent.T}(UInt16(v) => v for v in instances(TradingEvent.T)) + +""" + safe_stat_type(raw::Integer) -> StatType.T + +Interpret a raw `StatMsg.stat_type` code. Unrecognized codes map to +`StatType.UNKNOWN` without a warning (publisher-specific codes are legitimate). +""" +safe_stat_type(raw::Integer) = get(_STAT_TYPE_LOOKUP, UInt16(raw), StatType.UNKNOWN) + +""" + safe_trading_event(raw::Integer) -> TradingEvent.T + +Interpret a raw `StatusMsg.trading_event` code. Unrecognized codes map to +`TradingEvent.NONE` ("no additional information") without a warning. +""" +safe_trading_event(raw::Integer) = get(_TRADING_EVENT_LOOKUP, UInt16(raw), TradingEvent.NONE) + +# Record `flags` bit field (MBO / MBP / trade / BBO records). Values match the +# official DBN `flags` module. + +""" +Flag bit: last record in the event for a given `instrument_id`. Since the +2026-07 CME normalization change, GLBX.MDP3 MBO emits this on a standalone +record (`action = Action.NONE`, `price = UNDEF_PRICE`, `size = 0`) that +follows the book updates, rather than on the final book update itself. +""" +const F_LAST = 0x80 +"""Flag bit: top-of-book record, not an individual order.""" +const F_TOB = 0x40 +"""Flag bit: record sourced from a replay, such as a snapshot server.""" +const F_SNAPSHOT = 0x20 +"""Flag bit: aggregated price-level record, not an individual order.""" +const F_MBP = 0x10 +"""Flag bit: `ts_recv` is inaccurate due to clock issues or packet reordering.""" +const F_BAD_TS_RECV = 0x08 +"""Flag bit: an unrecoverable gap was detected in the channel.""" +const F_MAYBE_BAD_BOOK = 0x04 +"""Flag bit: publisher-specific event.""" +const F_PUBLISHER_SPECIFIC = 0x02 + +""" + has_flag(flags, flag) -> Bool + +`true` if the `flag` bit (e.g. [`F_LAST`](@ref)) is set in a record's `flags`. +""" +has_flag(flags::Integer, flag::Integer) = (flags & flag) != 0 diff --git a/test/runtests.jl b/test/runtests.jl index e405ba8e..195c66af 100644 --- a/test/runtests.jl +++ b/test/runtests.jl @@ -25,6 +25,7 @@ include("test_utils.jl") include("test_show.jl") # compact one-line Base.show for record types include("test_symbols.jl") # symbol_map / symbol_for / add_symbol_column! / records_to_dataframe(records, metadata) include("test_issue40_mbp_dataframe.jl") # regression: MBP-1/MBP-10/BBO records_to_dataframe via nested levels (issue #40) + include("test_cme_2026_07.jl") # CME 2026-07 normalization: per-leg defs, standalone F_LAST, StatType/TradingEvent enums, flags include("test_mbo_wire_layout.jl") # regression: MBOMsg wire layout (order_id/price/ts_recv offsets) vs Databento's own fixtures include("test_stat_quantity_sentinel.jl") # regression: v3 StatMsg UNDEF quantity written as typemax(Int64), not -1 diff --git a/test/test_cme_2026_07.jl b/test/test_cme_2026_07.jl new file mode 100644 index 00000000..e00959c5 --- /dev/null +++ b/test/test_cme_2026_07.jl @@ -0,0 +1,229 @@ +# Coverage for Databento's CME (GLBX.MDP3) normalization changes of 2026-07-07 +# (https://databento.com/blog/cme-normalization-changes-2026-07) and the DBN +# enum/flag additions that go with them: +# 1. per-leg definition records: one InstrumentDefMsg per strategy leg +# 2. standalone F_LAST MBO record: action='N', flags=F_LAST, price UNDEF, size 0 +# 3. StatType UPPER_PRICE_LIMIT (17) / LOWER_PRICE_LIMIT (18) +# 4. TradingEvent IMPLIED_MATCHING_ON (3) / IMPLIED_MATCHING_OFF (4) +# 5. InstrumentClass FX_SPOT ('X') for FX spot, and INDEX ('I') from the spec +# +# Everything is round-tripped through the real encoder/decoder so the tests +# exercise the wire layout, not just the Julia constructors. + +@testset "CME 2026-07 normalization / enum coverage" begin + ts0 = Int64(1_783_411_200_000_000_000) # 2026-07-07T00:00:00Z + + mk_meta(schema) = Metadata(UInt8(3), "GLBX.MDP3", schema, ts0, + ts0 + 3_600_000_000_000, nothing, + SType.RAW_SYMBOL, SType.INSTRUMENT_ID, false, + String[], String[], String[], + Tuple{String,String,Int64,Int64}[]) + + function roundtrip(schema, records) + tmp = tempname() * ".dbn" + try + write_dbn(tmp, mk_meta(schema), records) + return read_dbn(tmp) + finally + safe_rm(tmp) + end + end + + @testset "enum and flag values match the DBN spec" begin + # StatType (UInt16) + @test UInt16(StatType.OPENING_PRICE) == 1 + @test UInt16(StatType.SETTLEMENT_PRICE) == 3 + @test UInt16(StatType.OPEN_INTEREST) == 9 + @test UInt16(StatType.UPPER_PRICE_LIMIT) == 17 + @test UInt16(StatType.LOWER_PRICE_LIMIT) == 18 + @test UInt16(StatType.AUCTION_COLLAR_LOWER_PRICE) == 26 + @test UInt16(StatType.VENUE_SPECIFIC_VOLUME_1) == 10001 + @test UInt16(StatType.VENUE_SPECIFIC_PRICE_1) == 10002 + @test safe_stat_type(17) == StatType.UPPER_PRICE_LIMIT + @test safe_stat_type(UInt16(18)) == StatType.LOWER_PRICE_LIMIT + @test safe_stat_type(0x7FFF) == StatType.UNKNOWN + @test safe_stat_type(0) == StatType.UNKNOWN + + # TradingEvent (UInt16) + @test UInt16(TradingEvent.NONE) == 0 + @test UInt16(TradingEvent.NO_CANCEL) == 1 + @test UInt16(TradingEvent.CHANGE_TRADING_SESSION) == 2 + @test UInt16(TradingEvent.IMPLIED_MATCHING_ON) == 3 + @test UInt16(TradingEvent.IMPLIED_MATCHING_OFF) == 4 + @test safe_trading_event(3) == TradingEvent.IMPLIED_MATCHING_ON + @test safe_trading_event(UInt16(4)) == TradingEvent.IMPLIED_MATCHING_OFF + @test safe_trading_event(999) == TradingEvent.NONE + + # InstrumentClass single-char codes + @test InstrumentClass.FX_SPOT == InstrumentClass.T(UInt8('X')) + @test InstrumentClass.INDEX == InstrumentClass.T(UInt8('I')) + @test DBN.safe_instrument_class(UInt8('X')) == InstrumentClass.FX_SPOT + @test DBN.safe_instrument_class(UInt8('I')) == InstrumentClass.INDEX + + # Record flags bit field + @test F_LAST == 0x80 + @test F_TOB == 0x40 + @test F_SNAPSHOT == 0x20 + @test F_MBP == 0x10 + @test F_BAD_TS_RECV == 0x08 + @test F_MAYBE_BAD_BOOK == 0x04 + @test F_PUBLISHER_SPECIFIC == 0x02 + @test has_flag(0x80, F_LAST) + @test has_flag(F_LAST | F_TOB, F_TOB) + @test has_flag(UInt8(0xA0), F_SNAPSHOT) + @test !has_flag(0x00, F_LAST) + @test !has_flag(F_TOB, F_LAST) + end + + @testset "standalone F_LAST MBO record round-trips" begin + # MBOMsg is 56 bytes -> hd.length 14 + hd(iid) = RecordHeader(UInt8(14), RType.MBO_MSG, UInt16(1), UInt32(iid), ts0) + add = MBOMsg(hd(42), UInt64(1001), Int64(5_000_000_000_000), UInt32(3), + 0x00, 0x00, Action.ADD, Side.BID, ts0 + 10, Int32(100), UInt32(1)) + # Event terminator as CME now publishes it: no book effect, only the + # F_LAST flag (and its own ts_recv) carry information. + last = MBOMsg(hd(42), UInt64(0), UNDEF_PRICE, UInt32(0), + F_LAST, 0x00, Action.NONE, Side.NONE, ts0 + 20, Int32(100), UInt32(2)) + + recs = roundtrip(Schema.MBO, [add, last]) + @test length(recs) == 2 + @test all(r -> r isa MBOMsg, recs) + + # The book update itself no longer carries F_LAST + @test !has_flag(recs[1].flags, F_LAST) + @test recs[1].action == Action.ADD + + r = recs[2] + @test r.action == Action.NONE + @test r.flags == F_LAST + @test has_flag(r.flags, F_LAST) + @test r.price == UNDEF_PRICE + @test r.size == 0 + @test r.side == Side.NONE + @test r.ts_recv == ts0 + 20 + @test r.hd.instrument_id == 42 + + # Export renders the undefined price as NaN, and keeps the flag byte + mbos = MBOMsg[x for x in recs] + df = records_to_dataframe(mbos) + @test size(df, 1) == 2 + @test isnan(df.price[2]) + @test df.flags[2] == F_LAST + @test df.size[2] == 0 + @test !isnan(df.price[1]) + + # Compact printing must not choke on the UNDEF_PRICE sentinel + s = sprint(show, r) + @test !isempty(s) + end + + @testset "price-limit statistics (stat_type 17/18) round-trip" begin + # v3 StatMsg is 80 bytes -> hd.length 20 + hd = RecordHeader(UInt8(20), RType.STAT_MSG, UInt16(1), UInt32(7), ts0) + mk_stat(stype, px, seq) = StatMsg(hd, UInt64(ts0 + 1), UInt64(ts0), Int64(px), + typemax(Int64), UInt32(seq), Int32(0), + UInt16(stype), UInt16(0), UInt8(1), UInt8(0)) + upper = mk_stat(StatType.UPPER_PRICE_LIMIT, 5_100_000_000_000, 1) + lower = mk_stat(StatType.LOWER_PRICE_LIMIT, 4_900_000_000_000, 2) + + recs = roundtrip(Schema.STATISTICS, [upper, lower]) + @test length(recs) == 2 + @test all(r -> r isa StatMsg, recs) + @test recs[1].stat_type == 17 + @test recs[2].stat_type == 18 + @test safe_stat_type(recs[1].stat_type) == StatType.UPPER_PRICE_LIMIT + @test safe_stat_type(recs[2].stat_type) == StatType.LOWER_PRICE_LIMIT + @test recs[1].stat_type == UInt16(StatType.UPPER_PRICE_LIMIT) + @test recs[1].price == 5_100_000_000_000 + @test recs[2].price == 4_900_000_000_000 + # quantity is unset for price statistics + @test recs[1].quantity == typemax(Int64) + + df = records_to_dataframe(StatMsg[x for x in recs]) + @test df.stat_type == [17, 18] + end + + @testset "implied-matching status events round-trip" begin + # StatusMsg is 40 bytes -> hd.length 10 + hd = RecordHeader(UInt8(10), RType.STATUS_MSG, UInt16(1), UInt32(7), ts0) + mk_status(ev) = StatusMsg(hd, UInt64(ts0 + 1), UInt16(1), UInt16(0), UInt16(ev), + UInt8('Y'), UInt8('Y'), UInt8('~')) + on = mk_status(TradingEvent.IMPLIED_MATCHING_ON) + off = mk_status(TradingEvent.IMPLIED_MATCHING_OFF) + + recs = roundtrip(Schema.STATUS, [on, off]) + @test length(recs) == 2 + @test all(r -> r isa StatusMsg, recs) + @test recs[1].trading_event == 3 + @test recs[2].trading_event == 4 + @test safe_trading_event(recs[1].trading_event) == TradingEvent.IMPLIED_MATCHING_ON + @test safe_trading_event(recs[2].trading_event) == TradingEvent.IMPLIED_MATCHING_OFF + @test recs[1].is_trading == UInt8('Y') + + df = records_to_dataframe(StatusMsg[x for x in recs]) + @test df.trading_event == [3, 4] + end + + @testset "per-leg definition records and FX-spot / index classes" begin + # v3 InstrumentDefMsg is 520 bytes -> hd.length 130 + function mk_def(iid, sym, cls; leg_count = 0, leg_index = 0, leg_iid = 0, + leg_sym = "", leg_side = Side.NONE, leg_cls = InstrumentClass.OTHER) + hd = RecordHeader(UInt8(130), RType.INSTRUMENT_DEF_MSG, UInt16(1), UInt32(iid), ts0) + InstrumentDefMsg(hd, ts0 + 1, + # prices / qty + 25_000_000, 1_000_000_000, + ts0 + 86_400_000_000_000, ts0 - 86_400_000_000_000, + UNDEF_PRICE, UNDEF_PRICE, UNDEF_PRICE, 0, 0, 0, 0, + # ints + 0, 0, UInt64(iid), 0, 10, 1, 1000, 1, 1, 1, 1, 1, 0, 1, 0, 1, 2026, 0, 1, + # strings + "USD", "USD", "", sym, "ES", "XCME", "ES", "FFICSX", "FUT", "Index", "", "", + # class / misc + cls, UNDEF_PRICE, 'F', 0, 0, 0, 0, 0, 0, 'A', 12, 0, 0, false, 0, 0, 0, + # legs + leg_count, leg_index, leg_iid, leg_sym, leg_side, 0, leg_cls, + 0, 0, 0, 0, 0, 0) + end + + # A straddle now produces one definition record per leg, sharing the + # strategy's instrument_id. + leg1 = mk_def(100, "UD:1V: VT 0625K5000", InstrumentClass.OPTION_SPREAD; + leg_count = 2, leg_index = 0, leg_iid = 201, leg_sym = "ESM6 C5000", + leg_side = Side.BID, leg_cls = InstrumentClass.CALL) + leg2 = mk_def(100, "UD:1V: VT 0625K5000", InstrumentClass.OPTION_SPREAD; + leg_count = 2, leg_index = 1, leg_iid = 202, leg_sym = "ESM6 P5000", + leg_side = Side.ASK, leg_cls = InstrumentClass.PUT) + # FX spot now reports instrument_class = 'X' instead of 'F' + fx = mk_def(300, "EURUSD", InstrumentClass.FX_SPOT) + # 'I' is in the DBN spec; make sure it survives instead of falling to OTHER + idx = mk_def(400, "SPX", InstrumentClass.INDEX) + + recs = roundtrip(Schema.DEFINITION, [leg1, leg2, fx, idx]) + @test length(recs) == 4 + @test all(r -> r isa InstrumentDefMsg, recs) + + legs = [r for r in recs if r.hd.instrument_id == 100] + @test length(legs) == 2 # one record per leg, not per strategy + @test all(r -> r.leg_count == 2, legs) + @test [r.leg_index for r in legs] == [0, 1] + @test [r.leg_instrument_id for r in legs] == [201, 202] + @test [r.leg_raw_symbol for r in legs] == ["ESM6 C5000", "ESM6 P5000"] + @test [r.leg_side for r in legs] == [Side.BID, Side.ASK] + @test [r.leg_instrument_class for r in legs] == [InstrumentClass.CALL, InstrumentClass.PUT] + @test all(r -> r.instrument_class == InstrumentClass.OPTION_SPREAD, legs) + @test all(r -> r.raw_symbol == "UD:1V: VT 0625K5000", legs) + + @test recs[3].instrument_class == InstrumentClass.FX_SPOT + @test recs[3].raw_symbol == "EURUSD" + @test recs[3].leg_count == 0 + + @test recs[4].instrument_class == InstrumentClass.INDEX + + # DataFrame export keeps one row per leg and the class names + df = records_to_dataframe(InstrumentDefMsg[x for x in recs]) + @test size(df, 1) == 4 + @test count(==(100), df.instrument_id) == 2 + @test occursin("FX_SPOT", df.instrument_class[3]) + @test occursin("INDEX", df.instrument_class[4]) + end +end