Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 66 additions & 22 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,57 @@ All notable changes to DatabentoBinaryEncoding.jl are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [0.1.7] - 2026-09-17

### Added

- **`StatType` enum** (`UInt16`, matches the DBN spec) for interpreting
`StatMsg.stat_type`, including `UPPER_PRICE_LIMIT = 17` and
`LOWER_PRICE_LIMIT = 18`, which CME GLBX.MDP3 publishes since Databento's
2026-07 normalization change, plus the block-volume, indicative-close,
market-wide-circuit-breaker, auction-collar and venue-specific codes.
`safe_stat_type(raw)` maps unrecognized codes to `StatType.UNKNOWN`. The
struct field stays a raw `UInt16` (no breaking change).
- **`TradingEvent` enum** (`UInt16`) for `StatusMsg.trading_event`:
`NONE`, `NO_CANCEL`, `CHANGE_TRADING_SESSION`, `IMPLIED_MATCHING_ON = 3`,
`IMPLIED_MATCHING_OFF = 4` (the latter two are emitted by CME since 2026-07).
`safe_trading_event(raw)` maps unrecognized codes to `TradingEvent.NONE`.
- **`InstrumentClass.INDEX`** (`'I'`), present in the DBN spec but previously
missing here; such definitions decoded as `InstrumentClass.OTHER` with a
warning. (`FX_SPOT = 'X'`, which CME now uses for FX spot instruments instead
of `FUTURE`, was already supported.)
- **Record flag constants** `F_LAST` (128), `F_TOB`, `F_SNAPSHOT`, `F_MBP`,
`F_BAD_TS_RECV`, `F_MAYBE_BAD_BOOK`, `F_PUBLISHER_SPECIFIC` and a `has_flag`
helper. Relevant to the 2026-07 CME change where `F_LAST` moved from the
final MBO book update onto a standalone `action = Action.NONE` record with
`price = UNDEF_PRICE`, `size = 0`; that record already decoded, exported
(`NaN` price) and printed correctly, and is now covered by tests.
- Test coverage (`test/test_cme_2026_07.jl`) for the 2026-07 CME normalization
shapes: one definition record per strategy leg, the standalone `F_LAST` MBO
record, price-limit statistics, implied-matching status events and FX-spot /
index definitions.

## [0.1.6] - 2026-09-10

### Changed

- Replay pacing now uses a monotonic clock by default, so NTP or manual
wall-clock adjustments cannot change replay timing.

### Added

- Timestamp-paced **replay**: `replay_dbn(f, filename; ...)` re-emits a DBN file's
records in time order, pacing each `f(record)` callback by the gap between record
timestamps to simulate a live feed (backtesting, demos, driving real-time
consumers). `replay_records(f, records; ...)` does the same over an already-loaded
collection. Zstd-aware and skips unknown record types like `DBNStream`. Pacing is
anchored to absolute wall-clock targets so callback execution time is absorbed
instead of accumulating as drift. Options: `speed` (time-compression multiplier;
`Inf` = no waiting), `timestamp` (`:ts_event` or `:ts_recv`, with fallback),
`max_sleep` (cap large gaps; re-anchors after a clamp), and `precise` (busy-wait
sub-millisecond gaps, since `Base.sleep` only resolves ~1 ms on Unix / ~15 ms on
Windows). `clock`/`sleep_fn` are injectable for deterministic testing.

### Fixed

- Parquet compression names are validated before being interpolated into the
Expand Down Expand Up @@ -46,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the bug). Files written by
earlier versions with undefined statistics quantities contain `-1` on disk.

## [0.1.6] - 2026-06-24
## [0.1.5] - 2026-06-24

### Changed

Expand All @@ -59,29 +103,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Timestamp-paced **replay**: `replay_dbn(f, filename; ...)` re-emits a DBN file's
records in time order, pacing each `f(record)` callback by the gap between record
timestamps to simulate a live feed (backtesting, demos, driving real-time
consumers). `replay_records(f, records; ...)` does the same over an already-loaded
collection. Zstd-aware and skips unknown record types like `DBNStream`. Pacing is
anchored to absolute wall-clock targets so callback execution time is absorbed
instead of accumulating as drift. Options: `speed` (time-compression multiplier;
`Inf` = no waiting), `timestamp` (`:ts_event` or `:ts_recv`, with fallback),
`max_sleep` (cap large gaps; re-anchors after a clamp), and `precise` (busy-wait
sub-millisecond gaps, since `Base.sleep` only resolves ~1 ms on Unix / ~15 ms on
Windows). `clock`/`sleep_fn` are injectable for deterministic testing.
- `dbn_to_parquet` now compresses with **ZSTD by default** and accepts a `compression`
keyword — one of `"zstd"` (default), `"snappy"`, `"gzip"`, or `"uncompressed"`.
- Symbol resolution helpers that join the human-readable `raw_symbol` back onto
records using `Metadata.mappings`: `symbol_map(metadata)` builds an
`instrument_id -> [(start_date, end_date, raw_symbol)]` lookup,
`symbol_for(map_or_metadata, instrument_id, ts_event)` resolves the symbol
valid at a record's timestamp, `add_symbol_column!(df, metadata)` joins a
`:symbol` column onto a records DataFrame in place, and
`records_to_dataframe(records, metadata; symbols=true)` does the conversion +
join in one call. The join keys on `instrument_id`/`ts_event` (so it also works
for the row-expanded MBP-10 frame) and only applies when the query was resolved
with `stype_out = SType.INSTRUMENT_ID`.

### Fixed

Expand All @@ -104,6 +127,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `create_metadata_from_dataframe` (used by `parquet_to_dbn` / `csv_to_dbn`) no longer
throws from `minimum`/`maximum` on a zero-row DataFrame; an empty frame yields
`start_ts = end_ts = 0`, so a fully-empty Parquet/CSV round-trips back to DBN.

## [0.1.4] - 2026-06-22

### Added

- Symbol resolution helpers that join the human-readable `raw_symbol` back onto
records using `Metadata.mappings`: `symbol_map(metadata)` builds an
`instrument_id -> [(start_date, end_date, raw_symbol)]` lookup,
`symbol_for(map_or_metadata, instrument_id, ts_event)` resolves the symbol
valid at a record's timestamp, `add_symbol_column!(df, metadata)` joins a
`:symbol` column onto a records DataFrame in place, and
`records_to_dataframe(records, metadata; symbols=true)` does the conversion +
join in one call. The join keys on `instrument_id`/`ts_event` (so it also works
for the row-expanded MBP-10 frame) and only applies when the query was resolved
with `stype_out = SType.INSTRUMENT_ID`.

### Fixed

- `records_to_dataframe` (and therefore `to_dataframe` / `dbn_to_parquet`) no
longer throws `FieldError` on **TBBO / MBP-1** and **MBP-10**. The converters
read non-existent flat `bid_px_00…` fields; the bid/ask data actually lives in
Expand Down Expand Up @@ -182,7 +223,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
conversion to JSON/Parquet/CSV; byte-for-byte compatibility with the official
Rust implementation ([#18], [#19]).

[0.1.7]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.6...v0.1.7
[0.1.6]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.5...v0.1.6
[0.1.5]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.4...v0.1.5
[0.1.4]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.3...v0.1.4
[0.1.3]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.2...v0.1.3
[0.1.2]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.1...v0.1.2
[0.1.1]: https://github.com/tbeason/DatabentoBinaryEncoding.jl/compare/v0.1.0...v0.1.1
Expand Down
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name = "DatabentoBinaryEncoding"
uuid = "90689371-c8cb-40d1-831f-18033db90f74"
version = "0.1.6"
version = "0.1.7"
authors = ["Tyler Beason <tbeas12@gmail.com>"]

[deps]
Expand Down
52 changes: 52 additions & 0 deletions docs/src/api/enums.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,12 +125,64 @@ codes:
- `CALL` (`'C'`) - Call option
- `PUT` (`'P'`) - Put option
- `FUTURE` (`'F'`) - Futures contract
- `INDEX` (`'I'`) - Index
- `BOND` (`'B'`) - Fixed income
- `FX_SPOT` (`'X'`) - Foreign exchange spot
- `COMMODITY_SPOT` (`'Y'`) - Commodity spot
- `MIXED_SPREAD` (`'M'`), `FUTURE_SPREAD` (`'S'`), `OPTION_SPREAD` (`'T'`) - Spreads
- `OTHER` (`'?'`), `UNKNOWN_0`, `UNKNOWN_45` - Fallbacks for unclassified instruments

## Statistic Types

```@docs
StatType
safe_stat_type
```

`StatMsg.stat_type` is stored as a raw `UInt16`; use `StatType` to interpret it,
e.g. `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)` or
`safe_stat_type(r.stat_type)`. Notable values: `OPENING_PRICE` (1),
`SETTLEMENT_PRICE` (3), `OPEN_INTEREST` (9), `CLOSE_PRICE` (11),
`UPPER_PRICE_LIMIT` (17) and `LOWER_PRICE_LIMIT` (18) — the last two are the
session price limits CME GLBX.MDP3 publishes since 2026-07.

## Trading Events

```@docs
TradingEvent
safe_trading_event
```

`StatusMsg.trading_event` is stored as a raw `UInt16`; `TradingEvent` names the
codes: `NONE` (0), `NO_CANCEL` (1), `CHANGE_TRADING_SESSION` (2),
`IMPLIED_MATCHING_ON` (3), `IMPLIED_MATCHING_OFF` (4).

## Record Flags

```@docs
F_LAST
F_TOB
F_SNAPSHOT
F_MBP
F_BAD_TS_RECV
F_MAYBE_BAD_BOOK
F_PUBLISHER_SPECIFIC
has_flag
```

The `flags` byte on MBO/MBP/trade/BBO records is a bit field. `F_LAST` (128)
marks the last record of a venue event. Since 2026-07, CME MBO carries it on a
standalone `action = Action.NONE` record with `price = UNDEF_PRICE` and
`size = 0`, so book builders should apply updates on every record and treat the
`F_LAST` record purely as an event boundary:

```julia
foreach_mbo("glbx.mbo.dbn.zst") do r
r.action == Action.NONE || apply_to_book!(book, r)
has_flag(r.flags, F_LAST) && on_event_complete!(book)
end
```

## Usage Examples

### Working with Schemas
Expand Down
4 changes: 3 additions & 1 deletion src/DatabentoBinaryEncoding.jl
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ export foreach_cmbp1, foreach_cbbo1s, foreach_cbbo1m, foreach_tcbbo, foreach_bbo
export replay_dbn, replay_records # Timestamp-paced replay
export compress_dbn_file, compress_daily_files
export Schema, Compression, Encoding, SType, RType, Action, Side, InstrumentClass
export StatType, TradingEvent, safe_stat_type, safe_trading_event
export F_LAST, F_TOB, F_SNAPSHOT, F_MBP, F_BAD_TS_RECV, F_MAYBE_BAD_BOOK, F_PUBLISHER_SPECIFIC, has_flag
export price_to_float, float_to_price, ts_to_datetime, datetime_to_ts, ts_to_date_time, date_time_to_ts, to_nanoseconds
export record_length_bytes
export DBN_VERSION, FIXED_PRICE_SCALE, UNDEF_PRICE, UNDEF_ORDER_SIZE, UNDEF_TIMESTAMP
Expand All @@ -130,4 +132,4 @@ export dbn_to_csv, dbn_to_json, dbn_to_parquet, records_to_dataframe
export symbol_map, symbol_for, add_symbol_column!
export json_to_dbn, parquet_to_dbn, csv_to_dbn

end # module DatabentoBinaryEncoding
end # module DatabentoBinaryEncoding
157 changes: 157 additions & 0 deletions src/types.jl
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ Classification of financial instruments. Values match Databento's DBN
- `CALL`: Call option contracts
- `PUT`: Put option contracts
- `FUTURE`: Futures contracts
- `INDEX`: Index instruments
- `BOND`: Fixed income securities
- `MIXED_SPREAD`: Mixed spread instruments
- `FUTURE_SPREAD`: Futures spread instruments
Expand All @@ -263,6 +264,7 @@ Classification of financial instruments. Values match Databento's DBN
BOND = UInt8('B')
CALL = UInt8('C')
FUTURE = UInt8('F')
INDEX = UInt8('I')
STOCK = UInt8('K')
MIXED_SPREAD = UInt8('M')
PUT = UInt8('P')
Expand Down Expand Up @@ -665,3 +667,158 @@ warning (once) and map to `InstrumentClass.OTHER`.
end
return @inbounds _INSTRUMENT_CLASS_LOOKUP[idx]
end

# ---------------------------------------------------------------------------
# Venue-code enums and record flags
#
# `StatMsg.stat_type` and `StatusMsg.trading_event` stay raw `UInt16` on the
# record structs: the wire fields are plain integers and publishers may emit
# codes outside the published enum. The enums below are for *interpreting*
# those codes. Compare with `r.stat_type == UInt16(StatType.SETTLEMENT_PRICE)`
# or convert with `safe_stat_type(r.stat_type)` / `safe_trading_event(...)`.
# ---------------------------------------------------------------------------

"""
StatType

Type of statistic carried by a [`StatMsg`](@ref) (`stat_type` field). Values
match the official DBN `StatType` enum (`UInt16`). `UNKNOWN = 0` is a local
sentinel returned by [`safe_stat_type`](@ref) for codes this package does not
recognize (publisher-specific, or newer than this release).

# Values
- `OPENING_PRICE = 1`: Price of the first trade of an instrument
- `INDICATIVE_OPENING_PRICE = 2`: Probable opening price, published pre-open
- `SETTLEMENT_PRICE = 3`: Settlement price
- `TRADING_SESSION_LOW_PRICE = 4`: Lowest trade price of the session
- `TRADING_SESSION_HIGH_PRICE = 5`: Highest trade price of the session
- `CLEARED_VOLUME = 6`: Contracts cleared on the previous trading date
- `LOWEST_OFFER = 7`: Lowest offer price of the session
- `HIGHEST_BID = 8`: Highest bid price of the session
- `OPEN_INTEREST = 9`: Number of outstanding contracts
- `FIXING_PRICE = 10`: VWAP over a fixing period
- `CLOSE_PRICE = 11`: Last trade price of the session
- `NET_CHANGE = 12`: Change from the previous session's close
- `VWAP = 13`: Session volume-weighted average price
- `VOLATILITY = 14`: Implied volatility associated with the settlement price
- `DELTA = 15`: Option delta associated with the settlement price
- `UNCROSSING_PRICE = 16`: Auction uncrossing price
- `UPPER_PRICE_LIMIT = 17`: Exchange-defined upper price limit (published for
CME GLBX.MDP3 since the 2026-07 normalization change)
- `LOWER_PRICE_LIMIT = 18`: Exchange-defined lower price limit (as above)
- `BLOCK_VOLUME = 19`: Block contracts cleared on the previous trading date
- `INDICATIVE_CLOSE_PRICE = 20`: Probable closing price
- `MWCB_LEVEL_1 = 21`, `MWCB_LEVEL_2 = 22`, `MWCB_LEVEL_3 = 23`: Market-wide
circuit-breaker thresholds (7% / 13% / 20%)
- `AUCTION_COLLAR_REFERENCE_PRICE = 24`, `AUCTION_COLLAR_UPPER_PRICE = 25`,
`AUCTION_COLLAR_LOWER_PRICE = 26`: Auction collar prices
- `VENUE_SPECIFIC_VOLUME_1 = 10001`, `VENUE_SPECIFIC_PRICE_1 = 10002`:
Venue-specific statistics
- `UNKNOWN = 0`: Unrecognized code (local sentinel, not part of the DBN spec)
"""
@enumx StatType::UInt16 begin
UNKNOWN = 0
OPENING_PRICE = 1
INDICATIVE_OPENING_PRICE = 2
SETTLEMENT_PRICE = 3
TRADING_SESSION_LOW_PRICE = 4
TRADING_SESSION_HIGH_PRICE = 5
CLEARED_VOLUME = 6
LOWEST_OFFER = 7
HIGHEST_BID = 8
OPEN_INTEREST = 9
FIXING_PRICE = 10
CLOSE_PRICE = 11
NET_CHANGE = 12
VWAP = 13
VOLATILITY = 14
DELTA = 15
UNCROSSING_PRICE = 16
UPPER_PRICE_LIMIT = 17
LOWER_PRICE_LIMIT = 18
BLOCK_VOLUME = 19
INDICATIVE_CLOSE_PRICE = 20
MWCB_LEVEL_1 = 21
MWCB_LEVEL_2 = 22
MWCB_LEVEL_3 = 23
AUCTION_COLLAR_REFERENCE_PRICE = 24
AUCTION_COLLAR_UPPER_PRICE = 25
AUCTION_COLLAR_LOWER_PRICE = 26
VENUE_SPECIFIC_VOLUME_1 = 10001
VENUE_SPECIFIC_PRICE_1 = 10002
end

"""
TradingEvent

Additional context for a [`StatusMsg`](@ref) (`trading_event` field). Values
match the official DBN `TradingEvent` enum (`UInt16`).

# Values
- `NONE = 0`: No additional information given
- `NO_CANCEL = 1`: Order entry is allowed; modification and cancellation are not
- `CHANGE_TRADING_SESSION = 2`: A change of trading session occurred; daily
statistics are reset
- `IMPLIED_MATCHING_ON = 3`: Implied matching is available (CME's matching
engine is constructing implied depth)
- `IMPLIED_MATCHING_OFF = 4`: Implied matching is not available

CME GLBX.MDP3 publishes `IMPLIED_MATCHING_ON`/`OFF` status records since the
2026-07 normalization change.
"""
@enumx TradingEvent::UInt16 begin
NONE = 0
NO_CANCEL = 1
CHANGE_TRADING_SESSION = 2
IMPLIED_MATCHING_ON = 3
IMPLIED_MATCHING_OFF = 4
end

const _STAT_TYPE_LOOKUP = Dict{UInt16,StatType.T}(UInt16(v) => v for v in instances(StatType.T))
const _TRADING_EVENT_LOOKUP = Dict{UInt16,TradingEvent.T}(UInt16(v) => v for v in instances(TradingEvent.T))

"""
safe_stat_type(raw::Integer) -> StatType.T

Interpret a raw `StatMsg.stat_type` code. Unrecognized codes map to
`StatType.UNKNOWN` without a warning (publisher-specific codes are legitimate).
"""
safe_stat_type(raw::Integer) = get(_STAT_TYPE_LOOKUP, UInt16(raw), StatType.UNKNOWN)

"""
safe_trading_event(raw::Integer) -> TradingEvent.T

Interpret a raw `StatusMsg.trading_event` code. Unrecognized codes map to
`TradingEvent.NONE` ("no additional information") without a warning.
"""
safe_trading_event(raw::Integer) = get(_TRADING_EVENT_LOOKUP, UInt16(raw), TradingEvent.NONE)

# Record `flags` bit field (MBO / MBP / trade / BBO records). Values match the
# official DBN `flags` module.

"""
Flag bit: last record in the event for a given `instrument_id`. Since the
2026-07 CME normalization change, GLBX.MDP3 MBO emits this on a standalone
record (`action = Action.NONE`, `price = UNDEF_PRICE`, `size = 0`) that
follows the book updates, rather than on the final book update itself.
"""
const F_LAST = 0x80
"""Flag bit: top-of-book record, not an individual order."""
const F_TOB = 0x40
"""Flag bit: record sourced from a replay, such as a snapshot server."""
const F_SNAPSHOT = 0x20
"""Flag bit: aggregated price-level record, not an individual order."""
const F_MBP = 0x10
"""Flag bit: `ts_recv` is inaccurate due to clock issues or packet reordering."""
const F_BAD_TS_RECV = 0x08
"""Flag bit: an unrecoverable gap was detected in the channel."""
const F_MAYBE_BAD_BOOK = 0x04
"""Flag bit: publisher-specific event."""
const F_PUBLISHER_SPECIFIC = 0x02

"""
has_flag(flags, flag) -> Bool

`true` if the `flag` bit (e.g. [`F_LAST`](@ref)) is set in a record's `flags`.
"""
has_flag(flags::Integer, flag::Integer) = (flags & flag) != 0
Loading
Loading