Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
ade5035
fix(reference): preserve mark-index stale causes for consumer recovery
BobbyAxerol Sep 27, 2026
3436b8e
docs: record reader recovery rollout and thirty-minute observation
BobbyAxerol Sep 27, 2026
609bb32
Bind public-market workload keys to paper sandbox and live realms
BobbyAxerol Sep 27, 2026
fba8863
Materialize verified recent bars without certifying overlapping nativ…
BobbyAxerol Sep 27, 2026
a179bdb
Record three-realm reader acceptance and native burst capacity bounda…
BobbyAxerol Sep 27, 2026
6ac1a60
Preserve exact rejected quality and bound native5000 burst profiles
BobbyAxerol Sep 27, 2026
84352ca
Diagnose rejected market views against exact canonical lineage
BobbyAxerol Sep 27, 2026
b25c136
Separate execution reference acceptance from generic mark snapshots
BobbyAxerol Sep 27, 2026
6f9bb30
Align captured trade fixture with production delivery contract
BobbyAxerol Sep 27, 2026
53beea1
Preserve RPC timeout at diagnostic observation boundary
BobbyAxerol Sep 27, 2026
fb58ae6
docs: record exact consumer burst limits and unfinished recovery
BobbyAxerol Sep 27, 2026
baa50b5
docs: record consumer atomic recovery evidence and rollout boundary
BobbyAxerol Sep 28, 2026
a1a9b41
docs: record BNB L2 freshness evidence and consumer handoff result
BobbyAxerol Sep 28, 2026
166cda1
fix: preserve verified L2 continuity across processing backlog
BobbyAxerol Sep 28, 2026
461577b
docs: record BNB L2 rollout recovery and exact eligibility limits
BobbyAxerol Sep 28, 2026
59917e9
docs(plan): record approved dev-only source synchronization
BobbyAxerol Sep 28, 2026
4c6999f
style(rust): format verified L2 continuity regressions
BobbyAxerol Sep 28, 2026
a3d1b40
merge: integrate verified Data Layer recovery into dev (#24)
BobbyAxerol Sep 28, 2026
7ee4af0
docs(plan): record verified dev integration and scoped branch cleanup
BobbyAxerol Sep 28, 2026
b394a36
fix(okx): preserve inverse underlying and derivative currency identity
BobbyAxerol Sep 30, 2026
0b90e8b
feat(config): declare bounded OKX inverse sandbox acquisition demand
BobbyAxerol Sep 30, 2026
1615eb2
feat(config): prepare sealed additive inverse sandbox read packet
BobbyAxerol Sep 30, 2026
486bacc
docs(p18e): record exact native and KN inverse packet additions
BobbyAxerol Sep 30, 2026
37909dd
docs: record additive inverse sandbox rollout and real readback
BobbyAxerol Sep 30, 2026
7e28f02
fix: derive hot L2 cadence for execution demand aliases
BobbyAxerol Sep 30, 2026
8559d64
docs: record inverse L2 rollout and actual risk readback
BobbyAxerol Sep 30, 2026
7015adf
docs: freeze one-phase execution readiness repair and runtime audit
BobbyAxerol Sep 30, 2026
62d50f8
fix(core): bound batch collection by an absolute deadline
BobbyAxerol Sep 30, 2026
355f3d3
fix(l2): wire session-fenced resnapshot feedback behind runtime opt-in
BobbyAxerol Sep 30, 2026
92e82aa
fix(l2): ignore resync hints covered by a fresh bootstrap anchor
BobbyAxerol Sep 30, 2026
fbbe34e
fix(l2): bound feedback checkpoint pressure and fair socket polling
BobbyAxerol Sep 30, 2026
e807447
docs: record bounded raw capture and optimized recovery artifacts
BobbyAxerol Sep 30, 2026
0c4548b
docs: attest RF3 replay and autonomous owner recovery evidence
BobbyAxerol Sep 30, 2026
0e00c81
docs: record matched RF3 throughput and recovery limits
BobbyAxerol Sep 30, 2026
45ba4ec
docs: attest Rust recovery candidate and scoped rollout packet
BobbyAxerol Sep 30, 2026
7673adf
docs: record Rust rollout and real consumer recovery acceptance
BobbyAxerol Sep 30, 2026
556db7c
fix(release): version SDK diagnostics artifact as 2.0.6
BobbyAxerol Sep 30, 2026
f460dee
docs(release): attest v2.2.1 recovery delta and scoped retirement
BobbyAxerol Sep 30, 2026
e366f75
fix(security): update JWT and HTTP dependencies for release audit
BobbyAxerol Sep 30, 2026
1c6c35a
docs(plan): reconcile runtime acceptance and security packaging check…
BobbyAxerol Sep 30, 2026
ec8d473
docs(release): seal v2.2.1 runtime and query security acceptance
BobbyAxerol Sep 30, 2026
830ac97
Merge #25: certify bounded execution readiness v2.2.1
BobbyAxerol Oct 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions DATA_LAYER_SERVICE_ACCESS_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,22 @@ execution or risk authority except a separately declared fresh
`MARK_INDEX_PRICE` read. A new alpha must add its own manifest requirement
before calling a reference metric; it must not reuse another alpha's identity.

### Execution MARK/INDEX Read Contract

Trading System's `latest_execution_feed(..., MARK_INDEX_PRICE)` uses
`POST /v2/market-data/reference:batch` with EXECUTION grade, one complete row,
`limit=page_size=max_pages=1`, and the identity's signed freshness/session policy.
The Kafka-native backend reads its verified market cache; this is not venue REST.
Component/session evidence may admit unchanged values only within the signed
component cadence. Disconnect, gap, generation/config mismatch or expired
component evidence remains blocked. Original source timestamps are preserved.

Generic `snapshot(MARK_INDEX_PRICE)` is a different read contract. Its event-age
rejection does not establish a failure of the component-aware execution reference
path. Tests must name the endpoint they actually exercised and retain both
outcomes; never silently substitute one to turn a failing gate green. TRADE
session liveness likewise never makes an old last-trade price execution-eligible.

## 1. Connection Architecture

Production services should **never** connect directly to external exchanges (Binance, DNSE, etc.) if they are running within the `bobby_network`. Instead, they must use the `data_layer` as a unified gateway.
Expand Down
2,119 changes: 2,119 additions & 0 deletions DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md

Large diffs are not rendered by default.

49 changes: 47 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,57 @@ It currently serves:
- Redis Pub/Sub streams for live consumers.
- REST endpoints for warmup, recovery, diagnostics, and health checks.

## Execution Readiness Patch (2026-09-30)

The v2.2.1 candidate release set is deployed and qualified for the affected
Data Layer scope. Publication is established by the [release tag](https://github.com/BobbyAxerol/quant-data-layer/releases/tag/v2.2.1), not this document alone.
Rust now uses an absolute batch deadline and bounded, generation-fenced L2
resnapshot feedback. The two Query readers include PyJWT2.15.1/urllib32.8.0
security fixes. No Kafka offsets, cache contents, TS strategies or order path
were reset or changed. The ten retired SQLite-path containers were removed;
state and named rollback images are preserved.

SDK2.0.6 is the new reproducible artifact; existing TS SDK pins were not changed.
Final300s observation: paper60 and sandbox22 demanded routes were READY on11
published heartbeats each. Four OKX disconnects recovered with fail-closed
refusals. This is not a continuous-uptime or always-eligible-price guarantee.
The sandbox manifest has64 entitlements, not64 active subscriptions.

After the Query security rollout,64 products x2replicas returned128 typed views:
125 usable,3 strict TRADE-age refusals. Two were provider-confirmed quiet last
trades; OKXSOL had a newer trade propagating through the pipeline. It was not
mislabelled quiet. Never use last-trade age or healthy session status alone as
permission to execute; execution prices require the appropriate QUOTE/L2 and
Risk policy. No freshness threshold was relaxed.

Latest SDK snapshot request-to-validated-usable **observed maximum, milliseconds**
(n8-12 usable samples/feed/venue, **not p99**, metadata resolution separate):

| Feed | Binance | OKX |
|---|---:|---:|
| QUOTE | 31.78 | 13.69 |
| MARK/INDEX | 49.49 | 20.64 |
| TRADE | 29.22 | 18.73 |
| BOOK_SNAPSHOT | 45.30 | 52.97 |
| BOOK_DELTA | 37.17 | 12.32 |
| BAR latest | 18.02 | 35.12 |

Full-stack CPU in the original final window: mean4.70cores, sampled peak5.12,
without cap increases. Redis polling measurements include up to1s observation
delay, not exact commit latency. Bounded authentic RF3 replay covered4k/s and
5k/s total ingress windows with catch-up, not indefinite5k/s per consumer.
See [v2.2.1 certificate](./upgrade/evidence/releases/v2.2.1/certificate.json),
[per-binding report](./upgrade/evidence/releases/v2.2.1/endpoint-report.json) and
[repair journal](./DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md#dl-execution-readiness-convergence-20260930).
Unchanged history/batch/reference capacity below is inherited dated evidence.

## KN Production Status (2026-09-27)

The **new Kafka-native read path is serving Trading System**, with SDK2.0.5.
Production final load passed: 50 logical alpha sessions, 15,504 requests,
90 streams, 12 reconnects, actual TS60 READY in32/32 samples; no order actions.
Ten old SQLite projector/query/stream roles are stopped, with state and exact
rollback artifacts retained. The v2.2.0 runtime certificate is below; publication
At that dated benchmark, ten old SQLite projector/query/stream roles were stopped;
they were retired on2026-09-30 with state and the current rollback set retained. The v2.2.0 runtime certificate is below; publication
is established by green remote CI, the main-line tag and GitHub release, not
by a healthy container alone. SDK cursor persistence is bounded and off-loop,
with durable acknowledgement after successful fsync.
Expand Down Expand Up @@ -69,6 +113,7 @@ and [cutover/rollback runbook](./docs/runbooks/kn5-paired-cutover.md).
- [Fund-grade implementation tracker](./DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md)
- [Fund-grade architecture and migration guide](./upgrade/quant-data-layer-fund-grade-upgrade-architecture.md)
- [OKX V5 market-data implementation guide](./upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md)
- [V2.2.1 release notes](./upgrade/evidence/releases/v2.2.1/RELEASE_NOTES.md)
- [V2.2.0 release notes](./upgrade/evidence/releases/v2.2.0/RELEASE_NOTES.md)
- [V2.2.0 runtime certificate](./upgrade/evidence/releases/v2.2.0/certificate.json)
- [Contributing guide](./CONTRIBUTING.md)
Expand Down
43 changes: 43 additions & 0 deletions config/v2/okx-inverse-sandbox-demand.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
schema: qdl.v2.production-demand.v1
revision: 1
consumers:
- consumer_id: trading-system.sandbox.stable
consumer_grade: EXECUTION
requirements:
- venue: OKX
market: SWAP
product_type: PERPETUAL
native_symbol: BTC-USD-SWAP
feed: QUOTE
interval: null
source_policy_id: crypto_primary_v2
- venue: OKX
market: SWAP
product_type: PERPETUAL
native_symbol: BTC-USD-SWAP
feed: MARK_INDEX_PRICE
interval: null
source_policy_id: crypto_liquid_v2
max_freshness_ms: 2000
require_live: true
index_native_symbol: BTC-USD
- venue: OKX
market: SWAP
product_type: PERPETUAL
native_symbol: BTC-USD-SWAP
feed: BOOK_SNAPSHOT
interval: null
source_policy_id: crypto_liquid_v2
depth_per_side: 100
max_freshness_ms: 60000
require_live: true
- venue: OKX
market: SWAP
product_type: PERPETUAL
native_symbol: BTC-USD-SWAP
feed: BOOK_DELTA
interval: null
source_policy_id: crypto_liquid_v2
depth_per_side: 100
max_freshness_ms: 2000
require_live: true
14 changes: 7 additions & 7 deletions poetry.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ dependencies = [
"httpx (>=0.28.0,<1.0.0)",
"idna (>=3.15,<4.0)",
"pillow (>=12.3.0,<13.0.0)",
"pyjwt[crypto] (>=2.13.0,<3.0.0)",
"pyjwt[crypto] (>=2.15.1,<3.0.0)",
"urllib3 (>=2.8.0,<3.0.0)",
"soupsieve (>=2.8.4,<3.0.0)",
"starlette (>=1.3.1,<2.0.0)",
"tomli (>=2.0.1,<3.0.0) ; python_version < '3.11'"
Expand Down
54 changes: 41 additions & 13 deletions qdl/adapters/binance/bar_edge.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@
from typing import Callable

from app.providers.binance.rest import BinanceProviderError, fetch_klines
from qdl.adapters.intervals import BarHistoryGapError, canonical_interval_ms, latest_closed_boundary_ms
from qdl.adapters.intervals import (
BarHistoryGapError, BarHistoryOverlapError, canonical_interval_ms,
is_valid_bar_open_ms, latest_closed_boundary_ms,
)
from qdl.provider.v1 import raw_provider_pb2
from qdl.raw.capture import capture_exact_frame

Expand Down Expand Up @@ -236,6 +239,43 @@ def fetch_closed_bar_history_raw_envelopes(
observed_ms=observed_ms,
interval_ms=interval_ms,
)
if len(closed) > page_limit or any(int(row[0]) > page_end_ms for row in closed):
raise RuntimeError("Binance closed-bar history page exceeds requested end boundary")
for row in closed:
open_time = int(row[0])
previous = selected_by_open.get(open_time)
if previous is not None and previous != row:
raise RuntimeError("Binance history pages conflict for one open time")
selected_by_open[open_time] = row
selected = tuple(selected_by_open[key] for key in sorted(selected_by_open))
# Walk newest first: only the suffix after the newest discontinuity can
# be certified, including when that discontinuity crosses REST pages.
for index in range(len(selected) - 1, 0, -1):
previous, current = int(selected[index - 1][0]), int(selected[index][0])
if current - previous == interval_ms:
continue
error = BarHistoryGapError("Binance", binding.native_symbol, binding.interval,
previous, current)
suffix = selected[index:]
latest_open = latest_closed_boundary_ms(
binding.interval, observed_ms, provider="BINANCE"
) - interval_ms
if (error.kind == "OVERLAPPING_PROVIDER_WINDOWS" and binding.interval == "3d"
and int(suffix[-1][0]) == latest_open
and all(is_valid_bar_open_ms(binding.interval, int(row[0]), provider="BINANCE")
for row in suffix)):
received_at_ns = time.time_ns()
envelopes = tuple(_capture_row(
binding, row, origin="BACKFILLED", received_at_ns=received_at_ns + offset,
test_provenance=test_provenance,
) for offset, row in enumerate(suffix))
raise BarHistoryOverlapError(
"Binance", binding.native_symbol, binding.interval, previous, current,
requested_rows=limit, recent_envelopes=envelopes,
)
raise error
if any(int(row[6]) > page_end_ms for row in closed):
raise RuntimeError("Binance closed-bar history page exceeds requested end boundary")
exhausted = False
if len(closed) != page_limit:
if not (allow_short and len(closed) < page_limit and _provider_exhausted_before(
Expand All @@ -247,18 +287,6 @@ def fetch_closed_bar_history_raw_envelopes(
f"requested={page_limit} observed={len(closed)}"
)
exhausted = True
if any(int(row[6]) > page_end_ms for row in closed):
raise RuntimeError(
"Binance closed-bar history page exceeds requested end boundary"
)
for row in closed:
open_time = int(row[0])
previous = selected_by_open.get(open_time)
if previous is not None and previous != row:
raise RuntimeError(
"Binance history pages conflict for one open time"
)
selected_by_open[open_time] = row
if exhausted or not closed:
break
earliest_open_ms = min(int(row[0]) for row in closed)
Expand Down
24 changes: 24 additions & 0 deletions qdl/adapters/intervals.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,30 @@ def __init__(self, provider: str, symbol: str, interval: str, previous: int, cur
super().__init__(f"{provider} closed-bar history discontinuity kind={self.kind} "
f"symbol={symbol} interval={interval} previous_ms={previous} current_ms={current}")

def retry_delay_seconds(self, attempt: int) -> int:
# Overlapping native windows cannot be repaired by fetching a missing
# bar. Recheck for provider corrections daily, retaining failed coverage.
if self.kind == "OVERLAPPING_PROVIDER_WINDOWS":
return 86_400
return min(2 ** min(attempt, 6), 30)


class BarHistoryOverlapError(BarHistoryGapError):
"""Incomplete requested history with an independently validated recent suffix.

Only history materialization may use these envelopes. Request fetchers still
raise: this is never a successful short response or proof of a listing floor.
"""

def __init__(self, provider: str, symbol: str, interval: str, previous: int,
current: int, *, requested_rows: int, recent_envelopes: tuple):
super().__init__(provider, symbol, interval, previous, current)
if self.kind != "OVERLAPPING_PROVIDER_WINDOWS" or not recent_envelopes:
raise ValueError("history overlap outcome requires a nonempty recent suffix")
self.requested_rows = requested_rows
self.recent_envelopes = recent_envelopes
self.older_prefix_status = "UNCERTIFIED_PROVIDER_OVERLAP"


_UNIT_MS = {
"s": 1_000,
Expand Down
6 changes: 4 additions & 2 deletions qdl/adapters/okx/instruments.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,9 @@ def parse_public_instrument(
multiplier_text = format(Decimal(ct_val_text) * Decimal(ct_mult_text), "f")
except InvalidOperation as exc:
raise ValueError("OKX contract multiplier fields must be exact decimals") from exc
family_parts = identity.canonical_symbol.split("-")
# Contract value currency is not the underlying asset for inverse contracts.
# Dated/option IDs include expiry/strike; currency identity comes from family.
family_parts = str(payload.get("instFamily") or identity.canonical_symbol).split("-")
family_base = family_parts[0] if len(family_parts) >= 2 else ""
family_quote = family_parts[-1] if len(family_parts) >= 2 else ""
record = InstrumentRecord(
Expand All @@ -93,7 +95,7 @@ def parse_public_instrument(
asset_class=_ASSET_CLASSES[inst_type],
native_symbol=inst_id,
base_asset=str(
payload.get("baseCcy") or payload.get("ctValCcy") or family_base
payload.get("baseCcy") or family_base or payload.get("ctValCcy")
).upper(),
quote_asset=str(payload.get("quoteCcy") or family_quote).upper(),
settlement_asset=str(
Expand Down
2 changes: 2 additions & 0 deletions qdl/consumer/manifest.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,8 @@ def __post_init__(self) -> None:
raise ValueError("consumer manifest contains an unknown data-plane permission")
if self.execution_dependency not in {"FORBIDDEN", "PAPER_ONLY", "ALLOWED"}:
raise ValueError("consumer execution dependency policy is invalid")
if self.environment in {"sandbox", "live"} and self.execution_dependency == "PAPER_ONLY":
raise ValueError("sandbox/live consumer cannot declare PAPER_ONLY execution dependency")
if not self.requirements:
raise ValueError("consumer manifest requires at least one data requirement")
if self.rollback_contract not in {"V1", "V2"}:
Expand Down
14 changes: 10 additions & 4 deletions qdl/reference/execution_live.py
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,8 @@ async def fetch(
request,
capability,
code,
"execution MARK/INDEX live view did not return a current active record",
"execution MARK/INDEX live view did not return a current active record: "
+ ",".join(dict.fromkeys(errors)),
)

def stats(self) -> dict[str, int]:
Expand Down Expand Up @@ -236,8 +237,13 @@ def _bounded_reason(response: httpx.Response) -> str:
except (ValueError, AttributeError):
return "UNAVAILABLE"
normalized = str(detail).upper()
if normalized.endswith(":STALE"):
return "STALE"
# Component cadence and session age are freshness failures, not an
# absent provider. Preserve the bounded reason for consumer diagnosis.
reason = normalized.rsplit(":", 1)[-1]
if reason in {"STALE", "COMPONENT_STALE", "SESSION_LIVENESS"}:
return reason
if reason in {"SESSION_STATE", "QUIET_POLICY_UNAVAILABLE", "LINEAGE_INVALID"}:
return reason
if normalized.endswith(":GAP_OR_RESYNC"):
return "GAPPED"
if normalized.endswith(":IDENTITY_MISMATCH"):
Expand All @@ -250,7 +256,7 @@ def _bounded_reason(response: httpx.Response) -> str:

@staticmethod
def _failure_code(errors: list[str]) -> str:
if "STALE" in errors:
if any(reason in {"STALE", "COMPONENT_STALE", "SESSION_LIVENESS"} for reason in errors):
return "LIVE_VIEW_STALE"
if "GAPPED" in errors:
return "LIVE_VIEW_GAPPED"
Expand Down
8 changes: 7 additions & 1 deletion qdl/runtime/kn_query_backend.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@
from qdl.adapters.intervals import canonical_interval_ms
from qdl.common.v1 import common_pb2
from qdl.marketdata.v2 import market_data_pb2
from qdl.query import DataRequirement, FeedType, GapRecord, HistoryResult, MarketDataItem, RecoveryPolicy
from qdl.query import CoverageStatus, DataRequirement, FeedType, GapRecord, HistoryResult, MarketDataItem, RecoveryPolicy
from qdl.query.contracts import CanonicalErrorCode, QueryProblem
from qdl.query.results import NON_REPLAYABLE_STREAM_CURSOR, QueryBackendError, GapScanResult
from qdl.query.row_cache import BoundedRowCache
Expand Down Expand Up @@ -368,6 +368,12 @@ def history_with_envelopes(
)
if result is None:
return None, ()
# Native Binance 3d has overlapping historical grids. A retained recent
# suffix does not prove listing exhaustion or certify an older prefix.
if (binding.feed is FeedType.BAR and binding.interval == "3d"
and binding.instrument.identity.venue == "BINANCE"
and start_ns is None and len(result.items) < requested):
result = replace(result, coverage=CoverageStatus.PARTIAL)
selected = records
if start_ns is not None:
selected = tuple(
Expand Down
Loading
Loading