diff --git a/DATA_LAYER_SERVICE_ACCESS_GUIDE.md b/DATA_LAYER_SERVICE_ACCESS_GUIDE.md index 7f78821f..3afac45d 100644 --- a/DATA_LAYER_SERVICE_ACCESS_GUIDE.md +++ b/DATA_LAYER_SERVICE_ACCESS_GUIDE.md @@ -99,6 +99,22 @@ execution or risk authority except a separately declared fresh `MARK_INDEX_PRICE` read. A new alpha must add its own manifest requirement before calling a reference metric; it must not reuse another alpha's identity. +### Execution MARK/INDEX Read Contract + +Trading System's `latest_execution_feed(..., MARK_INDEX_PRICE)` uses +`POST /v2/market-data/reference:batch` with EXECUTION grade, one complete row, +`limit=page_size=max_pages=1`, and the identity's signed freshness/session policy. +The Kafka-native backend reads its verified market cache; this is not venue REST. +Component/session evidence may admit unchanged values only within the signed +component cadence. Disconnect, gap, generation/config mismatch or expired +component evidence remains blocked. Original source timestamps are preserved. + +Generic `snapshot(MARK_INDEX_PRICE)` is a different read contract. Its event-age +rejection does not establish a failure of the component-aware execution reference +path. Tests must name the endpoint they actually exercised and retain both +outcomes; never silently substitute one to turn a failing gate green. TRADE +session liveness likewise never makes an old last-trade price execution-eligible. + ## 1. Connection Architecture Production services should **never** connect directly to external exchanges (Binance, DNSE, etc.) if they are running within the `bobby_network`. Instead, they must use the `data_layer` as a unified gateway. diff --git a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md index 36e2a25b..907c1003 100644 --- a/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md +++ b/DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md @@ -60124,3 +60124,2122 @@ immediate reader rollback5b66c2cd retained with same KN config. Old10SQLite role stopped; state retained, not assumed replayable beyond Kafka retention. V1/VN, otherTSroles,alpha execution and order path unchanged. This final documentation receipt will follow dev->main without altering the published tag or artifact hashes. + +### Post-release TRADE Burst Recovery And Live Consumer Readiness (2026-09-27) + +Status: IN_PROGRESS; owner approved narrow TRADE recovery throughput and +MARK_INDEX rejection correction, isolated stress measurement, then live binding +readiness. Guide: Kafka-native architecture review sections 18.12-18.13 and +KN cursor v3 contracts. No new architecture or full-catalogue rerun. + +Baseline: v2.2.0 Query f7351c3b / Rust 7fe34806 / TS reader 7e7e2d02; +TS manifest10, alpha manifests14; auth environment paper. Observed 13:00-13:05UTC +TRADE replay/overflow recurrence; later TS Binance BNB/DOGE MARK_INDEX +SOURCE_UNAVAILABLE. Earlier bounded acceptance is historical, not burst proof. +Owner plan edits elsewhere must be preserved. + +Scope: exact rejected-view diagnosis, bounded consumption/replay correctness, +shared Binance/OKX five-symbol regression, isolated real-capture stress with +throughput/queue/recovery/latency/resource evidence. Do not increase buffers, +relax freshness, drop durable trades, enable orders, reset state or broad-prune. +Record exact role/image rollback packet before runtime replacement. Live +identities stay inactive until affected acceptance passes; environment/cursor +fencing cannot be bypassed by renaming paper. + +Exit: regress apply/ACK failure, cancellation, reconnect/replay and MARK_INDEX +stale/disconnect/generation; isolated stress preserves identity/order and drains +bounded queues after burst; measure actual consumer readiness and request latency +separately from event-to-cache age. In-scope failures remain open. Cleanup only +new disposable resources; preserve active and named rollback artifacts. + + +Source correction checkpoint: MARK/INDEX private reader collapsed COMPONENT_STALE +and SESSION_LIVENESS into SOURCE_UNAVAILABLE. Preserve bounded cause, classify +these as DATA_STALE through existing query mapping; actual disconnect/fence/gap +remain fail-closed.47 targeted live-view/latency/lineage tests PASS(network none). +Read-only actual-cache diagnostic:12 reads each of10 MARK/INDEX products,0rejects +in this short window; this does NOT prove the earlier transient is eliminated. +The first diagnostic used InstrumentIdentity.native_symbol and stopped; corrected +to InstrumentRecord.native_symbol. Initial runtime image had no pytest; tests used +the existing TS test-capable image, no packages installed into runtime. + +TS source removes serialization/revalidation round-trip for its internal dual +market/execution projection, preserving external raw validation. Polling MARK +keeps failure health blocked but retries transient typed failures on its existing +cadence rather than escalating stream backoff; no stale cache rewrite.252 TS +consumer/bridge/projector/health regressions PASS including five symbols/two venues, +ACK failure and paired projection parity. Source-only; no runtime promotion yet. + +Isolated capture: md.canonical.v2 read_committed manual assign, auto commit/store +both false, no consumer-group join or offset mutation. Capture5543real TRADE rows +covers10products; SHA078fa7602f5d2b5077e9401ef3811ddda602adc6eda57958ebe36e12ce107599. +Typed templates fetched with existing TS identity,10products. Files private outside +Git at ~/.local/state/qdl-v2/burst-recovery-20260927. Test-clock replay preserves +provider bytes/timestamps and is not a live freshness certificate. + +Same 1CPU client/0.5CPU isolated Redis: legacy2668.9event/s vs paired3067.8event/s, +ordered execution payload digest identical e9c3a62e3a69aba1cf0232438c3294fceaf60f042b9fb0a947d94073e049c244. +Open-loop paired2000event/s,20test-only capture repetitions:110860offered/received, +0overflow,queue peak1batch(of16),55.431s,CPU38.845s,producer max lateness19.28ms. +This isolates projection+Redis pubsub, NOT gRPC/replay/ACK end-to-end. Do not infer +production burst closure or live identity activation from this component test. +Runtime stays v2.2.0; live mapping and full stream recovery stress remain open. + +Cleanup checkpoint: isolated ts-burst-redis stopped/removed; disposable clients +used --rm. Docker inventory confirms no burst test containers remain. No images +built, no cache/image prune and no shared volume cleanup. Post-check disk free +112754847744 bytes; no pre-clean measurement, so no reclaimed-byte claim. +Query1/2, Stream1/2 and market_data_service retain original image/start times, +restart0/OOMfalse. Live mapping inactive. Full gRPC/replay/ACK recovery stress +and affected runtime acceptance remain implementation work, not certified debt. + +Continuation: owner approved end-to-end recovery and runtime acceptance plus a +30-minute read-only observer subagent. Narrow packet: Query KN1/KN2 overlay +ade5035 on f7351c3b, TS market_data overlay75df1c4 on7e7e2d02; build/tests first, +record full resulting digests before rolling. Preserve configs/mounts/TLS, +Rust/Kafka/cache/V1/alpha/order path. Rollback exactly the replaced role images. +No live mapping until recovery and actual reader acceptance pass. Client fault +tests own isolated Redis/cursors; never pause shared production consumers. + +Runtime packet frozen before action: private burst-recovery-20260927/packet +query-candidate/rollback.compose.json and ts-candidate/rollback.compose.json. +Only image fields change: query_kn_1/query_kn_2 +sha256:f7351c3bda080a6d1266d49480dc65d7e590375f178252f858f87a04b1a4d685 -> +sha256:3dd54bfd8b9de0b143d317798ab9c63cebc502d01fe3cf6f513032d6484b5b11; +market_data (container market_data_service) +sha256:7e7e2d026a9f301db0c32237063b4f518a6503e986e6206ad9a7e1a79b01ceea -> +sha256:1d86d3e820795a50b2539b09a6d35b102efd461e7bd7dec1801730a5daffdc46. +Exact patched-file parents equal deployed files byte-for-byte. Packaged TS +252 tests PASS; first run had5 fixture mount path failures, corrected mounts. +Query normal import path passes and patched module SHA matches source. Direct +execution_live import exposes existing package cycle; normal query entry imports +query first. Initial FROM sha256 syntax could not resolve in BuildKit; corrected +to existing local tag and verified resolved digest. No dependency/network changes. +Roll Query1 then healthy, Query2 then healthy; then TS market_data only. Stop and +rollback affected role on startup error or persistent new unhealthy regression. +Retain same revision10/SDK2.0.5/config/network/mounts; no live activation yet. + +Runtime continuation evidence (2026-09-27): Query KN1/KN2 rolled sequentially +15:23UTC, TS market_data15:24UTC, same mounts/config/revision10/SDK2.0.5. No +other role recreated. Candidate images3dd54bfd/1d86d3e8 remain active; rollback +f7351c3b/7e7e2d02 retained. New source scripts accept_market_trade_recovery.py +and measure_v2_binding_reads.py in TS provide repeatable no-order measurements. + +Fullchain client uses actual Rust gRPC, TS facade/projection, isolated Redis and +SDK fsynced ACK. An independent stream starting at the same signed cursor is +compared through the final durable ACK; matching physical offset gaps are legal. +Initial20additionalstreams exceeded TS identity50stream quota (production40), +so OKX was refused; all5Binance prefixes matched. Fixed harness to one venue, +10additionalstreams, not quota relaxation. First OKX run was invalidated by my +premature stop of test-only Redis; errors kept in recovery-okx-interrupted.json. +Rerun OKX PASS5/5,8s inducedclient loss, recovered8.126-10.893s including pause; +Binance harder30s pause PASS5/5, recovered30.362-30.778s, exact12336records through +ACK,0cursorerrors. At-least-once reconnect may replay unapplied/unacknowledged +prefix; not an exactly-once broker-event publication certificate. Actual market +rate during these runs is not an accelerated2000/s end-to-end capacity proof. +Seven harness regressions PASS missing/changed/extra/empty-prefix and percentile +labelling. No production Kafka/Redis/cursor writes from fault clients. + +Separate external read-only TS Redis observer300.027s:160169notifications, +60feed/product groups,0cache missing/identity/watermark/future-time errors. +Request->typed usable SDK matrix120reads across2Queryreplicas:116usable, +4TRADEexecutioneligibility refusals retained, no MARK/QUOTE/L2/BAR refusals. +Per-feed n20(orTRADE16) is insufficient forp99. TargetedTRADE follow-up20reads, +18usable/2refused: exact rejected quality LIVE/sessionLIVE/gapfalse/complete, +eventSTALE at3126/3400ms vs3000ms. Do not call quiet last-trade execution usable. +Evidence at burst-recovery-20260927/{fullchain,cache-observer.json,observer-30m}. + +Live activation boundary checked in code, not assumed from identity names: +current TS manifest execution_dependency=PAPER_ONLY, environment=paper; +Query stable.py:529 and Rust auth.rs:302 enforce one configured auth environment. +KN cache LPK/cursor also includes environment. Live credentials cannot simply +be renamed/reused or accepted by relaxing environment verification. Existing +mainnet market bytes and tested TS60 do not certify a new live auth realm. +Live mapping remains inactive pending coherent identity/realm/cursor binding; +no widened authorization or automatic order enablement was performed. + +Closure preparation:259 focused TS tests PASS, Ruff changed scripts/tests PASS. +Disposed only ts-burst-cache, ts_burst_recovery_test network and --rm clients; +removed temporary client credential envs, test-only cursor directories and two +build contexts after hashing. Retained all market capture/receipt evidence. +Pruned exactly4 unshared source.local BuildKit records from the two15:20builds: +exxu40ud2r41aynmldcnw69lg,s1w22i9qi9uqketb116vdgvf6, +7xpedbd4elu2wq3lcernqwfm7,zfpet5gsafuvnqid1udyldlvm(~180kB reported). +Filesystem available112067633152->112331137024bytes; other host writes/retention +continue, so this delta is not attributed solely to cleanup. Both new images +are active, original two are exact rollbacks; no test-only image left from this +slice, no broad prune or old production-state removal. DataLayer has one checkout; +TS existing Portal/upgrade worktrees preserved, none newly created. + +Final bounded observation:1800s,2026-09-27T15:20:28Z..15:50:28Z,61samples; +after TS rollout52samples,51READY including startup,48/48post-startupREADY60/60. +One startup sample not-ready,0fallback/0v2errors after startup. Stream717401live +and4675replayed events (includes isolatedclients),0newoverflow/reader/decode/ +duplicate errors,queueend0.35quota refusals belonged to first oversized test; +no production TS session degradation there. No unplanned restart/OOM. +MARK reference refresh rejection counter0 on new TSreader. Execution readiness +is separate: final telemetry50/60execution-ready vs60/60session-ready, not a +claim every TRADE last price is usable. Known fiveBinance3d history source +refusals remain outside this correction; no fabricated bars or broadened cert. + +Resource mean DataLayer4.47665vCPU excluding TS/V1,TS0.72865vCPU; observed DL +mean within5core but not a new maximum-capacity certificate. TS cgroup peak186MiB; +no cap changes. Ingestor throttled-period fraction6.82%/8.16%, not percent CPU lost. +StableRedis dockerlogs timed out; bounded direct read of configured json-file +log confirms file0bytes. Health/cgroup accessible; do not claim unseen log proof. +Acceptance-index SHA398fb34a889f10b0e8710fbc51cbf2e119e79039393558d9fb4aabcc96be43f3. + +Recorded status: BOUNDED_READER_RECOVERY_RUNTIME_ACCEPTED / LIVE_MAPPING_INACTIVE. +No full accelerated gRPC burst capacity claim;2000/s was projection+Redis only. +Overall live-consumer task remains IN_PROGRESS due unimplemented coherent live +identity/realm mapping, not a request to loosen gates or repeat full C2. Current +TS paper/no-order reader can use the accepted correction. Published release +remainsv2.2.0; these immutable overlays are not a newly published stable tag. +No push/merge, no alpha/order activation. Main plans preserve owner edits. + +#### Approved continuation: native burst and live read identities + +Status: IN_PROGRESS. Owner requests full gRPC burst/recovery through 4000 events/s +and actual live mapping, not component-only throughput or renamed paper keys. +Continue under KN guide 18.12-18.13; preserve provider bytes, freshness, ACK-after- +apply, realm/cursor fences, existing paper access and all order services. +Tests: isolated captured-provider native gRPC -> SDK -> projection -> Redis -> +durable ACK, offered/completed accounting, bounded queue/RAM, disconnect/replay, +slow consumer isolation and drain. Report capacity only at measured rates. +Live credentials require explicit consumer entitlement, independent identity, +realm-bound auth/cursors and no-order reads. Journal exact runtime image/config +and per-role rollback before activation; do not reset shared market history. +Inspect known Binance 3d native historical window rejection without fabricated +repair. Source fixes require targeted regressions; no unrelated catalogue rerun. +Cleanup only this continuation's isolated resources; retain active/rollback images. + +Live-realm source edit was rejected by the automatic approval reviewer before +execution: widening authentication needs explicit key/consumer/realm proof. +No authentication file or runtime was changed. A narrow clarification was sent +for per-key paper/live realm binding, unchanged subject/revision/consumer checks, +shared public-market storage realm and consumer-bound cursor. Do not bypass this +rejection. Continue unaffected isolated burst and native history investigation. + +Owner explicitly approved narrow key-to-realm authentication packet and added +sandbox mode. Implement exact pinned key realms paper/sandbox/live, with existing +subject/consumer/revision and mTLS checks; common public-market storage namespace +and consumer-bound cursor remain unchanged. No per-request network auth lookup, +new service, order rights or silent credential reuse. Regress wrong realm/key/ +consumer/revision, old paper compatibility and cross-consumer cursor refusal. + +Source checkpoint: explicit JWT key realms implemented for paper/sandbox/live in +Python Query and Rust Stream; legacy single-realm config remains strict. Realm +rotation participates in Stream reauthorization. Public data LPK and signed +cursor storage realm remain unchanged; token subject/consumer/revision and cursor +consumer checks remain required. No new service or per-request network lookup. +Focused Python auth/compiler suite23PASS. Rust worker reports30unit+33native RPC +PASS (artifact receipt pending integration). Compiler scope restricted to exactly +TS/Binance-alpha/OKX-alpha approved IDs; FORBIDDEN alpha direct execution dependency +is preserved, not promoted. Initial compiler rejected FORBIDDEN alpha; fixed with +explicit approved-ID/policy map rather than a permissive generic promotion. +A Python3.10 invocation of KN backend tests failed import StrEnum; runtime is +Python3.12. Query integration tests require disposable Redis, skip without it; +skips are not acceptance. No live identity/config activated at this checkpoint. + +Auth slice verification: Python3.12 runtime image unittest auth/contract/async +cursor suite45PASS. Compiler now excludes deferred VN policy explicitly; TS +sandbox/live each60crypto requirements, alpha Binance/OKX each375. Six draft +manifests compiled outside Git, not active. Unit matrix covers paper/sandbox/live, +wrong key/realm/consumer/revision, unknown mappings, preserved quotas/purposes, +no implicit other-consumer promotion, and FORBIDDEN alpha authority preservation. +Ruff targeted security/compiler/tests PASS with --isolated (repository Poetry +requires-python caret is rejected by installed Ruff TOML parser otherwise). + +#### Native history and full transport measurement checkpoint + +Native Binance3d overlap confirmed by admitted real provider calls across five +symbols, original response bytes/hashes retained in native3d-audit-20260927T163746Z. +Implemented typed validated recent suffix outcome: Query larger requested windows +remain PARTIAL, edge may materialize only independently verified recent final bars. +No rewriting timestamps, no silently successful short history. Old overlap retries +are daily, sink retry reuses bounded captured suffix, final-bar watermark remains +separate from failed deep-history coverage. Tests114PASS/2explicit skips; isolated +Redis41/41PASS. Receipt native3d-suffix-20260927T170112Z/handoff.json records commands, +source hashes and cleanup. Five further real provider reads preserve original bytes. + +Full capture transport acceptance now includes native Rust gRPC with ephemeral +mTLS, production SDK validation and fsynced cursor ACK, TS projection and isolated +Redis. At2000event/s:110860/110860 across10products in55.710s,0duplicates/reconnects/ +cursorerrors, exact per-product canonical digest order and finalACK; no OOM. +This is capture replay under explicit projection-time clock, not current-price +eligibility nor a Kafka-ingestion capacity test. Constant original provider bytes +are reused20times with test-only monotonic log offsets. No production market writes. + +4000event/s debug attempt FAILED, followed once by optimized release comparison: +release offered110860 in27.714s,max producerlateness5.24ms,hub sampledlag17, +3overflows,client31251received/30991applied before harness stopped on reconnect +superseding an unACKed batch. No OOM/decode/reader/cursorerrors. End queue0 after +abort is NOT recovery success. Do not certify4000fullchain or edit away failure. +Changing test to discard/replay unACKed batch was rejected by automatic review; +no such edit executed. Existing fail evidence is retained. User-requested4000 +capacity question therefore remains negative for this current single consumer. + +New identity manifests prepared, NOT active: TS60 and alpha375 each for sandbox +and live. CA private key never persisted, independent workload TLS/JWT keys, +additive client trust copies only. Scope: existing Query1/2 and Stream1/2; image/ +config rollback recorded before action. No broker ACL, canonical realm, Kafka, +cache, V1, TSorder or alpha mutation. BAR-edge rollout follows Query coverageguard, +normal verified five-symbol3d bootstrap writes only; no offset reset/state deletion. +After suffix publication, do not rollback Query's PARTIAL guard: auth rollback can +restore old paper config on the fixed image. Old query image remains emergency +rollback with the pre-existing3d coverage limitation explicitly not certified. + +#### Three-realm reader packet (2026-09-27) + +Owner approval covers paper/sandbox/live read authentication, not order activation. +Prepared consumer-realms/reader-candidate.compose.json and reader-rollback.compose.json +outside Git. Query candidate sha256:0ed57643ae40087fe7d35ec04a778b2c8847d520f02aa0394a87b71dcdbeb959; +Stream candidate sha256:65918a656dcb92ab92b687a92195caa5642a185dfdc60c4bd3ebbc702609c09a. +Both source fba8863. Roll exactly query_kn_1/2 and stream_kn_1/2 sequentially. +Rollback Query to 3dd54bfd, Stream to 7fe34806 using prior mounts/config. +Trust changes are additive private copies, not writes to shared stable_tls. +No Kafka, cache, TS, alpha or order changes. BAR writer remains unchanged here. +First packaged unit invocation: six auth tests passed; two compiler tests could +not import development scripts. Rerun with read-only compiler/manifest fixtures; +do not count missing-fixture errors as successful packaged acceptance. + +#### Three-realm runtime acceptance and cleanup + +Status: AUTH_REALM_MAPPING_ACTIVE / EXECUTION_ACCEPTANCE_NOT_FULL. Packaged realm +suite8/8PASS after mounting development-only compiler fixtures. Sequential rolling +completed at17:18:07UTC for exactly two Query and two Stream roles. Query health +healthy; Stream has no Docker healthcheck, verified through real RPC and metrics. +All four restart_count0/OOMfalse; no other service recreated. TS remains paper +identity and V2_PRIMARY, heartbeat60/60sessionREADY,50/60executionREADY,0fallback/ +blocked/error at17:20:51UTC. No alpha started and no order rights activated. + +First auth probe:288typed responses,36negative refusals, but12stream ACK failures +were a harness/SDK packaging mismatch: reader image bundles pre-async-ACK SDK. +No production change required; mounted committed consumer SDK2.0.5 read-only in +the replacement client. Keep first receipt. Replacement:12/12stream receive+ACK, +36/36wrong realm/revision/consumer refused,287/288typed responses. One OKX BTC +MARK_INDEX_PRICE on Query2 refused SOURCE_NON_AUTHORITATIVE at SDK eligibility +validation; four quiet TRADE typed views not execution eligible.283/288usable. +Do not claim full execution acceptance or retry seeking a green observation. +Candidate auth is active; public data quality rules remain unchanged. + +Usable snapshot call-to-validated-result latency (48requests/feed, not p99): +QUOTE median8.27/max22.02ms; MARK median8.36/max12.98ms (47eligible); +BOOK_SNAPSHOT21.14/46.08ms; BOOK_DELTA10.35/67.84ms; BAR13.88/24.68ms; +TRADE8.44/156.75ms (44eligible). These are request latency, not venue event age +or TS Redis commit latency. Evidence:consumer-realms/evidence/acceptance-sdk205.json +SHA256a3240aaf1b3dd79efbcd8bdfd794481d0eb564b60748ddb2e23420884ab8718d; +activation-receipt.json records all exact digests/config/negative cases. + +Cleanup: acceptance clients auto-removed. Removed only two unmounted build +contexts13,893,772bytes; filesystem used198417211392->198403395584bytes. Recorded +image inventory; retained active Query0ed57643/Stream65918a65 and rollback +Query3dd54bfd/Stream7fe34806. No broad image/cache prune, no volumes or identities +deleted. Four-reader start times unchanged during cleanup. Canonical remains +/home/bobby/data_layer on fix/live-consumer-burst-recovery; release remainsv2.2.0, +not a newly published stable tag. No push/merge. BAR suffix source fba8863 is not +yet deployed to BAR writer.4000/s fullchain remains FAILED as recorded above; +2000/s exact captured transport/Redis/durableACK passed. + +#### Approved exact-quality and production-adapter capacity closure + +Status: IN_PROGRESS. Owner excludes Binance 3d from this closure (no data deletion +or binding disable). Guide KN18.12-18.13 and post-release realms apply. Preserve +provider timestamps, raw payload provenance, quality gates, ACK-after-apply and +all order services. Capture the exact rejected view and match trade/component +source evidence, not a subsequent successful read. Use the actual TS production +adapter batching/recovery path in isolated 4000/5000 total-event/s tests; profile +before optimizing, no larger buffers to conceal backlog. Test steady/burst/drain, +disconnect/replay, slow-reader isolation, exact ordering/idempotency/cursor state, +and latency through consumer Redis. Quiet stale TRADE must remain non-executable. +Full requires explicit positive fresh-data, expected negative quality, lineage, +capacity and recovery evidence; no certificate for unmeasured fan-out or Kafka +ingest capacity. No new phase/topology, no alpha/order activation. Record exact +per-role runtime packet/rollback before any required deploy. Cleanup only owned +test resources; preserve active/rollback artifacts and owner working-tree edits. + +Exact-quality slice checkpoint: SDK validation now attaches the rejected typed +view quality/source/contract/timestamps/watermark to existing ContinuityError +without changing its code or policy. Payload and signed cursor are excluded. +Acceptance tool keeps exact quality also for successful non-executable views. +Projection/quality unittest13PASS includes immutable error evidence and preserved +stale/disconnect/gap behavior. Runtime image lacks pytest; use existing unittest +runner, no package installation. Isolated driver now supports bounded5000/s, +repeat120 with scheduled offer<=120s and explicit low/high/low profile; original +provider bytes remain unchanged. New scheduling unit1PASS and release buildPASS. +Initial builder attempts failed toolchain PATH, then wrong Cargo cache layout; +correct existing registry mount is qdl-cargo-home:/usr/local/cargo/registry. +No production rollout or test capacity claim from these source/build results. +SDK stream/cursor unittest26PASS; combined affected unit count39PASS. + +#### Exact lineage checkpoint - canonical ahead of rejected views + +Status: IN_PROGRESS; EXECUTION_ACCEPTANCE_NOT_FULL remains unchanged. +Read-only180s probe captured5543canonical records and3228Query reads on both +replicas/real live+sandbox identities. BTC/OKX MARK had10rejections: all10 had a +newer same-session/generation canonical pair observed before the request started. +Across the three inspected Binance/OKX products,40TRADE stale views also had newer +canonical data before request start. This proves downstream lag for those samples, +not the exact stage responsible; no blanket quiet-market conclusion is valid. +The raw topic returned ACL29; no alternative credential or ACL bypass attempted. +Remaining samples cannot establish provider silence. Canonical inline capture +lineage supports the positive source-arrival findings. Query caches parsed rows +by full canonical bytes, not price-only hash; native projector timing remains +under investigation. No freshness threshold or runtime setting changed. + +Evidence: consumer-realms/evidence/exact-market-lineage-window.jsonl under the +20260927 burst-recovery evidence root; SHA256 +2100d05eef95a8152ef9de5c23f11c56b11586a8291d80fdd0527835a660f99b. +Probe unit6PASS (python3 -m unittest tests.test_probe_exact_market_lineage -v). +Tests cover exact original identity, pre-request ordering, generation mismatch, +incomplete capture, secret exclusion and bounded exclusive evidence files. +Production observation180s:13/13sessionREADY samples; not a burst certificate. +No production mutation; disposable probe clients removed; no image build or +broad cleanup. Published release remainsv2.2.0; candidate branch unchanged. + +#### MARK endpoint semantics and native state timing checkpoint + +Status IN_PROGRESS, not FULL. Read-only exact-hash canonical-to-Redis observer +interval553samples: median361.184ms,p95589.722ms,max743.927ms. These are observer +intervals, not broker commit timestamps. State-topic ACL29 prevents attributing +A commit versus state visibility versus B apply; no bypass or tuning performed. +TS-shaped MARK reference:batch658/658itemOK. Adjacent generic snapshot2/658refused +at2251/2275ms while reference accepted the same watermark with signed component +cadence/session evidence. The earlier generic SDK SOURCE_NON_AUTHORITATIVE must +not be reported as rejection of the TS reference path or loss of source authority. +Keep strict generic snapshot behavior and independently report both paths. +Acceptance/guide correction scope: explicitly exercise execution reference batch +with the same manifest requirements; retain generic refusal evidence, no silent +replacement of a required endpoint or weakening freshness. Add focused tests. +Timing probe5unitPASS; evidence projector-stage-timing-window.jsonl SHA256 +2635ff41265720c8e3156299f915b3b9f9fe3f5105bd2fad6bb6646745156dbe. +Runtime unchanged; scoped client removed, no build or broad cleanup. + +Acceptance now records an additional execution reference path for each declared +EXECUTION MARK requirement, preserving generic snapshot results and all existing +gates. Reference item errors stay visible with require_all=False as in TS; +status must beOK, and no retry or quiet-price exemption is introduced. +Guide clarifies the two contracts.8focused unitPASS (3path+5timing), existing +Query image with source mounted read-only/network none. No runtime rollout. + +#### Actual adapter capacity checkpoint - no sustained capacity pass + +Native driver fixture corrected require_final_bars=false to match actual TS +TRADE signed digest;2Rust tests/release buildPASS. TS actual-facade harness and +focused regressions169PASS. Core production conversion is no longer replaced by +a lookalike test loop. Exact offset/provider-byte oracle, final Redis and durable +cursor checks retained. No buffer increase, no production changes. + +Steady4000/s:55050/55430applied in34.276s, CPU34.239s; bounded replay backlog +exhausted after overflow. Steady5000/s:54939/55430applied in33.459s, CPU33.581s; +not certified. Captured low1000/high5000/low1000 burst33.253s plus2sclient loss +and10sslow reader:55430/55430ordered identities PASS, Redis/ACK10/10; caught up +7.126s after producer finish,4overflows/13148replayed then queue0. This is bounded +burst/drain evidence, not sustained5000 or production Kafka ingest capacity. +Per-reader saturation attribution remains limited; do not infer isolated queue +telemetry from aggregate overflow. Original provider clocks unchanged/test clock +explicit. Corrected profile harness self-time0.425s/10.90s; serial consumer CPU +remains the sustained bottleneck. Execution status remains NOT_FULL. + +TS serialization patch5970299 eliminates redundant legacy deep-copy;9unitPASS +and identical-operation microbenchmark measured38.44% less CPU for that function, +not a38% whole-system improvement. Harness evidence and commands indexed at +burst-recovery-20260927/adapter-capacity-checkpoint.json. Disposable clients removed; +no new image. Six anonymous-volume candidates retained without exact ownership, +not broadly deleted. Active/rollback images unchanged. No push/merge/release. + +Lineage reporting correction: the three traced UIDs include Binance BTC, +OKX BTC and OKX BNB, not three OKX products. UID/native venue mapping checked +against captured authenticated instrument templates. BTC/OKX10/10MARK finding +is unchanged;40TRADE lag findings span Binance and OKX. No evidence rewritten. + +Isolated native transport causal experiment: four arms verified6000/6000 captured +records, default fetch500/backoff1000 versus bounded20/10ms. Median transport +152.1->131.5ms; offset submission+final commit103.1->102.7ms; final commit alone +under1ms. It does not reproduce production361ms/744ms tail, omits Redis and RF3, +and does not justify a production tuning change. Evidence fetch-causal/summary.json; +opt-in instrumented scratch harness preserved outside Git with its orchestration. +Broker/client/network/executable removed. Disk used197753720832->197721161728bytes +includes concurrent host writes, not attributed entirely to cleanup. No images +built, no cap changes, active runtime images/start times unchanged. + +Targeted MARK10-product/two-replica fast matrix ran read-only with live TS identity. +All20reference cells have successful reads. One BinanceBTC reference was cancelled +at21.6ms by the six-second observer window, not a full service timeout; retain +this failed/censored receipt. Correct probe admission: observation end stops new +requests, admitted RPCs retain2.5s timeout; mark responses completing after window. +At most one RPC extends a window by2.5s. No quality/production policy changed. +First client attempt could not create evidence due UID permissions, before reads; +replacement client mounted only selected identity read-only and ran as container +root on read-only FS. Existing reference response times mostly8-10ms median. + +MARK fast matrix result:20/20product x replica cells returned successful execution +reference items. Initial209reference requests:208OK plus1observer-window cancel; +retain it. Targeted BinanceBTC follow-up after timeout-boundary fix:12/12OK, +6per replica, median8.95/9.80ms,max11.04/11.16ms. Other cells median7.85-10.00ms, +max21.42ms;11samples/cell, no per-cell p99. These are warm HTTP+SDK call durations +(the adjacent generic call establishes TLS), not TS Redis-commit latency. Generic +snapshot separately refused2BNB/OKX views; no gate or source timestamp changed. +Evidence mark-reference-matrix/summary.json and btc-followup.jsonl; original +receipts preserved. Observer helper unit6PASS. Both clients auto-removed, no new +image or production mutation. Actual TS runtime inspect: CPU/memory limits0 +(uncapped);2->4CPU budget increase applies only to isolated worker tests. + +#### Final capacity checkpoint - execution acceptance remains NOT_FULL + +TS source-only checkpoint fa9a597 adds opt-in fixed product-family workers inside +one market_data_service, no new service topology. Default1 unchanged; worker crash +resume deliberately refuses committed cursors until atomic whole-family state/ +watermark and nonregressing replay are implemented. This is unfinished scope, +not an approval blocker and not production-ready. Combined309unitPASS4.42s. + +Four-worker fresh-start replay of authentic capture:4000/s and5000/s both applied +55430/55430 ordered identities with Redis and durableACK parity. Worst route p99 +scheduled-offer-to-Redis1220.793ms at4k and4068.205ms at5k; finalACK drain461.032ms +and3240.890ms respectively.5k had2RATE_LIMITED recoveries. These short receipts do +not certify sustained60s capacity, crash recovery, or5000/sKafka ingress/fanout. +Evidence workers-4000-fresh-r10 and workers-5000-fresh-r10 under existing burst root. +Profile worker0 completed; pydantic conversion, SDK validation, async scheduling +and Redis serialization remain distributed costs. No notification coalescing, +buffer expansion, freshness relaxation or production rollout used to forcePASS. + +All disposable clients/Redis/gateway containers removed; Redis testdata tmpfs, +no new images built. Active/rollback retained, no broadprune. Publishedv2.2.0 +main8299838 unchanged. CanonicalDL fix/live-consumer-burst-recovery and canonicalTS +fix/data-layer-r10-consumer-handoff; owner source edits preserved. No newworktree, +push/merge/tag. RuntimeQuery0ed57643/Stream65918a65/TS1d86d3e8 unchanged; consumer +runtime remains paperidentity, additive live/sandbox readidentities are not an +execution-capacity certificate. Next required work remains monotonic atomic resume, +then sustained4k/5k and bounded actualconsumer acceptance, not another phase. + +2026-09-28 approved narrow consumer runtime packet, READY_TO_APPLY. +TS4e04c96 image sha256:08e0c9072f7c72d4c2ee5752018d25eff421be5166b0b5a931636528028487a8; +base/rollback sha256:1d86d3e820795a50b2539b09a6d35b102efd461e7bd7dec1801730a5daffdc46. +Only trading_system/market_data (container market_data_service). Preserve networks, +allcurrentenv except MARKET_DATA_FIXED_WORKERS=4, mounts,TLS,identitypaperrev10; +noother TS/DataLayer/alpha/order/Kafka/Redisflush/SQLite mutation. Stop oldconsumer, +backup andretainlegacycursor; copy preflightedownership into separatefixed-workers-v1 +cursor namespace. FreshQuerybootstrap validatesexistingexecutionwatermark, never +blindresume legacycursor. V2routes60,workerallocation24/12/12/12,82VNlegacyretained. +Candidatepackaging342PASS46.04s (actualinstalledSDK,notSDKsourcemount). +Packet/evidence: burst-recovery-20260927/atomic-rollout-4e04c96. +Currentbaseline59/60 withOKXBNB BOOK_SNAPSHOT DATA_STALE; BOOK_DELTA last-event +age~2494s/sessionLIVE/eligiblefalse. Separatefindingretained,notTRADElatencyclaim. +After rollout observe300s health/cache/logs/cursor/workertelemetry; failclosed. +Rollback recreateonlymarket_data usingrollback.compose.json,legacycursor remains. +PublishedDLv2.2.0 unchanged, noFulluntilconsumeracceptance. + +2026-09-28 closure checkpoint: IMPLEMENTED / ISOLATED_TESTED, RUNTIME_PENDING, +EXECUTION_ACCEPTANCE_NOT_FULL (no production-certification claim). +TS source4e04c96; image08e0c9072f7c72d4c2ee5752018d25eff421be5166b0b5a931636528028487a8. +Final TLS5k aggregate10TRADEroutes304865exact, maxroutep99678.120ms, finalACK +354.252ms afteroffer; client2.475avgcores/362.64MiBpeak/noCPUthrottle. +4k243892exact,p99431.449ms,ACK222.782ms; client1.991cores/341.61MiB. +Finalsame-source burst1000/5000/1000 plus2sdisconnect/10sslowreader:110860exact, +1reconnect,allRedis+ACKhashesmatch,p991950.866ms,finalACK37.279ms; zeroorders. +These are scheduledfixtureoffer-to-actualconsumerRedis latencies, notprovider +eventage/liveeligibility,not5000perconsumerorfullKafkaingest/fanoutcertification. +Index: ~/.local/state/qdl-v2/burst-recovery-20260927/atomic-rollout-4e04c96/acceptance-index.json. +Production stop was REJECTED by automatic approval review beforeexecution: exact +market_data_service outage/recreate impact needs explicit confirmation. Async +question sent withdigest,4workers,cursorbootstrap,briefoutage and1d86d3e rollback. +Noindirectworkaround. ConfirmedoldserviceRUNNING,1d86d3e,restart0; no300sactual +consumeracceptance orliveidentityactivation performed. Existing BNB/OKXsnapshot +stale remainsseparatebaselinefinding; noFullreleaseclaim. +Cleanup: allcapture gateway/client/Rediscontainers auto-removed; exact disposable +ts-atomic-recovery-redis andts-atomic-recovery-20260928 removed. Zero ts-atomic/ +qdl-atomic containers remain. Removed21testprivatekeys+238662Bbuildcontext; +retainedcandidate08e0c907 andactive/rollback1d86d3e forpendingpacket, publicTLS +certs/receipts/captures. No broadprune,sharedvolume/networkcleanup orworktree +delete. BuildKitcleanup deferred untilruntimeoutcome toavoidsharedcache removal. +Hostusedbytes before202716180480,after203296567296 (backgroundgrowth; doNOTclaim +netreclaimedspace). DLruntimeQuery0ed57643/Stream65918a65 unchanged; published +v2.2.0 main8299838. CanonicalDLbranchfix/live-consumer-burst-recovery fb58ae6; +canonicalTSbranchfix/data-layer-r10-consumer-handoff4e04c96; otheractiveTSfeature +worktrees untouched. No push/merge/release. Ownercompose/symbols/planhunks retained. + +2026-09-28 read-only BNB/OKX L2 investigation (sidecar, no source/runtime change). +Scope only BNB-USDT-SWAP UIDf2e37e2b-1386-5a32-9b79-0fd39ec7a5a3; +actual TS paperidentity/manifestrev10, existingSDK RestQueryTransport inside +market_data_service image1d86d3e at probe time, credential/TLS read in-process, +no secret output/cursor writes/stream/order/C2. Query0ed57643,Stream65918 unchanged. +TwoQueryreplicas, status+snapshot forBOOK_SNAPSHOT andBOOK_DELTA (8reads): +Snapshot bothSTALE,eventSTALE,sessionNOT_APPLICABLE,complete=true,gap=false, +eligible=false,LAST_EVENT_STALE. Offset274362767; source_event1790560208308000000, +received1790560208558479471. At request1790566026583565907 freshness5818360ms; +Query2 subsequentfreshness5818413ms. Both snapshot reads correctly DATA_STALE. +Delta bothLIVE/eventSTALE/sessionLIVE,eligible=false,complete=true,gap=false; +watermark274362869,source_event1790560208908000000,received1790560208957310113, +generation66791,sequence_verified=true,native_sequence_end17411513614, +snapshot_sequence17402419892. Sessionage7..999ms; eventage5817779..5817831ms. +Both exactsameview contractcatalog11/config19/authority1; authoritative=true, +sourceidokx-swap-bnb-usdt-swap-book-primary-v2. Snapshotcalls8.29/10.84ms, +delta8.96/8.77ms; statusfirstcold88.55ms/others8.29..14.13ms. Tinybounded +sample only, notp99/acceptance. Age97min is real olddata, notrequestlatency. +OneapprovedpublicOKX read viaexistingOkxRestClient.get('/api/v5/market/books', +instIdBNB-USDT-SWAP,sz100,bucketmarket,attempts1,timeout3s) fromQuery1: +start1790566065461403848/end1790566065585639096;124.235mscall; +providerts1790566065552,33.639msageatreceive,seqId17412847137,100levels/side, +bid769.2/ask769.3. This positively disproves providerquiet interpretation for +current stale book. Strictsnapshot rejection is correct, mustnotwiden60s gate +or relabel stale bookeligible fromsessionheartbeat. +Source review: Rust materializedsnapshot generated only onverifiedDeltaApplied/ +Keepalive via due_l2_materialized_snapshot; core canrequestresync; OKX ingestor +book_session_bootstrap usesInitialSnapshotAndGapResync (notperiodicRESTanchor). +ConfiguredBNBbooks exists,depth100,snapshot_refresh_seconds30. This alone does +NOT prove whichpipeline stagefroze. Boundedcorelogs show aggregate growing +REJECTED_AWAITING_SNAPSHOT but noBNBspecificstatechange inselected120m logs; +oneidentifiedsequencegap belongedtoBINANCEBTCquarterly, NOTBNB. DoNOTattribute +aggregatecountertoBNB. No raw/canonical-topic capture inthisboundedinvestigation: +latestQuerymaterialization is old, but root betweenproviderWSingest/core/resync/ +projector remains unlocalized. Nextnarrowstep: traceonlyBNBraw/canonical/head +andresyncownership; donotchangeSDKsnapshotsemantics orconsumerworkers tohideit. +Investigationdone, BNB executionreadinessUNRESOLVED, notFULL. No testresources +created; cleanupnotneeded, active/rollback/artifactsunchanged bysidecar. Main +separately reportsTSrollout693d866 duringinvestigation; probeabove predatesit. + +Sidecar runtime provenance correction: final dockerinspect still observed actual +market_data_service sha256:1d86d3e820795a50b2539b09a6d35b102efd461e7bd7dec1801730a5daffdc46 +running/restart0. Main message mentioned693d866 candidate; sidecar didnotverify +thatimageactive and doesnotassertsuccessfulrollout. No mutation bysidecar. + +2026-09-28 bounded read-only BNB localization follow-up, DONE_WITH_UNLOCALIZED_LAYER. +Within5min scope: inspected exactBNBbooks runtime configs acrosscore.json/core-002/ +core-003, allhaveUIDf2e37e2b/nativebooks/depth100/materializedsnapshot1000ms, +providerrefresh30s; ingestorbindingpresent. Bounded130m/tail20000percore logs +contain noBNBstatechange. Observedsequencegap isBinanceBTCquarterly, explicitly +NOTBNB evidence. Ingestor running/restart0, boundedstdout logs empty, json-file +rotation3x50MB; absenceoflog isnotabsenceofrawframes. Projector1/2 boundedstatus +haserrors{}, stageAinputs=outputs, stageBlatestappliedincreasing; theseaggregate +numbers donotruleoutproduct-specificfailure. No raw/canonical BNBrecord captured +inthiswindow, no existing authorized reader wasestablished for exacttopictrace; +no credentialescalation/ACLchange/groupjoin/offsetcommit performed. +Source review: core due_l2_materialized_snapshot publishes onlyverifiedDeltaApplied/ +Keepalive. OKXingestorInitialSnapshotAndGapResync disablesperiodicRESTanchor; +core request_resync islocaladapterstate. This suggestscheckingfeedbackfromcore +resynctoOKXsubscription, but doesNOTproveBNBcurrentlyresyncing. DoNOTdeclare +corethecause fromaggregateREJECTED_AWAITING_SNAPSHOT. Correct narrowbranching: +freshBNBraw+noBNBcanonical ->core/continuity/resync; freshBNBcanonical+oldlatest -> +projector/materialization; noBNBraw despitefreshprovider ->ingestorsubscription. +Exactmissing evidence: boundedBNBrawseq/time/session+canonicalseq/offset+latest +onmatchinginterval, thenownerforrequiredresync. No source/runtimechanges, no +furtherproviderrequests (previoussinglepublicread alreadyprovesfreshvenuebook), +noresourcescreated/cleanupneeded, noFullclaim. 60routeclosure stillblockedby +BNBbookmaterialization; doNOTwidenfreshness orrestartunrelatedservices. + + +### 2026-09-28 Final consumer-only rollout and 300-second receipt + +Status: `CONSUMER_RECOVERY_PASS_OBSERVED_300S`; aggregate +`EXECUTION_ACCEPTANCE_NOT_FULL`. No new release, push or merge. + +Approved scope completed: only market_data_service replaced at 03:45:20 UTC +with TS source 0a5400a, image sha256:1037c56a48d65156bb64136db78325678e4940b51bc12dbee050162cf9549217. +Four fixed workers; existing config/mounts and paper identity revision 10 kept. +Rollback image 1d86d3e and legacy cursor backup retained. No live-realm +activation, alpha/order action, Data Layer rollout, reset or flush performed. +Two earlier candidate attempts were rolled back, not counted as successful: +08e0c907 exposed stale Query refresh versus newer stream watermark; 693d866 +exposed QUOTE replay versus TRADE/BOOK-only health handling. Source corrections +10701df and 0a5400a retain strict source/generation/freshness fencing, make +verified older refresh a no-op, and keep replay health separate from execution +eligibility. Actual final-image affected suite: 197 passed in 10.75 seconds. + +Final observer exited 0 after 302 seconds; data window 300.009 seconds, +151 samples / 11 distinct service heartbeats. All heartbeat observations: +59/60 session-ready, four worker telemetry epochs present, no fallback. +Execution-ready counter was 48-49, NOT 59-60: session readiness must not be +reported as executable-price availability. No observed watermark regression or +execution checksum violation; cursor errors 0, ACK delta 26,897 and commit +delta 13,749. These sampled invariants do not certify every historical write. +CPU average 0.720 core, peak cgroup memory 434.24 MiB, throttle delta 0; +runtime running, restart 0, OOM false. Local Redis MGET p50 1.403 ms/p99 5.350 ms +is NOT Data Layer endpoint latency or event-to-Redis commit latency. No new +forced disconnect or stress run in this window; inherit separately documented +4k/5k captured-event stress and disconnect/recovery evidence, not live freshness. + +Remaining true defect: OKX BNB BOOK_SNAPSHOT repeatedly DATA_STALE; both Query +replicas serve matching book roughly 97 minutes old at diagnostic time, while +one provider read returned a 33.639 ms-old book. BOOK_DELTA session LIVE also +does not make that old book executable. Provider quiet is disproved. Exact +fault layer between raw ingest, core/resync and materialization is not yet +localized. Do not widen freshness, restart blindly or declare overall FULL. +Next narrow task is matching BNB raw/canonical/cache evidence and repairing +that specific path; no reason to repeat all previously certified products. +Candidate consumer remains active because this pre-existing book rejection +persists correctly, while the consumer recovery integration fixes passed. + +Cleanup completed within this task: disposable Redis/client containers and +network removed, two superseded candidate images 08e0c907/693d866 removed, +owned build contexts removed, exact two owned BuildKit IDs pruned (117 kB and +44.77 kB). Retain active 1037c56a plus explicit rollback 1d86d3e. No broad prune, +volume deletion, worktree removal or unrelated service change. Filesystem used +205,413,998,592 bytes before cleanup; 204,792,332,288 after (concurrent filesystem +activity means the whole difference is not attributable to this cleanup). + +Canonical TS /home/bobby/trading_system on fix/data-layer-r10-consumer-handoff; +canonical DL /home/bobby/data_layer on fix/live-consumer-burst-recovery. Existing +other-feature TS worktrees remain unchanged. Published DL v2.2.0/main8299838 +unchanged; Query 0ed57643 and Stream 65918a65 unchanged. Owner compose/symbols +and unrelated plan edits preserved. Final receipt outside Git: +`/home/bobby/.local/state/qdl-v2/burst-recovery-20260927/atomic-rollout-4e04c96/runtime-0a5400a/acceptance-receipt.json` +SHA256: 60b99f03dc80179ed39ed28551bc3be44df5479dc95232728c5b807c02efc028. + + +### 2026-09-28 BNB OKX L2 recovery correction + +Status: IN_PROGRESS. Owner requests narrow diagnosis and repair, not catalogue retest. +Guide: Kafka-native architecture review K4-T05 (snapshot/delta/resync) and K5 +runtime safety. Trace bounded raw -> canonical -> cache metadata first; fix +the owning shared Rust boundary without changing checksum/sequence/generation +or freshness. Test reproduced failure, recovery/duplicate/gap and symbol +isolation, then two Query replicas and TS consumer. No forged freshness, +offset reset, Redis flush, unrelated rollout, alpha or order activation. +Record role/digest/config and rollback before any required deployment. Retain +active image as rollback; clean task resources only. FULL requires verified +fresh book, not session heartbeat. apply_patch helper is broken; Rule32 exact +append fallback used, owner changes preserved. + + +BNB localization and approved runtime packet checkpoint: +- Core identity, manual Kafka assign, no group join/commit: live raw12s =125BNB + books of9898records, generation66791, partition4. Historical9s around freeze + contains90 contiguous BNBframes. Last accepted17411513614 -> nextraw17411513655 + withprev17411513614. Canonical12s =7018records, zeroBNBbook but221otherBNB. +- Core1 processing age rose20.6s ->53.7s at01:50UTC; quarantine3->343 then + awaiting-snapshot1697 persisted after backlog returned below1s. Source guard + resyncs on30s processing age, before applying otherwise contiguousdelta. + Exact BNB quarantine attribution not captured (readACL denied; no bypass). +- Correctness repair: lossless processing with original timestamps; local queue + age is not provider continuity failure. Real sequence/gap/generation rejection + stays. Reader freshness still rejects old records. No heartbeat-as-price. +- Owner approved exact rollout3cores, then onlyOKXingestor reconnect. Keepall + config/TLS/offsets/cache and servicequotas unchanged. Rollbackthreecores to + sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f. + OKXingestor stayssameimage; reconnect getsrealWSsnapshot afterlastrebalance. + This briefly affectsallOKXfeedsonthatservice, notonlyBNB. NoQuery/Stream/TS, + Kafkaadmin, identity, Redis, V1 ororder mutations. Candidate digest follows + completedtests/build. Existingcompose at releases/v2.2.0-02cd827/ + producer-candidate.compose.json; exactcurrentruntime inspected beforeapply. + + +Source test checkpoint (before deployment): lossless L2 replay patch plus four +new dual-venue regressions in rust/qdl-realtime-core/src/lib.rs. Old guard restored +in an outside-Git overlay makes the new contiguous replay regression FAIL; +correct source passes. Cargo cache initially retained the overlay single-package +binary; package-scoped offline clean then broad rebuild removed ambiguity. +Final affected Rust lib/bin gates:145passed,0failed,1ignored (provider admission +Redis integration, unchanged). Clippy realtime-core all-targets -Dwarnings PASS. +Reader/cache/SDK with disposable Redis:40passed,0skip (initial run without Redis +was11pass/29skip and is not counted as complete evidence). +An optional run of the unchanged ignored provider-admission test FAILED at +provider_admission.rs:706: it uses50,000ns cooldown rounded to1ms Redis TTL; +second network round trip can expire it. No admission code/policy changed and +no claim of full admission recertification. Report separately, do not widen +production cooldown or hide the test. Disposable Redis will be removed. +No runtime mutation yet. Source patch does not make historical prices executable; +Query/SDK still evaluate original event age and source/session fences. + +Runtime packet ready: source166cda1; immutable core image sha256:f2040ac9e9380d067e8bcb4c965b1d7ea39c181c08993b7dba1b096337f7ca79 +Verified packaged binary SHA256 85b56bbb49fb46aac163744246993cd4f9504d807efd7ab02856e22fbfa125e5. +Roll three cores sequentially using bnb-l2-recovery-20260928/candidate.compose.json, +then same-image OKX ingestor once after rebalance; rollback.compose.json has +verified current image/environment/command/CPU/RAM for all four roles. +All other services untouched. Owner explicitly approved this exact packet. + + +### 2026-09-28 BNB L2 repair deployed; scoped acceptance result + +Status: `BNB_L2_FREEZE_RECOVERY_PASS`. Aggregate execution remains +`EXECUTION_ACCEPTANCE_NOT_FULL`; no release/push/merge. Source166cda1. +Only rust_core/2/3 rolled sequentially04:24:54..04:25:46UTC to +sha256:f2040ac9e9380d067e8bcb4c965b1d7ea39c181c08993b7dba1b096337f7ca79. +OKX ingestor recreated04:26:41 at unchanged7fe34806 aftercorecatchup; real WS +snapshot generation66792 sequence17413544732 movedBNB AWAITING_SNAPSHOT->READY. +Sameconfig/TLS/resourcequotas and persistent generation; no offsets/reset/flush. +Query0ed57643,Stream65918a65,TS1037c56a,V1,alpha/order paths untouched. +Rollback3cores:7fe34806 with exact rollback.compose.json. OKXimage unchanged. + +Affected source tests:145Rustpass/1ignored,40Query/cache/SDKpass usingdisposable +Redis,clippyPASS. Oldguard counterfactual failsnewregression asintended. Optional +unchanged ignoredadmission test has1msTTL wall-clock failure, notedabove, not +included inL2passclaim. No admission policy change. Build initially used raw +imageID inFROM (Docker treatedasregistryname); corrected toverifiedlocaltag, +packagedbinaryhash checkedbeforedeploy. Counterfactualcache cleaned before +finaltests/build; no stale overlay binary deployed. + +Real TS observer300.008s:151samples/11distinctheartbeats,60/60session-ready +throughout,execution-ready49..50,zeroobservedwatermark/checksum violations, +cursorerrors0,ACKdelta30815/commitdelta17394. Telemetry complete146/151samples; +5samples44..52s had3/4reporters while allworkerheartbeatsREADY. Do notdescribe +thisascompletecounters at everyinstant. Everyrolledcontainer running,restart0, +OOMfalse. Thiswindow observedconsumer recovery; no forceddisconnect/stressrerun. + +100actualSDKreads perfeed perQueryreplica fromTScontainer (400total): +| BNB OKX read | Replica1 p50/p99 ms | Replica2 p50/p99 ms | Eligibility | +| --- | --- | --- | --- | +| BOOK_SNAPSHOT |21.55/35.19|19.64/41.44|200/200 eligible| +| BOOK_DELTA |8.65/16.66|9.24/16.72|195/200 eligible| +All400responses parsed successfully, sequenceverified/complete/no gap; +184same-watermark cross-replica metadata comparisons,0mismatch. Watermarks +advanced274768916->274779564(snapshot),274768985->274779637(delta),generation66792. +Requesttimer coversSDKcall->validatedreturn, notTSRediscommit. Sourceeventage +p99snapshot2879/2908ms; delta2679/2701ms. Five delta views at04:31:06..08 exceeded +2000ms freshness (2014..2905ms), correctly LIVE-session/STALE-event/ineligible. +Core2 contemporaneous raw-age max1686.9ms and commit max941.4ms are supporting +aggregate lag evidence, not exactperviewcausal attribution. Thisisnot recurrence +of97-minute frozenbook and notquiet-market proof. DoNOTclaim allreads executable +or widen2s. Tail-latency localization remains beforeunqualifiedexecutionFULL. + +Cleanup: testRedis/container/network removed; allprobe/build/testclients exited; +ownedbuildcontext7,344,642bytes andoldguardoverlay139,747bytes removed. Exact +BuildKit cache-idprune reclaimed0 (active-image sharedcache retained); no broad +prune orvolumesremoved. Filesystem usedbefore=206071111680, +after=205677232128bytes (concurrentactivity,notreclaimed-sizeclaim). +Activecoref2040ac9 andnamedrollback7fe34806 kept; latter also powersingestors/ +projectors. Existingbuilder/sharedtarget cache notbroad-pruned; package-scoped +counterfactual clean removed751.4MiBbeforefinalbuild. No temporaryworktreecreated. +CanonicalDL /home/bobby/data_layer,fix/live-consumer-burst-recovery; stablev2.2.0 +main8299838unchanged. TS canonical/featureworktrees unchanged; ownerplanhunks +andTScompose/symbols edits preserved. +Receipt: `/home/bobby/.local/state/qdl-v2/bnb-l2-recovery-20260928/receipt.json` +SHA256:e8f3e24bab7bb7d7bda33b78e311847a2644a1cb92999a18873bc7f1950853fa. No allcatalogue/evidence rerun; exactremaining findings areexplicit. + +### 2026-09-28 Approved Source Synchronization To Dev + +Status: IN_PROGRESS. Owner approved pushing tested patch branches and PRs to dev +only. Main, v2.2.0, runtime images, manifests and order paths remain unchanged. +Gate: reviewed CI triggers (no dev deployment), preserve owner changes, green CI +before merge, then fast-forward canonical dev and remove only verified merged +feature branches. TS local dev's ten commits are already ancestors of its consumer +branch; no duplicate cherry-pick is needed. Existing test evidence is inherited; +this source synchronization does not grant execution FULL or a new certificate. +Rollback: reviewed source revert, not runtime mutation or history rewrite. +Cleanup excludes images, state and unrelated active feature worktrees. +Guide: workspace AGENTS.md Git/Canonical Runtime rules and preceding rollout +receipt; existing BNB delta freshness limits remain explicit. + +Source-sync CI update: PR #24 opened after push 59917e9. Contract CI failed at +Rust formatting, before clippy/tests. Applied Rust 1.82 formatter to the affected +realtime-core file; cargo fmt --all -- --check PASS in existing bounded builder, +network disabled. Formatter containers removed automatically; no image built, +no runtime changed. Initial slim-toolchain/UID attempts failed without source +changes, corrected using existing builder and host UID 1001. +TS eb6e3af pushed; API token returns 404 for ExecutorBroker (access pending). +Three inherited TS commit subjects violate governance; do not bypass that gate +or rewrite published history. Main and release remain untouched. + +### 2026-09-28 Dev Integration Receipt (No Release Or Runtime Change) + +Data Layer source synchronization: COMPLETE. PR #24 merged through GitHub into +dev as a3d1b40d35ee1ea134997d28ed32cf1bec3360dd after CI run 36381804332 +passed contract-tests, kn-native-integration, sdk-python310 and unit-tests. +Full Python discovery: 2472 cases, 48 explicit skips, zero failures (1978.665s); +isolated Redis follow-up: 41 cases passed. Remaining API/operations/migration/ +load/replay/security/SBOM job steps passed. No new production certification. +Canonical /home/bobby/data_layer is now dev, tracking origin/dev. Both local +and remote fix/live-consumer-burst-recovery were deleted only after verified +ancestry and source-tree equality. No extra Data Layer worktree exists. +Owner's two plan hunks preserved byte-for-byte (diff SHA256 +1b86a683b2677ecc05eb3300a275157e1487bfac3f0e559dedb1369cbf2a1306). +Main remains 8299838 and published v2.2.0 unchanged. Runtime remains core +f2040ac9, Query 0ed57643, Stream 65918a65, ingestor/KN projector 7fe34806, +BAR edge f7351c3b; TS market_data_service 1037c56a. No manifest/realm/order +activation. Existing rollback images retained; no image build or prune. +Git-ref cleanup only, no worktree directories or runtime data removed. Host +filesystem used after cleanup: 208473567232 bytes; no disk-reclamation claim. +Disposable lint/test containers auto-removed. Superseded CI 36381448087 canceled; +its partial evidence did not replace the required successful current run. +Receipts outside Git: /home/bobby/.local/state/source-sync-20260928/ +(data-layer-merge.json, data-layer-ci-summary.json, owner patch backup). +This post-merge commit records operations only; no executable source changes. + +Cross-repo status: TS PR #17 is pushed, local 1420 unit tests and changed-file +Ruff pass, but hosted Actions fail before checkout without job logs. Owner was +asked for UI annotation; fine-grained PAT Checks limitation was corrected in +TS plan. TS is not merged, and its active feature/upgrade worktrees remain. +Do not call the combined two-repository synchronization complete yet. + + + +### TS P18.3E Additive OKX Inverse Sandbox Readiness (2026-09-30) + +Status: OWNER_APPROVED / IN_PROGRESS / NO_QDL_RUNTIME_CHANGE_YET. +Owner resumed the TS closure plan at +/home/bobby/.worktrees/trading-system-next-upgrade/TRADING_SYSTEM_UNIFIED_IMPLEMENTATION_PLAN.md +anchor ts-v2-p18-3e-closure-plan-20260930. This is its dependency repair, not a +new Data Layer phase or a rerun of KN certification. Follow the KN architecture +review sections18.4/18.12/18.13 and Post-release consumer realms; TS native Risk +and broker money acceptance remain TS-owned. + +Scope: exact OKX SWAP BTC-USD-SWAP execution read dependencies for +trading-system.sandbox.stable. Derive the minimal QUOTE/MARK_INDEX_PRICE/L2 +snapshot+delta set from actual TS demand; add TRADE only if required. No BAR, +universe, new service, alpha activation, live order, V1 or DNSE change. +Preserve current60 TS products and all other consumer entitlements. Compile +catalog/acquisition, sandbox manifest/JWT revision and sealed TS binding together; +do not widen paper/live identity scope or reuse their credentials. + +Source finding: parse_public_instrument currently prioritizes ctValCcy as base +asset when baseCcy is empty. For inverse BTC-USD-SWAP, USD is contract value +currency, not underlying BTC. Correct derivative family/base identification and +test exact units/settlement/multiplier/identity, preserving linear/spot and existing +dated contracts. This parser is used by catalog/inventory compilation, not the +Query hot path; no reader image rebuild follows automatically from this source +change. Generated runtime metadata must be correct before any feed activation. + +Tests: focused OKX instrument/production catalog/realm/binding regressions, +including wrong product/currency, stale/gap, no cross-realm entitlement promotion. +Use actual bounded provider metadata with capture hash outside Git. After packet +render, test exact inverse routes through both readers and real sandbox identity +to candidate Redis/Risk, plus unchanged60-route entitlement smoke. No whole299 +C2 or5000/s rerun absent an affected shared hot-path change. Current certified +scope is inherited, never extended to inverse without its own real evidence. + +Runtime decision boundary: prepare an additive immutable config packet and exact +per-role diff first; reuse current binaries when supported. Only roles whose +config actually changes may roll with existing mounts/state/offsets and per-role +rollback. No Kafka reset, cache flush, SQLite deletion, topology or authority +redesign. Keep Kafka-native materialization, never legacy spool. Disable inverse +entry and reconcile owned exposure before removing its binding on rollback. + +Git: canonical /home/bobby/data_layer now fix/okx-inverse-sandbox-readiness from +dev7ee4af0, no new worktree. Owner's existing plan edits (4insertions/1deletion) +preserved separately; only this appended journal may be staged by this task. +Original patch SHA9f915998b24e6c7f80a25de9e72c9fb2c6657e4293bb6e012069266f0b400696. +No push/merge/tag authorized by preparation. Cleanup only owned disposable test +resources; retain active and named rollback images and all runtime data. + +#### Metadata correction verification + +Before patch: 12 instrument test methods with four failing subcases demonstrate +inverse USD incorrectly used as underlying and dated/option expiry/side used as +quote currency. After patch, the existing immutable Query image0ed57643ae40 with +read-only source mount, network none and bounded1GiB/2CPU ran: +`python -m unittest tests.test_phase1_instrument_domain tests.test_production_catalog tests.test_phasec36_liquid_crypto_features -q` +Result:27PASS/0FAIL. Fixtures are contract evidence, not broker certification. +Underlying/quote come from instrument family; ctValCcy remains independently +preserved, settlement BTC and contract multiplier100 remain exact. Native IDs +and registry identity are unchanged. No Query hot-path or Rust change. + +One bounded public OKX metadata GET returned HTTP403 before capture; no authentic +metadata receipt or inverse readiness is claimed. Runtime sandbox manifest was +read successfully: revision1, max_streams50, existing policies unchanged. New +inverse demand must be additive to sandbox only and compiled together with its +sealed binding; do not clone additional scope into paper/live. TS bridge derives +active demanded feeds from bindings, so inverse BAR is not implicitly required. + +Disposable test clients auto-removed, no image built or runtime changed for this +slice. Existing active and rollback images retained; no broad cleanup. Owner plan +hunks remain outside this commit. Inverse activation and authentic E03 still open. + +#### Additive inverse demand and authentic metadata compile + +Added config/v2/okx-inverse-sandbox-demand.yaml, four exact execution dependencies +for trading-system.sandbox.stable: QUOTE, MARK_INDEX_PRICE, BOOK_SNAPSHOT, +BOOK_DELTA. This is acquisition demand, not activated entitlement. QUOTE acquisition +uses existing schema defaults; its strict consumer policy remains manifest-owned. +First test run rejected two unsupported QUOTE demand keys; removed those keys, +not the validation. Existing paper/live demand files remain unchanged. + +Tests:29PASS/0FAIL across test_okx_inverse_sandbox_demand, +test_phase1_instrument_domain, test_production_catalog and +test_phasec36_liquid_crypto_features. New regression checks additive preservation, +four-feed sandbox ownership, inverse currencies, native books contiguous sequence +and exact BTC-USD index mapping. No certification claim for mock provider fixtures. + +Existing container OKX client succeeded with one-attempt public instruments GET; +host403 is not evidence of unsupported inverse. Authentic capture SHA256: +82517a7c7f54f80673476046de2a74e21854a83b455905d5175aa68083ffa666. +Capture/receipt stored outside Git under TS p183e-20260930-dependencies. Offline +compile against that capture PASS: UID8482b54b-6d9d-5aad-abe8-e22447d183a1, +baseBTC/quoteUSD/settleBTC, multiplier100USD, tick0.1, quantity step0.1. +Acquisition SHA2560b88d3e92cf48e8874d18ab955883433ac0a2981d2a444ebe6a31f9d7b95072d. +Two compile attempts hit capture file0600 permission before parsing; final client +uses the host file-owner UID1001, read-only mount, network none, no added capability. +All clients --rm; no new image, container, network or volume retained. + +Remaining: additive runtime catalog/acquisition and sandbox revision/binding packet, +consumer policy/auth regressions, per-role diff/rollback and real inverse readback. +No runtime QDL mutation, order, reset, release, push or merge in this slice. + +#### Sandbox packet compiler scope (in progress) + +Implement a pure additive packet compiler using the existing catalog, acquisition, +ConsumerManifestLoader and ConsumerRouteBinding contracts. Inputs are exact current +public runtime artifacts plus bounded authentic metadata and reviewed inverse demand. +Preserve all old binding/source IDs and policies; reject conflicting/duplicate scope, +wrong realm, missing policy template, non-execution/fallback drift. New route policies +inherit the corresponding same-venue/product template, not hand-written relaxed SLA. +Seal a new configuration generation digest; never reuse the old certificate digest +as proof that inverse passed. Output PREPARED_NOT_ACTIVATED only. Unit negatives and +real mounted-config dry-run must precede any reader/core/ingestor packet apply. + +#### Sandbox compiler verification and mounted-config dry-run + +Implemented qdl/runtime/sandbox_extension.py and +scripts/prepare_sandbox_read_extension.py. Both are offline control-plane code; +no reader hot-path/image change. Packet preserves previous catalog/acquisition rows, +source IDs,60 products and quota; adds only four inverse sandbox products. Explicit +realm/subject/revision, same-venue template, exact coverage/collateral, fail-closed +policy and duplicate-application checks reject invalid extensions. Configuration +provenance uses a NEW digest with PREPARED_NOT_ACTIVATED_NOT_CERTIFIED status; +it does not inherit an execution certificate for the new inverse product. + +Test commands in existing immutable images, source RO/network none: +- unittest sandbox_extension + inverse_demand + instrument_domain + + production_catalog + phasec36_liquid_crypto_features + trading_consumer_scope: + 39PASS/0FAIL before the final subject/revision negative was added. +- final test_sandbox_extension:6PASS/0FAIL (includes subject/revision negative). +- test_consumer_key_realms + test_consumer_realm_execution_path:11PASS/0FAIL. +- Ruff changed three files --isolated --no-cache:PASS. Default Ruff failed parsing + existing requires-python="^3.10" in pyproject; no unrelated config edit made. + +Four exact public mounted inputs copied to protected external qdl-inputs (no keys). +prepare_sandbox_read_extension ran offline against real captured metadata and those +inputs; all loaders PASS. Output qdl-prepared/inverse-r2:64 requirements, sandbox +revision2, binding SHA81371d71e5e618d4331124e597e916aa5d13aeb40317a334717cb02b7e583b75. +Full input/output file hashes in receipt.json. TS independently parsed the packet +and retained the exact binding/hash using its SDK, source mounted read-only. +No runtime file was replaced. Public runtime key/type census only; one inspection +request that would have printed arbitrary scalar values was rejected by tools; +replaced it with a keys/counts-only read, no secret output. + +Rollout preparation must also extend the KN gateway/projector bundle (not just +Query YAML), and compile native core/OKX acquisition additions with existing +performance overrides/authority/offsets retained. Existing OKX ingestor has99 physical +entries; compiler must measure the new bounded connection split rather than raise +connection quotas implicitly. No extra role/container may be created. Retirement, +cache reset and whole-catalogue C2 are not part of this repair. +All disposable test clients removed. No image built, no runtime restarted, no broker +order, push, merge or release. TS E03/E04/E09 remain incomplete. + +#### Native and KN config preparation checkpoint + +Used StableAcquisitionPlan.core_config/native_ingestor_configs with exactly the +four new binding IDs, preserving current authority and excluding all old mappings +from generated additions. Result native-additions.json SHA256: +9ec45611f7323fe1ce1f2f263bd35f1b72ca46d3b014a56b902df9cebecde23d. +Each core695 ->699 physical mappings; OKX ingestor99 ->103. These are entries, +not a claim of103 connections; retain existing per-connection cap/lane splitting. +All old source/subscription IDs disjoint from additions. No default compiler tuning +is applied to existing roles: append the reviewed rows only, keep current headers. + +KN bundles rendered using scripts.kn_gateway_bundle.compile_bundle. Every old +catalog binding compared exactly equal to generated counterpart. Projector catalog +712 ->716, six manifests unchanged. Stream catalog712 ->716, twelve identities +retained, only trading-system.sandbox.stable replaced with revision2/64 requirements. +Projector new bundle hash6d5b1734594c929ccf8651b69796e58bf099785991ecee828422251950e25f36; +Stream c5d964c4af7c876402e3f30b9fe0792828f5d1ae5259e67f9c4b0802ac4e1861. +Raw file hashes and original bundle hashes: qdl-prepared/inverse-r2/kn-bundle-diff.json. +Physical market realm remains paper; no paper/live workload entitlement change. + +Safety: raw protected config copy was rejected because it might include secrets. +Did not bypass: inspected the documented non-secret authority schema, exported +only allowlisted public authority/instrument fields plus original file hashes; +separately validated gateway bundle schema/public manifest fields before export. +No credentials, JWT keyring, private keys, password or runtime env were exported. + +Next application packet must bind the exact current image/env/mount/config hashes, +retain rollback for three cores, one OKX ingestor, two KN projectors and four KN +readers. No Binance ingestor/BAR edge/V1/Kafka/Redis configuration or offset reset. +Reader catalog and KN bundles must converge before inverse source publication; +then hand off sandbox manifest revision to candidate TS market service only. +Do not call prepared mappings real feed readiness or E03 money evidence. All packet +artifacts are outside Git, no runtime file changed by this checkpoint. Tests used +existing images; all --rm clients gone; no new images/cache or cleanup required. + +#### Approved dependency runtime application packet (2026-09-30) + +Authority: owner resumed the critical-path dependency repair plan. Apply only the +prepared additive inverse market-read configuration; no broker order from this packet. +Verified all ten current Compose services resolve the exact current image, +environment, command and bind mounts (no drift). Reuse those existing Compose files, +no image build or new service. Baseline projector1 restart count1, all other nine0, +all OOMfalse; do not misreport the existing restart as caused by this packet. + +Exact role/image retention and rollback (same images, restore original config): +- rust_core/rust_core_2/rust_core_3: f2040ac9e9380d067e8bcb4c965b1d7ea39c181c08993b7dba1b096337f7ca79. +- ingestor_okx_swap and market_projector_1/2: 7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f. +- query_kn_1/2: 0ed57643ae40087fe7d35ec04a778b2c8847d520f02aa0394a87b71dcdbeb959. +- stream_kn_1/2: 65918a656dcb92ab92b687a92195caa5642a185dfdc60c4bd3ebbc702609c09a. + +Before writes: compare each original hash to preparation receipt, validate public +schema and store exact original bytes/mode/owner in protected rollback directory. +Atomically replace catalog/acquisition, KN projector+stream bundles, sandbox manifest, +and append four native mappings to each core/OKX config, preserving other fields. +Only OKX ingestor config_revision follows the new acquisition revision; session +checks use each envelope.config_revision, not a global assumed Binance revision. + +Roll sequentially: projectors, Query, Stream, cores, then OKX ingestor last so no +new source publishes before all reducers/readers know it. Existing all60 routes +retain requirements, but catalog-bound old cursors may expire and resnapshot as +specified by v3; no relaxation or fabricated continuity. Sandbox revision1 tokens +will fail after manifest2 activates until candidate market handoff. This is an +expected bounded no-order interruption, not permission to use stale cached prices. +Observe each role startup/restart/OOM and old feed progress; abort and restore +config/image packet on sustained source/identity/offset corruption or failed startup. +New inverse market writes use existing Kafka/native cache pipeline only. Never reset +Kafka, cache, offsets, Redis, SQLite, V1 or touch Binance ingestor/BAR edge or broker +mode. After reader convergence, change only ts-p183 market config/binding/JWT revision +and selected inverse symbol, preserving its separate Redis/cursor namespace. + +Application still pending at this journal checkpoint. No inverse execution +certification exists until actual Query/Stream -> candidate Redis/Risk readback. + +#### Inverse runtime application and actual readback result + +Applied the nine backed-up public configuration files and recreated exactly the +approved ten roles on their existing immutable images; no image build, offset +reset, cache deletion or topology change. Catalog12/716 bindings and sandbox +manifest2/64 requirements are active. Old60 requirement bodies unchanged. +Rollback files: p183e-20260930-dependencies/qdl-runtime-rollback-inverse-r2. +Stream/core/OKX recreation receipt: qdl-role-rollout.json. Projector startup +required rebalance before progress resumed; no claim of uninterrupted service. +A Query2 verifier compared environment-list ordering and stopped; dictionary +comparison confirmed equality, so Query2 was not recreated again. + +Actual SDK sandbox readback on both Query replicas:8/8 PASS, four inverse feeds, +complete/no-gap/eligible. First QUOTE calls125.76/134.18ms include connection setup; +MARK8.39/15.08ms, snapshot23.44/23.50ms, delta10.08/9.23ms. These are single-call +samples, NOT percentiles or broad latency certification. MARK lineage is existing +native execution live view, not substituted USDT data or direct REST fallback. +Candidate TS market recreated alone with same5dd12956 image, sandbox revision2, +new sealed digest and inverse symbol; shared TS market and order services unchanged. +Actual Redis candidate observation:12/12 heartbeat READY over55.86seconds,22/22 +session-ready. Stored eligibility is not guaranteed eligibility at a later read. +Read-only Risk MARKET/LIMIT check PASS; L2_LIMIT rejected43.445second snapshot +against intent5second freshness. Investigate snapshot selection without relaxing +Risk or claiming inverse order certification. No order submitted in this slice. + +Evidence: inverse-two-reader-readback.json, inverse-consumer-observation.json, +market-inverse-rollout.json under the dependency evidence root. Stable published +release remains v2.2.0; no push/merge/release. Active images also serve rollback +with restored public configs. No disposable image created; broad cleanup deferred +while rollback artifacts are needed. E03 remains9/24, not certified by feed reads. + +#### Inverse L2 Execution Cadence Correction (2026-09-30) + +Scope remains TS P18.3E approved dependency repair, not a new phase. Actual +candidate Redis/Risk six samples: MARKET/LIMIT6PASS, L2_LIMIT1PASS/5STALE under +unchanged5s intent gate; snapshot age4.109->14.133s, cache call2.69..9.42ms. +Prepared inverse native map omitted materialized_snapshot_interval_ms, falling +back to provider30s. Candidate TS snapshot read cadence also30s. Neither is a +provider support failure. Reuse existing1s execution-L2 materialization policy; +keep provider refresh30s, original event time, native sequence/generation and +Kafka-native path. Do not substitute BOOK_DELTA for a complete snapshot. + +Fix shared catalog compiler to derive hot cadence from execution-grade physical +book demand; both aliases must agree, including mixed consumer grades and +lexical dedup order. Research-only books unchanged. Test inverse/linear, +snapshot/delta alias equality, grade dedup and no unrelated acquisition changes. +No reader image rebuild required: config-only core3 cadence correction, exact +native-symbol/source match and backup, same Rust image; rollback only changed +core JSON. TS candidate market polling to1s is a separate existing-role config +change, journalled in TS plan, unchanged quality thresholds/identity/manifest. +No other consumer, Kafka offsets, Redis state, topology, V1 or order mutation. +First compile/test, then compare narrow runtime diff; exact inverse reader/Risk +observation after rolling affected roles. Inherited catalogue evidence remains. +Owner's two existing plan hunks remain unstaged; no push/merge/release yet. + +Verification:19 unittest methods PASS for inverse demand/catalog/shared L2 +materialization/core refresh, existing Query image with RO source, networknone, +2CPU/1GiB. First invocation used a nonexistent test_execution_l2 module; correct +module test_execution_l2_materialization rerun above. Execution-grade dedup and +mixed alias tests pass; provider30s unchanged. No image build needed for config. + +Cadence runtime check did NOT pass after rolling core3: all three samef2040ac9 +images running/restarts0; TS candidate only polling env changed, same5dd12956. +60s exact inverse+linear cache-to-Risk matrix saw BOOK stale/missing while +QUOTE/MARK continued. No broker entry. Config mismatch corrected, but snapshot +rebootstrap still unresolved after core recreation; not a PASS from process-up. +Next bounded recovery under approved inverse ingest scope: recreate only existing +ingestor_okx_swap, same7fe34806 image/config/mounts, to obtain authentic WS +snapshot; no offsets/cache reset. Record observed outcome, no assumption that +reconnect proves general autonomous recovery. Keep core config backups and +candidate market30s rollback. No whole-catalogue C2 or new image. + +#### Inverse Cadence Real Readback Result (2026-09-30) + +After same-image OKX ingestor reconnect, candidate Redis-to-Risk readback passed +30/30 samples per product for BTC-USD-SWAP and BTC-USDT-SWAP across MARKET, +LIMIT and L2_LIMIT (5s policy unchanged). Maximum book age2453/2893ms; maximum +cache read plus three policy checks6.843/44.460ms respectively. These are60s +bounded dependency observations, not endpoint p99 or E03 money evidence. +Evidence root: /home/bobby/.local/state/trading-system/p183e-20260930-dependencies/ +inverse-hot-cadence/; receipt risk-after-resnapshot.json. Failed first readback +risk-after.json retained. No broker entry occurred in this cadence slice. + +Correction to earlier provider-refresh shorthand: OKX uses initial WebSocket +snapshot plus gap resync, not a periodic30s snapshot. The retained numeric +snapshot_refresh_seconds does not change that policy. This cold-core rollout +required paired same-image OKX ingestor reconnect; this proves bounded manual +recovery, not autonomous core-restart recovery certification. No timestamps or +quality gates were relaxed; no BOOK_DELTA substitution for a complete snapshot. +Three QDL cores remain f2040ac9 with inverse hot cadence1000ms; OKX ingestor +remains7fe34806; candidate market remains5dd12956 with snapshot polling1s. +Config backups retained; no offsets/reset/flush/history deletion or order. +No new images built for this cadence slice; disposable test clients removed. +Native collector remains6a91aa52, rollback6cb0eb49. E03 remains9/24; E04/E09 +open. Next: freeze missing inverse lifecycle through SDK/service authority, +with fresh signed census, BEFORE money window and exact owned reduce-only exit. +No push/merge/release; published Data Layer v2.2.0 unchanged. Goal remains active. + + +## Execution Readiness Convergence - One Bounded Repair Phase (2026-09-30) + +**Status: IMPLEMENTATION_AND_RUNTIME_ACCEPTED / RELEASE_ARTIFACTS_SEALED.** +Publication is established by the immutable [v2.2.1 release](https://github.com/BobbyAxerol/quant-data-layer/releases/tag/v2.2.1), green release workflow and external final-publication receipt; this static build-time status alone never asserts publication. +**Owner request:** stop TS E work, investigate whether repeated failures are wrong +runtime/provenance or genuine Data Layer defects, repair the remaining related +defects in one phase, deploy affected immutable images and retire obsolete test/ +legacy artifacts. No repeat of passed, unaffected certifications. This is NOT +permission to mark E03/E09 complete or open another architecture program. + +### Goal And Governing Sources +Keep Kafka-native serving as the sole V2 path and make the existing execution +bindings recover predictably within their declared policy under the measured +workload. Preserve fail-closed behavior for genuinely quiet trades, stale prices, +gaps and generation changes. Finish with an exact role/image/config/manifest +receipt, affected consumer acceptance, scoped cleanup and a TS handback. + +Read workspace/project AGENTS first, this section, then: +- [KN detailed guide, KN-5](upgrade/DATA_LAYER_V2_KAFKA_NATIVE_ARCHITECTURE_REVIEW.md#kn-guide-phase-5): + K5-T02 hot partition/catch-up, K5-T04 paired handoff, K5.5 provenance, K5.6 retirement. +- The 2026-09-28 BNB L2 recovery correction/receipt in this plan + (source166cda1, receipt SHA e8f3e24bab7bb7d7bda33b78e311847a2644a1cb92999a18873bc7f1950853fa). +- Additive sandbox inverse extension and "Inverse Cadence Real Readback Result" + immediately preceding this section (same-venue contracts and real Risk readback). +- TS upgrade Unified Plan, "E07 Upstream Burst Attribution And Autonomous + Recovery (2026-09-30)" at TS source313770a. This is evidence input, not a + permission to edit TS order/Risk/alpha behavior. + +### Audited Runtime And Identity, Not Assumed Runtime +2026-09-30 inventory: one Data Layer Git checkout/worktree: +`/home/bobby/data_layer`, `fix/okx-inverse-sandbox-readiness@8559d64`. +Local dev7ee4af0; main8299838 / published v2.2.0. Local refs were inspected; +remote has NOT been fetched/CI-verified in this audit. +Two pre-existing owner plan hunks are deliberately preserved, not included in +this phase's commit. + +| Current role set | Active image prefix | Provenance / behavior | +| --- | --- | --- | +| Query KN1/2 | 0ed57643ae40 | fba8863, Kafka-native cache reader | +| Stream KN1/2 | 65918a656dcb | fba8863, native streaming | +| Rust core1/2/3 | f2040ac9e938 | 166cda1, includes backlog/continuity fix | +| Native Binance/OKX ingestors; KN market projectors1/2 | 7fe348060734 | 02cd827, not the retired SQLite projectors | +| BAR edge | f7351c3bda08 | 4fe7e92, retain unless affected | +| V1 fallback | dbfb57844977 | v1.2.4-2b0dcf7, intentionally retained | + +All current KN/core/ingestor/projector roles inspected have restart0/OOMfalse; +this is inventory, NOT execution acceptance. Kafka lifetime restart counters are +not a new incident. Runtime root is outside Git: +`~/.local/state/qdl-v2/releases/v2.2.0-02cd827`, with additive configuration +and realm-public mounts. Its directory name alone does NOT identify current config. + +Legacy query_v2_1/2, stream_v2_active/passive and SIX projector_v2* containers +are EXITED, not concurrent serving processes. Do not claim two active V2 data +architectures. V1 is intentional fallback, not a duplicate KN writer. +Two TS consumers are running: shared market_data_service (paper/revision10, +image1037c56a) and ts-p183 market_data (sandbox/revision2, image3b4ee780). +Both target https://qdl-v2-query:8200 and qdl-v2-stream-a/b:8210. The aliases +belong to active KN replicas; shared Query alias is intentional multi-replica DNS. +Neither consumer should be stopped/deleted as "duplicate Data Layer". + +Different per-role images are not themselves a defect. Freeze exact full digests +and relevant file hashes in the external packet; do not rebuild unrelated roles +merely to make image names identical. New sandbox inventory is64 requirements +(60 inherited +4 inverse), not permission to expand all consumer manifests. + +### Findings Ledger And Inherited Evidence +| ID | Finding / state | Required action in this phase | +| --- | --- | --- | +| DLR-01 | FIXED/DEPLOYED: processing backlog used to invalidate otherwise contiguous L2, freezing OKX BNB. Rust166cda1 + real resnapshot fixed frozen-book behavior. Snapshot200/200 eligible; delta195/200, NOT all-price FULL. | Inherit145Rust/40reader tests and frozen-book proof. Test only changed replay/recovery behavior; retain residual tail SLA as DLR-02. | +| DLR-02 | FIXED/DEPLOYED/BOUNDED_ACCEPTANCE: absolute batching deadline; RF3 replay4k/5k windows caught up. Original fault: hot raw partition/core backlog makes real Binance TRADE/BOOK too old. TS reads matched old Query data, while public Binance had newer trades. Core2 raw-age mean22.99s/max34.51s versus other cores~0.2s; later recovered to0.408s naturally. | Attribute exact affected product/raw partition -> core -> canonical -> cache -> consumer and correct limiting work, batching or fair scheduling. No assertion that CPU alone is causal. | +| DLR-03 | FIXED/DEPLOYED: existing quarantine feedback now restores four real-provider books after owner movement without manual ingestor restart (7830.93ms including5000ms stability). Original fault: cold-core L2 recovery needed manual OKX ingestor reconnect after rollout. | Prove or complete bounded resnapshot through existing ingestor/core ownership, real provider snapshot, same generation/sequence fences. No periodic fake snapshot or timer-based re-dating. | +| DLR-04 | FIXED/SCOPED: inverse missing acquisition/entitlement/native identity and hot-cadence alias mapping. Source0b90e8b/b394a36/1615eb2/7e28f02, real inverse+linear Risk30/30. | Preserve additive source/runtime bindings. Recheck affected readiness after any rollout, not replay all product/domain certification. | +| DLR-05 | TS-OWNED/FIXED, not a Data Layer patch: expired price keys plus ahead execution watermark caused bootstrap/replay loop. TS56dae41 market-only rollout3b4ee780,171source+171packaged PASS. | Inherit tests; consume actual candidate recovery evidence. Do not widen Data Layer scope or change TS adapter again without a reproduced new defect. | +| DLR-06 | RELEASE_SET_SEALED: twoQuerysecurityhandoff complete, ten legacycontainers and nineunusedimages retired. Explicitmixed-role digests/rollback and affectedcertificate recorded. Publication is proven by v2.2.1 tag/workflow plus externalfinalreceipt, not this build-time text. | Reconcile role/config/certificates, affected packaging, release procedure and exact cleanup manifest. | +| DLR-07 | Known test-only issue: ignored provider-admission Redis test used50,000ns cooldown rounded to1ms TTL, racing a network trip. Admission runtime was unchanged. | If retained as a required affected gate, fix deterministic test timing only; never change provider cooldown to satisfy test. Otherwise explicitly inherit unaffected admission scope, do not count ignored test as PASS. | + +Potential batching defect needing reproduction, not yet a proven incident cause: +qdl-realtime-core's collection loop resets batch_wait_ms per record; with +batch_size256/wait25ms, sustained spaced arrivals can keep a batch open well +beyond25ms. Inspect intended contract and reproduce actual elapsed batch age +before editing. A larger batch without an absolute deadline/fairness analysis +could worsen latency. This is part of DLR-02, not another phase. + +Explicit exclusions: Binance3d history (owner excluded), DNSE/VN V1 migration, +new symbols/products, strategy logic/order paths, TS E03 partial-fill matching, +Portal, Kafka topology/offset reset, Redis flush/SQLite deletion, whole-catalogue +299-product recertification and unrelated warmup/reference redesign. +Existing MARK/INDEX rejection diagnostics/realm-key fixes are inherited. If a new +MARK/INDEX failure appears, capture the exact rejected view and localize it; +do not call it a fresh defect merely from a generic error label. + +### Single Execution Sequence And Stop Boundaries +These are work steps INSIDE this one phase, not independently added phases. + +1. Freeze runtime and fault matrix before edits. + Save full role digests, config hashes/revisions, aliases, identities, quotas, + active subscriptions, per-role rollback and current workload. Reuse the + observed 2026-09-30 fault window; capture only missing bounded attribution. + Measure received/raw/canonical/served timestamps plus partition offsets and + consumer completed apply. Preserve event timestamps and record sampling clock. + Freeze declared freshness/latency/resource/recovery budgets from current + bindings and KN budget. An unspecified or unmeasured budget is UNVERIFIED, + not permission to invent a passing threshold. + +2. Repair shared owning boundaries only. + Reproduce DLR-02 using captured provider bytes and existing pipeline components + in a disposable namespace; isolate batch collection, normalization, commit and + downstream consumption costs. Check absolute flush deadlines, transaction + amortization, partition skew and bounded in-flight work. Prefer existing + shared Rust implementation; no Python bypass, dropped trades, new symbol + workers, new broker or invented control service. + DLR-03 must recover a cold/lost book owner through the existing acquisition + mechanisms with deduplicated bounded requests and fresh authentic snapshots. + Preserve checksums, real gap handling, leader/epoch fencing and idempotency. + If quota limits remain dominant AFTER code/config optimization, change only + measured affected role caps with before/after throughput and memory headroom; + no blanket doubling and no buffers that hide growing backlog. + Stop source changes when reproduced failures and targeted regressions pass. + +3. One affected candidate build and bounded runtime packet. + Build one immutable image per changed artifact family; reuse it across that + family's affected roles. Config-only fixes do not demand rebuilds. Compare + effective environment/mounts semantically, not raw Env-list order. + Packet must name roles, full before/after digests, config paths/hashes, + expected reconnect duration, state invariants and exact rollback commands. + Start with the actual fault owner. If shared core binary changes, name all + three core roles. Include ingestor(s) ONLY where the implemented recovery or + startup dependency requires them. Query/Stream/projectors/BAR/V1 stay unchanged + unless attribution proves an affected dependency and packet states why. + No shared TS recreation or alpha/order activation in this phase. + +4. Targeted matrix, then ONE final consumer acceptance. + Run the tests below first. No C2 loop to discover bugs. Once they pass, one + bounded300s affected no-order consumer acceptance on both Query replicas with + current identities. Observe existing TS sandbox, preserving shared TS as + noninterference guard; no strategy/order writes. If this final run fails, + save exact typed view/offsets, name failed predicate and stop promotion. + Fix that predicate, inherit unaffected results; explicitly document any + replacement run rather than silently retrying until green. + +5. Certify the affected delta, retire and hand back TS. + Link retained certificates + new affected evidence to the deployed release + set. Follow feature -> dev CI -> main/tag through approved release procedure; + do not publish a tag solely because runtime is healthy. Freeze release version + at this step rather than assume existing v2.2.0 covers new inverse scope. + After acceptance, remove explicitly inventoried retired containers/test images + and unused task build cache. Keep V1 and the named immediately preceding KN + rollback set with config/TLS/state references. Never delete market history, + runtime directories, volumes or owner source as cleanup. + Finish with exact TS prerequisites ready/not-ready and resume E only after + owner's separate instruction; no automatic broker orders. + +### Targeted Tests And Evidence Gates +| Gate | Cases / evidence | Pass condition and what is NOT claimed | +| --- | --- | --- | +| R1 runtime routing | Both Query aliases, streamA/B, shared paper and candidate sandbox identity/revision; active writer count; source/config hashes | No obsolete serving alias/second writer; correct entitlement and provenance. Inventory is not latency proof. | +| R2 batch/replay correctness | Fast burst, continuously spaced input, idle flush, shutdown/cancel, failed/ambiguous commit, replay/duplicates, two busy partitions plus quiet partition; record-before/after ordering | Bounded absolute batch age; no unexplained loss/duplication/early ACK; transaction and per-partition order preserved. Test only affected reducer/batch modules. | +| R3 L2 recovery | Binance/OKX, current BTC/DOGE failures + OKX BNB/inverse inherited boundary; core restart/owner move, snapshot delayed/duplicate, valid sequence jumps, genuine gap, stale epoch, repeated resync | Real snapshot restores verified state within frozen budget; no permanent awaiting-snapshot; failed quality remains blocked. Inherit unchanged full10book contracts. | +| R4 throughput/catch-up | Capture-based isolated replay at4k/5k TOTAL ingress/s, sustained window, burst/drain, hot partition skew, slow consumer and restart | Report offered/admitted/committed/applied counts, bytes, per-partition lag and recovery; no indefinitely growing queue at claimed load. Distinguish total ingress from per-consumer fanout. No production load flood. | +| R5 serving and exact rejection | Affected TRADE/QUOTE/BOOK plus unchanged MARK/INDEX sentinel; both replicas; freshness/session/gap/generation/hash; quiet trade negative | Every rejection attributed to that exact view. Quiet last trade can correctly be ineligible; pipeline delay cannot be excused as quiet. No event-age SLA widening. | +| R6 actual consumer |300s on existing sandbox22 active demand including inverse;64entitlement products independently checked by R5; shared paper60 is guard, not silently replaced; no-order | Auth/manifest/cursor/generation correct; replay converges; no stale execution accepted, no silent missing demanded route. Separate session availability from price eligibility. | +| R7 packaging/rollback | Changed binaries in immutable image, exact role config, affected rollout/return | New runtime maps to tested code; rollback state/cursors are compatible. No whole-catalogue rerun for unchanged packaging. | + +All latency in milliseconds with sample count and rejection denominator: +SDK/alpha request start BEFORE local queue -> validated usable return; +provider-event -> consumer completed Redis apply (not callback or stored quality); +recovery time after burst/disconnect; separate queue/service/commit contributions. +Keep failures in distributions/accounting; distinguish attempted/successful/usable. +n<100: observed min/max/range, no p99. Do not extrapolate observed short recovery +to indefinite5k/s or50-alpha capacity. Collect full-stack CPU/RSS/throttling/lag/ +I/O with per-role headroom, not a read-plane-only CPU number. +The knownTS150+ consumer regressions, E04 matrix, already certified reference, +BAR/universe and identity negative suites are inherited unless actually changed. + +### Cleanup, Rollback And Closure +Pre/post disk measurements plus exact image/container list; never claim concurrent +disk growth/shrink is cleanup savings. Remove only unreferenced task artifacts. +Old service candidates eligible AFTER new acceptance: query_v2_1/2, +stream_v2_active/passive, projector_v2 and projector_v2_2..6 (all stopped). +Verify mounts, labels and retained rollback reproducibility before removal. +Do not remove init jobs, Kafka, Redis, V1, TS shared/candidate, Portal, alpha +resources, volumes or stopped operational evidence containers by wildcard. +A rollback set can have different images by role; "one rollback" does not mean +one image can run unrelated binaries. The base7fe348 image is active in ingestors +and KN projectors, so it is NOT garbage even though newer core image exists. +Restore only affected role/image/config; no offset reset/cache flush. If rollback +would violate an advanced cursor/config fence, fail closed and use the pretested +compatible rollback path, never force an older schema onto new state. + +**Exit:** R1-R7 applicable gates pass with inherited evidence explicitly mapped; +no open DLR-02/03 execution/recovery defect; no falsely certified ignored tests; +new release-set receipt and exact cleanup complete. If an acceptance/resource +limit remains, report it as a failed requirement, not relabel it technical debt. +**Stop:** do not proceed to TS lifecycle/E09 or broaden scope when this exit is +unmet. Report fixed/open lists, runtime map, per-binding latency and cleanup. + +Audit receipt: ~/.local/state/qdl-v2/execution-readiness-audit-20260930/runtime-inventory.json +SHA256 9ab4d3d1814f203e88dd25f848dea4a962bc11b400dd52637f9033a721fb13fd. +Actual DNS from ts-p183 market process resolves Query to172.18.0.12/25 +(KN1/2), StreamA/B to172.18.0.26/27 (KN1/2), matching Docker addresses. +No legacy V2 endpoint resolved in this audit. No gate rerun, image build, +runtime mutation or cleanup performed. No TS goal resumed. Planning only; +phase exit requires subsequent implementation/evidence, not this audit commit. + +### Implementation Journal - Batch Deadline Reproduction (2026-09-30) +Owner goal explicitly approved this one phase. R1 inventory and prior DLR-01/04/05 +evidence are inherited; no catalogue/TS order gate is repeated. First source slice: +reproduce the rolling per-record timeout in qdl-realtime-core, then use one absolute +batch deadline and retain bounded cancellation, ordered input and transactional ACK. +Add bounded per-partition raw-age and collect/normalize/commit timing to distinguish +queue from compute/transaction pressure. No claim that the timeout explains the +entire previously observed 23s backlog: full batches may hit a separate throughput +limit. Tests use virtual time on the actual receive helper; synthetic unit fixtures +are not provider/capacity evidence. Builder uses existing image, bounded resources, +no production network/mounts. L2 restart/resync remains open and is not fixed by a +batch timeout. Runtime/config/freshness/provider quotas remain unchanged. +apply_patch helper failed (bubblewrap mountinfo); exact-match edit fallback used +under workspace rule32; owner plan hunks preserved. + +Batch counterfactual executed in qdl-rust-builder:r134-test551428d98db6, +network none,2CPU/3GiB/2jobs, disposable container and task-private target dir. +Extracted legacy25ms per-record algorithm:5cases,2PASS/3FAIL as expected +(continuous arrivals, idle remaining budget, expired ready receive). Not a +claim of whole-runtime baseline replay. Evidence: execution-readiness-20260930/ +evidence/batch-counterfactual.log. Candidate uses absolute deadline with explicit +expired-before-poll guard; cancellation drops only uncommitted input, to be +replayed from Kafka on restart (no ACK). Transaction code remains unchanged. +DLR-03 source localization: RealtimeCore adapter.request_resync changes only the +local state machine; native OKX ingestion has no downstream request listener. +Thus initial-snapshot-only sockets cannot know a cold core needs a fresh anchor. +No repeated provider timer or re-dated snapshot is an acceptable substitute. +Feedback implementation and autonomous live recovery remain OPEN. + +Batch slice result:10/10 qdl-kafka binary unit tests PASS (5 new deadline cases), +0fail/0ignored; clippy --offline --locked -p qdl-kafka --bin qdl-realtime-core +-- -D warnings PASS. Rustfmt scoped to the changed binary; git diff --check +for rust/qdl-kafka clean. No lockfile/dependency version change; test-util is +dev-only. The5 prior binary tests were rerun because that executable changed; +no catalogue/SDK/TS suites or C2 rerun. No claim of throughput/recovery pass. +Test containers used --rm and are removed; no new image built. Task target/cache +is retained only while this phase is active, to avoid repeated full compiles; +final scoped cleanup remains after acceptance. Production role digests unchanged. + +### L2 Autonomous Recovery Wiring - Design Before Source (2026-09-30) +DLR-03 will use the existing core -> transactional quarantine -> native ingestor +boundary, not a new service/topic, shared-file authority, or periodic fake book. +Core opt-in emits a versioned resnapshot marker only for its current non-ready +L2 binding, rate-bounded by existing snapshot_refresh_seconds and original raw +receipt time. The marker carries the original envelope and integrity evidence; +it commits atomically with consumed raw offsets. Ingestor opt-in listens on the +existing quarantine topic with a dedicated group per existing BOOK connection +lane, not per symbol. Only an exact current provider/binding/config/authority/ +lease/session/generation match can reconnect that BOOK lane; other feeds stay up. +Expired, duplicate, foreign or superseded markers cannot trigger reconnect. +Reconnect obtains authentic provider WS/REST bootstrap using the existing flow. +Control offsets are independent of canonical/raw groups. Activation requires +explicit packet listing additive read/group ACL, two optional config fields and +affected core/ingestor digests. No production ACL or config is changed by this +source step. Default absent config preserves current behavior. Rollback disables +feedback and restores only affected binaries/config/ACL. Rejection/failure remains +fail-closed. Required tests: cold owner, gap, ready suppression, request retry, +stale session/generation/identity/revision, expired/tampered marker, cancellation, +then isolated real Kafka committed-only request delivery and real resnapshot. + +L2 feedback source result (NOT runtime recovery certification):13/13 affected +qdl-realtime-core L2 tests PASS; qdl-kafka library20/20, native ingestor25/25, +realtime binary10/10 PASS; clippy for the library and both changed binaries +-D warnings PASS. New tests cover cold owner/retry/session fence, real gap +rearming after a ready book, both-venue exact identity/fence/tamper/expiry +rejection and disabled feedback. Initial compile caught a missing kafka_config +lane argument in the new helper; corrected before test PASS. Counterfactual +batch evidence retained separately. Unit fixtures are explicitly test evidence, +not authentic provider certification. Existing protocol tests were rerun only +for the changed core/ingestor binaries; no full-catalogue/C2/TS order tests. + +Control receiver uses read_committed/manual checkpoint and a dedicated group +starting latest only when new; existing offsets resume. It handles receive/ +checkpoint errors with bounded1..30s backoff separately, rather than aborting +all ingest lanes. New core/ingestor config defaults disabled. Production remains +unchanged. Still required before enablement: isolated committed-only feedback, +restart/rebalance resnapshot proof and feedback-vs-market-data fairness under +quarantine pressure, capture replay throughput, exact role/ACL packet, final +consumer acceptance. No FULL/production-ready claim from these source tests. + +Pre-commit source review found synchronous control-offset checkpoint could +block the BOOK socket poll. Control hints now use explicitly named asynchronous +checkpoint submission; this is NOT a durable market-data acknowledgement. +Lost/duplicate hint checkpoints are harmless through bounded core reissue and +exact session fencing. Existing market-data checkpoint stays synchronous and +unchanged. No detached task, unbounded queue or worker-per-message added. +Clippy rerun after the checkpoint-hint change PASS; earlier pure identity/L2 +unit results are inherited unchanged. The new Kafka offset submission behavior +requires isolated broker integration before runtime enablement. Source compiles +for both changed binaries (debug build); no immutable release image built yet. + +### Isolated Recovery Packet (2026-09-30, Before Start) +Previous goal turn classified PROGRESS:62d50f8 batch correction and355f3d3 +opt-in L2 feedback, targeted tests recorded; no running test handle to resume. +Next packet ONLY creates task-owned qdl-readiness-* containers/network: one +Kafka broker (existing9516fb image,1CPU/1536MiB, ephemeral log), one Rust core +(1CPU/384MiB), one ingestor per venue (0.5CPU/256MiB each), bounded client +(0.5CPU/256MiB). Core/ingestors use existing Rust base7fe348 with candidate +binaries mounted read-only; this is an isolated test, NOT immutable release. +Fresh disposable TLS, no production secrets/mounts/network, no host ports. +Kafka topics/groups/authority are task-local. Native acquisition uses approved +public provider URLs for existing BTC/BNB books plus TRADE/QUOTE sentinels; +no private/broker order API. Read-only production JSON is copied then narrowed +into task-local config, never modified in place. Stop/restart only isolated +core to reproduce cold state with committed offsets; observe real resnapshot +and non-BOOK lane continuity. Abort at bounded180s per scenario or provider +rate-limit. Capture bounded logs/results then remove exact task containers, +network and temporary keys; keep evidence and build cache until phase exit. +No use of old kn4 namespaces or production volume, no Kafka/Redis resets. +Resource baseline:31,379MiB host RAM,14,825MiB available; root74GiB free. + +Isolated run1 failed before application startup: broker readiness failed. +Harness generated private TLS as hostUID1001 but Kafka runsUID1000; private +directory/file permissions prevented access. No recovery/capacity result. +The auto-removed broker also lost its exit log; corrected harness retains +owned test containers until logs/state are archived, then removes them. +Run2 changes only task-local broker TLS ownership to1000, restores host ownership +for scoped key cleanup afterwards. First receipt retained at recovery-real/. +No source/runtime change, no production restart. Test network removed. + +Run2 exposed a real defect in NEW opt-in feedback: Binance requested recovery +from a pre-REST-anchor delta, then acted on that hint after publishing a fresh +REST snapshot. Ten reconnects in about13s; TRADE/QUOTE sessions stayed unchanged. +OKX recovered, but the harness's single READY observation falsely printed PASS. +That receipt is retained INVALIDATED, not acceptance. No production enablement. +Fix: remember successful queued authentic Binance snapshot receipt per binding/ +session/generation; ignore only hints whose original frame is covered by that +anchor. Newer requests and wrong-session anchors cannot be suppressed. Actual +core validation still decides readiness; failed publish follows existing retry/ +fencing. No timestamp rewrite or silent data drop. Add race regression. Harness +now requires stable LIVE session generations and READY for a bounded window, +plus no cross-feed reconnect. Run1 TLS ownership mismatch remains an identified +setup defect; missing first broker exit log prevents a stronger causal claim. + +### Recovery Anchor Race - Isolated Result (2026-09-30) +Source regression `cargo test --offline --locked -p qdl-kafka --bin +qdl-native-raw-ingestor resync`:5 PASS,21 filtered; clippy same binary +-D warnings PASS. Candidate debug binary rebuilt; no new Docker image. +Isolated real-provider run3 PASS for BTC/BNB books on Binance USD-M and OKX Swap: +all4 core views READY, all6 acquisition sessions LIVE and stable for5s; +cold-core restart preserving test Kafka offsets caused exactly one BOOK reconnect +per venue (generation1->2), while TRADE/QUOTE retained generation1/session. +Observed cold recovery + stability window8447.64ms, NOT request-to-usable latency +or a production SLA certificate. State rechecked after another5s; no reconnect +loop. All4 test containers OOMfalse/restarts0. Kafka startup included benign +TOPIC_ALREADY_EXISTS races for __transaction_state; no application panic/error. +Run2's transient-READY false PASS remains invalidated, not overwritten. + +Evidence outside Git: execution-readiness-20260930/recovery-real-r3/receipt.json, +SHA25648d064eec940517bc851310845eb70ffcb17070c7aabba47cc93772fc65f6414. +Task orchestration recovery_probe.py and source/binary hashes retained there. +Exact task containers/network and temporary private keys removed; label inventory +now0 containers/0 networks. No volume/image broad prune; active build cache kept +for unfinished phase. Production images/config/ACL and TS remain unchanged. +apply_patch helper still fails with mountinfo error; exact append used per rule32. + +This closes the reproduced pre-anchor reconnect race, NOT all R3/R4 gates. +Remaining before enablement: control-lane fairness under quarantine pressure, +committed-only/aborted hint delivery and owner-move proof; captured-provider +4k/5k total-ingress catch-up profiling; affected role packet and final consumer +acceptance. Current biased socket select puts feedback first, so pressure +fairness requires verification/correction, not an assumption from idle recovery. +Do not rerun unrelated catalogue or passed TS domain tests. DLR-02/03/06 remain +open overall. Canonical feature remains fix/okx-inverse-sandbox-readiness; +stable main8299838/v2.2.0; no push/merge/release or production recreation. + +### Feedback Fairness And Committed-Only Proof - Before Edit (2026-09-30) +Previous continuation made progress:92e82aa and real recovery run3, no active +process left. Same R3 scope: remove biased priority from the two BOOK socket +selects so sustained control backlog cannot deterministically starve market +read/publish/timer work. Use Tokio fair polling, no extra queue or delay policy. +Add explicit ignored-by-default real-Kafka test for new-live group, aborted +hint exclusion, checkpoint/resume and cancelled receive. Invocation must require +qdl-readiness-kafka isolated hostname and task topic; never production config. +Run this explicitly in the existing isolated packet and record actual result; +ignored default is not a PASS. Then bounded malformed control-pressure input +(test-only provenance) with real acquisition verifies non-BOOK continuity and +ongoing BOOK processing. No gate or provider quota is relaxed. + +Control proof run4 failed before acquisition: test waited for Kafka consumer +position() to become numeric without receiving a message; this is not a valid +startup barrier. Correct test to fresh-marker handshake with15s bound and drain +only generated handshake markers. Historical/aborted payload assertions remain +strict. No passing claim for run4. Its broker/test resources and keys cleaned. + +Control run5: real Kafka committed-only/new-live/resume/cancel test PASS1/1. +Pressure40,000 malformed TEST_ONLY hints acknowledged40,000 in10,784.60ms; +isolated Binance ingestor OOM at256MiB (exit137), so overall run FAIL before +cold restart. No production mutation. Suspect unbounded async commit requests: +current listener calls checkpoint_hint for EVERY rejected control record. +Fix control-only checkpoint coalescing to at most1 submission/s per listener; +this is a delivery-hint offset cadence, NOT a market-data ack/freshness/retry +policy. Last locally stored offset still advances on consumed hints. Lost hint +checkpoint remains safe through session fencing and bounded reissue. Confirm +with virtual-time scheduling regression and identical isolated pressure, plus +memory sampling. Do not claim OOM root cause proven until counterfactual evidence. +Do not increase ingestor RAM to hide control-plane queue growth. + +Run6 after bounded control checkpoint: same40,000 TEST_ONLY hints,40,000 ACK, +0 publish error,10,159.68ms offered window. No OOM/restart. Peak cgroup memory +Binance17,678,336B (16.86MiB), OKX16,011,264B (15.27MiB), core21,942,272B; +core progressed4,326 raw records across the pressure/stability window, all six +sessions made transport progress with unchanged identities. This before/after +supports async checkpoint flooding as the run5 memory defect; no RAM increase. +Regression resync6/6 PASS including40k calls coalesced to1 submission/s. + +Run6 raw receipt FALSE is retained: harness incorrectly demanded a reconnect +from BOTH venues. Cold core actually recovered4/4 READY: Binance received two +real periodic REST bootstraps (BOOTSTRAP_APPLIED with concrete snapshot_sequence, +then DELTA_APPLIED), OKX one feedback reconnect and two real SNAPSHOT_APPLIED. +All six sessions LIVE; non-BOOK identities unchanged. Core recovery +5s stable +window8,131.68ms. Evidence reviewed without another lucky retry; review.json +explicitly certifies isolated pressure/autonomous recovery only, not end consumer +or concurrent-owner movement or raw4k/5k capacity. Original receipt/logs unchanged. +Run5 committed-only/resume/cancel proof inherited unchanged (library source adds +test only). No synthetic market event was published; noise was explicitly +malformed TEST_ONLY control input, rejected by the ingestor. + +Evidence: execution-readiness-20260930/recovery-real-r5/control-test.log and +recovery-real-r6/{receipt.json,review.json,qdl-readiness-core.log,*-state.json}. +Exact isolated containers/network/private keys cleaned, no production changes. +Candidate source still opt-in; no runtime feedback activation or release claim. + +Fairness/checkpoint slice clippy(lib + ingestor + test targets) -D warnings PASS. +All task containers absent after run6; no pending process. Coherent commit +includes only shared ingestor correction, actual Kafka regression and journal; +owner's two pre-existing plan hunks remain unstaged. + +### Bounded Raw Capture Packet - Before Read (2026-09-30) +R4 input will be captured committed production raw provider envelopes, preserving +key/payload/headers/source partition+offset+timestamp, never rewriting event age +or making synthetic frames pass as market data. One temporary read-only client, +existing phase8-consumer TLS mount read-only, approved qdl-c40-handoff- audit +namespace, explicit assign/no group join/no auto offset store/no commit; no +producer or admin API. Read md.raw.stable.v1 only, freeze highwater bounds, +max100k records/partition,384MiB payload and180s wall deadline,4MiB/s pacing; +client cap0.5CPU/256MiB, no production config/service change. Capture outsideGit +with bounded compressed file + checksum and per-feed counts. Remove client on +exit. This is source data for isolated replay, not a live freshness certificate. +No broker/offset reset, no provider calls and no spool read. If permission/capture +fails, retain exact error, do not silently change principal/ACL or invent input. + +Capture setup correction: first audit read used retired md.raw.stable.v1 and +returned0 records (retention low==high), not useful replay evidence. Rechecked +actual mounted core/ingestor JSON: all active raw paths are md.raw.realtime.v2. +This was a harness topic-selection error, NOT production routing duplication. +Preserve empty receipt; repeat bounded read on exact active topic with the same +read-only safeguards, require nonzero rows and explicitly report partial bounds. +No ACL/principal change. Production offsets/runtime untouched. +Optimized core/ingestor binaries building from fbbe34e (2CPU/3GiB,offline locked) +for actual replay measurements; debug recovery numbers are not capacity claims. + +Active raw capture with phase8-consumer was denied TOPIC_AUTHORIZATION_FAILED; +no records read, no ACL changed. Scope-correct identity is phase8-core (the +active raw reader). Explicit capture packet amendment: mount only existing +core TLS read-only and use separate qdl-v2-production-core-r1-readiness-capture- +20260930 group under the already defined core audit prefix. Same assign-only, +no subscription/join/offset store/commit, no producer/admin API. No use of active +core group and no ACL grant. Keep both failed receipts. If existing permission +also denies this exact reader, do not grant privileges as an implicit fallback. + +### Optimized Artifacts And Replay Input Ready (2026-09-30) +Core-reader identity capture succeeded with existing ACL, no permission changes: +600,000 committed raw records (100k/partition,6 partitions),400,331,336 payload +bytes, frozen bounds completely read in95.42s. Providers: Binance283,825, +OKX316,175;150 provider/symbol/channel combinations. Includes154,830 TRADE, +177,743 QUOTE,124,391 BOOK and real MARK/INDEX/BAR frames. Compressed capture +SHA256d23a735a2942c847d3b601e889b2771447302c34e51ea55a4048487b9be03d06; +payload sequence SHA256bc7338e806b817c8f95f58f06e2fcdaef2b91d91b8a185b03e4dea55d7432cb2. +Path execution-readiness-20260930/evidence/raw-realtime-core-capture.jsonl.gz; +receipt alongside records bounds/feed counts. No group join/offset commit or +production writes. The two earlier empty/denied receipts remain explicitly +non-evidence. Do not confuse captured historical event age with transport latency +when replaying; original bytes/timestamps must remain intact. + +Optimized build from source fbbe34e completed in4m08s, offline/locked,2CPU/3GiB; +not an immutable Docker release image yet. SHA256: +- core:6788cdd21f12dca96c4af3e4ec4ccf76036bfa57a8a6a0215be51498e7e160aa +- ingestor:f7d2fd17f4ac8ddacd82d3f2e34c07af7c5da293fbcb44a97e20251bdcd5e2b8 +Private target retained only for current phase. No test/capture/build container +or isolated task network remains (name/label inventory empty). No active/rollback +image pruned and no new runtime image built. Production core caps confirmed +1/1/0.5CPU,256MiB each,OOMfalse/restart0; unchanged, not a capacity certificate. +Canonical stays /home/bobby/data_layer on fix/okx-inverse-sandbox-readiness; +main8299838/v2.2.0 and all active role digests remain as R1 inventory. No new +worktree, push, merge, TS lifecycle or production recreation. + +R4 replay measurements, concurrent owner movement, affected rollout/final +consumer acceptance and release/retirement are still pending. R3 pressure and +cold recovery evidence above is scoped, not a claim that the whole phase closed. + +### R4 First Replay Packet (2026-09-30, Before Start) +Previous turn PROGRESS (fbbe34e/e807447, tested recovery and authentic capture). +No live handle to resume. Reuse the same task-only broker/TLS orchestration, +optimized fbbe34e core, three isolated owners with distinct transactional IDs, +same shared group, caps matching production1/1/0.5CPU and256MiB. RF1 test broker +cap1CPU/2GiB, ephemeral1GiB Kafka log; this is NOT RF3/HA or production performance +certification. No production mounts/secrets/network; copy public runtime JSON +then change only shadow authority/topic/transaction names. Retain native provider +bytes, keys, source partition/offset headers and original timestamps. Replay +warmup20k@2k/s,120k@4k/s,150k@5k/s,40k@1k/s followed by bounded180s drain. +One1CPU/256MiB client records offered/delivered rates and observer reads group +committed offsets/highwaters once per second without committing. No market-data +fabrication or event-time rewrite. Bound entire packet to10min and cleanup. +Measure collect/normalize/commit, offered vs actual, lag slopes/catch-up, CPU/memory, +output counts/rejections. Missing initial L2 snapshot in a historical capture +must be reported, not passed as a complete execution-book workload. R3 authentic +bootstrap proof is separate. No C2 until attribution/affected tests are clean. + +R4 run1 supplied330k/330k records, actual4,001/s and5,001/s across30s windows; +no producer failure, cores peak36/42/31MiB, no scope fencing rejection. However +observer exited on transient NOT_COORDINATOR during new broker startup, recorded +zero lag samples. Do NOT certify catch-up from this run. Correct harness startup +barrier before load and bounded coordinator-error recovery(15s), fail promptly +if observer later fails; record send-to-durable-offset lag age using separate +replay wallclock, without touching original market timestamps. Save quarantine +reason/subscription and final stopped counters before task teardown. Replacement +run2 is required only for invalidated measurement, no source/runtime change. +Historical book boundary/expired MARK components remain explicitly non-live +inputs; raw processing throughput is not an execution eligibility certificate. + +R4 RF1 run2 profile completed:330,000 delivered ==330,000 summed committed +processed counters; final drain500ms observer resolution,0 producer/observer +error, no OOM/restart or scope fencing rejection. 4k window lag peak680 records, +oldest uncommitted max169.43ms (30 samples);5k peak2666,oldest max834.76ms +(29 samples); recovery1k ended40 lag then drained. These are sampled maxima, +NOT p99 and NOT consumer usable latency. Startup max2453ms is separate. Three +core peaks41.12/35.89/30.54MiB. Five SequenceGap quarantine records concern dated +Binance BTC/ETH261225 books in the truncated historical capture; no raw-level +proof of a fresh production defect. Full reason/subscription evidence retained; +missing bootstrap/component-expiry in historical input is not live eligibility. + +Do not promote from RF1: original incident commit cost came from production +RF3/minISR2. Next affected replay keeps same capture/CPU/binary but uses three +isolated brokers, RF3/minISR2, SSL inter-broker, durable files under new task-only +throughput-rf3-r1/data-{1,2,3}. Existing broker image, fresh TLS SANs for test +hosts only; cap1CPU/1536MiB each,256MiB Java heap. No production mount/network. +Core caps unchanged. Topic/transaction/group replication3, data topic minISR2. +Temporary logs bounded by the330k input; remove only these task-created broker +data directories after evidence and stopped containers, not any shared volume. +This remains same-host rehearsal, not independent failure domains/production HA. +Observer/clock corrections inherited. No new source or repeat of unit/domain gates. + +R3 owner-move packet prepared (run after RF3 namespace is cleaned, not concurrent): +reuse authentic BTC/BNB Binance/OKX recovery harness with optimized same binary, +initial one core then add a second owner under same test group/distinct transaction +ID; observe second owner reach READY for at least one assigned book, then stop +only first test core and require survivor READY for all4 books with stable live +sessions. Preserve test group offsets, no manual ingestor restart. Non-BOOK lane +sessions must remain identical. Existing4-book cold/retry/pressure evidence is +inherited; this covers the missing cooperative-rebalance/owner-move behavior. +No production Kafka/consumer mutation. Bound each readiness wait90s and cleanup. + +RF3/minISR2 run1:330,000/330,000 delivered and summed committed processed; +0 scope rejection, no OOM/restart, bounded drain1000ms sampling resolution. +4k window oldest-uncommitted maximum2714.76ms includes remaining startup backlog; +5k maximum2381.02ms,peak6663 lag/end4789; recovery1k end80 then drains. +Do NOT claim all execution freshness at5k. Five identical dated-book capture +quarantines now carry original raw hash/receipt/generation for attribution. +Receipt SHA256864b9b8f2dca65c7ea076174164da48c38d34159a9f32fc7d5d02e16cf591dfc. + +Authentic cooperative owner move PASS: second core reached READY on newly owned +Binance BTC; after first test owner stopped, survivor restored all4 Binance/OKX +BTC/BNB books without manual ingestor restart in7830.93ms INCLUDING5s stability +window. Both BOOK lanes advanced1->3 for join and leave recovery; all TRADE/QUOTE +sessions stayed generation1/same identity. Evidence owner-move-r1/receipt.json. +This closes the scoped concurrent-owner behavior in R3; no full consumer claim. +Task containers/network/private keys removed. No production mutation. + +Profiling correction before quota decision: RF3 test broker caps1/1/1CPU were +below actual production1.25/1.75/1.75CPU; normalize and commit delay cannot yet +be blamed solely on core quota. One matched-cap run uses current broker caps +and memory1.5/2/1.5GiB, same256MiB heap/durable isolated storage/RF3. Core caps +stay1/1/0.5CPU,256MiB; explicitly enable candidate book_resync_quarantine to +include its hot-loop/transaction overhead. No production quota change or RAM +increase. This is affected performance profiling, not rerunning passed domain +or299-product gates. Exact same capture and observer reused. + +### Matched RF3 Profile - Reviewed Result (2026-09-30) +Exact production broker caps,RF3/minISR2,SSL replication,durable isolated storage, +feedback enabled, core caps1/1/0.5CPU unchanged:330k delivered, all six committed +end offsets equal producer end offsets. No producer/observer errors,OOM or +container restart. Core2 did a supervised generation1->2 on initial group join +(Invalid group generation in transaction, aborted/retried). Therefore summing +ONLY terminal per-generation counters gives328720, not330000; that shortcut is +invalid. Final committed-offset coverage proves drain; do not claim a matching +lifetime counter or silently hide the supervised retry. First-generation1280 +contribution is inferred from coverage, not independently logged as a counter. + +Sampled maxima,NOT p99:4k/s window27 samples,peak2042 lag,oldest uncommitted750.96ms; +5k/s29 samples,peak3251 lag,oldest1315.15ms;recovery1k39 samples,max703.04ms, +then drain1500ms at1s observer cadence. Startup max4863.75ms separate. Core memory +peaks32.43/49.84/44.24MiB. Bounded input/core throughput/catch-up is demonstrated, +not request-to-consumer usable latency or live execution eligibility. Historical +missing L2 anchors/expired MARK components and quarterly gap records stay named; +feedback hints emitted during replay are not new provider-error counts. + +Same-source R3 owner movement restores four real books; R4 profile now supports +preparing affected image/packet. Standard committed Dockerfile.qdl-rust-runtime +only, one Rust image family for three cores+two native ingestors. Freeze source +and compare packaged binary provenance before any runtime action. Other roles +remain untouched. Existing authority revision1/candidate digest7fe348 is the +original acceptance anchor, not current executable (core already runsf204); +record actual runtime image hashes separately, do not silently rewrite authority +history or claim its anchor equals deployed binary. No new authority promotion. +All task containers/networks/private keys/RF3 test data directories removed. +No production change, push/merge/release or resource-cap change. Final affected +consumer acceptance and retirement still required. + +### Candidate Packaging And Minimum Feedback Wiring (2026-09-30) +Immutable candidate build uses committed Dockerfile.qdl-rust-runtime, source +0e00c817289ce32ea767cea002c859814349fc72, qdl-v2-rust:2.2.1-0e00c81. +Git archive excludes both owner plan edits. Single Cargo build job; no new +production resource caps. This is a candidate label, not a published release. + +Prepared packet helper is outside Git under execution-readiness-20260930; it +uses existing kn_production_packet service_from_inspect/external_compose helpers. +Only three cores and two native ingestors may change image/config. Preserve all +other fields, bindings, authority revision, mounts, TLS, quotas and offsets; +versioned new JSON avoids overwriting current config. Rollback uses actual +core f2040ac9... and ingestor7fe348... digests, not stale Compose labels. + +Read-only Kafka ACL inspection: md.quarantine.stable.v1 currently grants only +phase8-core WRITE/DESCRIBE. Recovery requires phase8-producer READ/DESCRIBE on +that exact existing topic and READ on dedicated qdl-v2-book-resync- group prefix. +Snapshot exact ACLs before applying; rollback removes only newly added entries +after stopping recovery clients. No new topic, raw/canonical group changes, +manual offset reset, history/cache mutation or authority promotion. + +At this checkpoint: no runtime or ACL changes; isolated test containers absent. +Host disk219GiB used/71GiB available, available RAM13428MiB during build. +These are inventory, not claimed cleanup savings. Build context/cache retained +until candidate attestation; release/old-container retirement remain pending. +apply_patch sandbox failed again; exact-match fallback used under workspaceRule32. + +Candidate build PASS: sha256:658a9570c5fc23f4906413aa2460f4d82e21772cc63c9ed900363d4d13d54023. +Packaged core50908e23ccccd731441ff4400360b775e5ec4e5bbaf16f954e0a277b92997578; +ingestor43358eeea2a57554e1a6d42285968409d279bc093f180d42f06dd41ea5fd71c5. +Both ldd checks resolve; network-disabled disposable attestation container removed. +Packaged hashes differ from separate optimized test build; source provenance is +same, not a claim of byte-identical artifacts. Affected runtime acceptance remains. +Prepared packet: releases/v2.2.1-0e00c81-recovery; candidate and rollback Compose +config --quiet both PASS. No apply/recreate/ACL change. Certificate subject read +confirms CN=phase8-producer; dedicated recovery group ACL query returns none. +Attestation evidence candidate-image-attestation.json SHA256 +c9ef6963ab78717ca0f0fa10fef78ea8cc04a4715822c4967006ea74aa2628b1. + +Cleanup: removed only23,361,111 logical bytes of disposable Git archive context. +Disk used235435982848 ->235409915904 bytes; concurrent activity means this delta +is NOT measured cleanup savings. No images deleted; active/rollback/candidate and +private build cache retained for active phase. Ten stopped legacy containers +reconfirmed, none running/removed. Production role digests/config unchanged. +Next: exact ACL delta, bounded five-role rollout, affected typed preflight and one +300s actual consumer acceptance, then release/retirement. TS work stays paused. + +### Five-Role Rollout And Consumer Scope Correction (2026-09-30) +Owner explicitly confirmed Binance ingestor recreate after tool safety rejection; +no bypass occurred. Exact feedback ACL applied: phase8-producer READ/DESCRIBE +md.quarantine.stable.v1, READ PREFIXED qdl-v2-book-resync-. Before/after stored +in feedback-acl-before.json / feedback-acl-apply.json. No offsets/topology changed. +Rolled sequentially Binance17:44:24Z,OKX17:45:02Z,core3 17:45:37Z,core2 17:46:22Z, +core1 17:46:54Z to658a9570...54023. Each showed startup and authentic book recovery. +Post-roll inventory: all five running,restart0,OOMfalse; every inventoried role +outside approved set retains exact image/start time. No Query/Stream/projector/ +BAR/V1/TS/alpha/order mutation. Compose orphan warning ignored; never remove-orphans. + +Important correction to R6 inventory: sandbox binding entitlement is64 products, +but its existing configured active demand is22, not64: Binance BTCUSDT/DOGEUSDT, +OKX BTC-USDT-SWAP/BTC-USD-SWAP. Fixed-worker heartbeat really aggregates22. +Shared paper runs60. Thus inherited plan wording conflated entitlement with +active subscriptions. Do not claim64 active sandbox routes or silently expand +TS config. Preflight must read all64 entitlement products x2 replicas with real +sandbox SDK identity; final observation separately reports actual paper60 and +sandbox22, with product union/overlap explicitly checked (60linear+4inverse). +Session READY is not execution-eligible: pre-roll sandbox22session/18execution. +This preserves64-product coverage and actual consumer proof, not fictitious +64-sandbox subscription evidence. No-order snapshot helper uses temporary +cursor/audit directory and never ACKs production cursors or writes market/order +state. Snapshot quality is captured before adapter eligibility validation; status +read after a failure is separately labelled, not called the same rejected view. + +Affected preflight complete:128/128 snapshots returned typed views;126/128 pass +execution conversion. Two exact rejected OKX trades: DOGE1358935530 ts1790790738429, +BNB239907276 ts1790790741311. Bounded official history-trades(type2,after=query +start timestamp,limit100) confirms each is the latest venue trade at its rejected +request timestamp. Thus these are valid last-trade age rejections, not guessed +quiet or missing canonical events. No relaxed3s policy. REST row quantity is not +claimed identical to aggregated WS quantity; this attribution proves ID/time only. +See trade-rejection-provider-attribution.json; official contract +https://www.okx.com/docs-v5/en/#public-data-rest-api-get-trades-history. +All QUOTE/MARK/BOOK/BAR preflight items usable on both replicas, no mapping error. +Post-rebalance60s logs: progress on all3 cores, no retry/error or recurring +ingestor resnapshot in that window. Not a long-soak claim. + +Proceed ONE final300s passive observation of existing paper60/sandbox22 consumers. +Observer executes only Redis SCAN/MGET in each existing consumer environment; +no SDK stream/ACK, market write, DB access, or order operation. Poll1s, capture +heartbeat (30s publication), actual execution cache visibility, exact lineage +generation/offset monotonicity, and host-visible resource samples. Repeated +heartbeat polls are NOT independent heartbeats. Event-to-observed-Redis timing +is a conservative upper bound with1s sampling, not exact Redis commit timestamp +and not interchangeable with request-to-usable. Initial cache observations and +unchanged rows are separated from newly observed watermarks. No fabricated +consumer throughput or silent substitution of callback timing. Tests/import +compatibility checked on both actual images before this final window. + +Observer correction inside SAME final window: paper uses EPHEMERAL_ONLY, sandbox +CORE_ONLY. Initial helper correctly read both heartbeat keys but incorrectly +looked for paper execution cache in core Redis (0rows). This is a measurement +error, not a production fault or proof of missing prices. Added a market-Redis +reader only for remaining original window; it sees50 paper execution keys while +sandbox sees19. Full300s heartbeat/worker coverage retained; paper cache latency +coverage is explicitly shorter and not fabricated as300s. No runtime source fix, +consumer restart, second acceptance, or quality-gate change. + +### Final Window Review And Release Boundary (2026-09-30) +ONE final observation completed302.25s host wall clock; both consumer observers +cover300s/301samples. Eleven distinct heartbeats each: paper60/60 and sandbox22/22 +READY at every publication. Heartbeat cadence30s does NOT prove uninterrupted +readiness between publications. Actual cache observations: sandbox19execution +products/301samples; corrected paper50products/177samples over176s. Union54 +execution products +10BAR verified in R5 =64 entitlement products. Zero observed +watermark regressions; no claim of per-event losslessness from polling snapshots. + +Do NOT report zero disconnects: at18:02:00-18:02:06Z OKX ingestor recorded4 native +transport disconnects (not feedback requests), and consumers logged12paper + +8sandbox SESSION_STATE refusals/reconnects. They subsequently resumed, retained +worker health and advanced caches/cursors. No scope/auth/manifest/quota or +Redis-write error appears in this bounded consumer log window. Underlying socket +close cause is not recorded by current native log, so do not blame venue/host +or call it a new proven code fault. This is observed fail-closed recovery, NOT +a zero-interruption uptime claim. No second C2 was run to hide these events. + +Cursor telemetry progressed paperACK1516714->1552299,commits874550->894920, +sandboxACK430067->443821,commits325125->336038; cursor errors0 on both. Paper +write.market.error cumulative1010 stayed1010 between the two available telemetry +samples; that is not a fabricated baseline for the entire300s. Pending at later +sample paper0/sandbox2, no overflow loop in the window logs. + +Snapshot SDK request -> validated usable observed maxima (n10-12 per feed/venue, +NOT p99): Binance/OKX QUOTE25.14/29.40ms; MARK26.64/18.84ms; TRADE15.09/15.12ms +(OKX8/10usable,2proven quiet rejections); BOOK_SNAPSHOT45.98/49.03ms; +BOOK_DELTA38.90/25.48ms; BAR20.69/37.83ms. Metadata resolve/startup is separate. +Redis polling timing remains event-to-observed-cache AGE with1s sampling, not +exact apply latency. MARK reference uses no advancing Kafka watermark in this +consumer projection; do not fabricate a first-apply latency distribution from +unchanged watermark. SDK MARK proof remains all22/22usable across both replicas. +Whole Data Layer Compose project CPU25samples mean4.70066cores,max5.1211cores; +no quota increase. Real core progress samples raw-age means~160-175ms, collection +~25-28ms; the earlier23s hot-core fault window is not a matched-load benchmark. + +Evidence: execution-readiness-20260930/evidence/consumer-final-300s/ contains +receipt, reviewed-summary, full bounded timestamped consumer/core/ingestor logs, +resource observations and cursor telemetry. R5 includes exact rejected views and +provider attribution. All five affected roles now use658a9570...54023, restart0/ +OOMfalse at inspected rollout checkpoint; all protected roles retain image/start +time. No order/DB mutation or TS configuration change. + +Source remains /home/bobby/data_layer on fix/okx-inverse-sandbox-readiness. +Remote fetched: dev7ee4af0/main8299838, published tagv2.2.0; no push/merge/tag yet. +Remaining release work: reviewed certificate/report with inherited gates and +these measured limits, CI/integration->main publication, exact legacy retirement +and build-cache inventory/cleanup. Keep native rollback coref204 and active +projector/ingestor-base7fe, V1, and untouched Query/Stream images. No TS E work. + +### Release Packaging Convergence (2026-09-30) +Current release workflow requires immutable SDK wheel SHA in certificate. +Compared v2.2.0..HEAD: SDK client already has tested typed refusal diagnostics +(_view_refusal, no payload/cursor leakage), but package/build VERSION remains +2.0.5. Publishing different bytes under the same SDK artifact identity would be +a provenance defect. Version-only correction to2.0.6 in SDK init/build script and +two current-version test assertions; preserve historical evidence/runtime pins. +No adapter policy/schema/auth changes, no TS or Query rebuild, no second C2. +Test deterministic wheel/metadata/import and current release-version assertions; +inherit existing behavioral diagnostics regressions. Certificate for v2.2.1 +will identify immutable per-role digests and link v2.2.0 evidence only for truly +unchanged behavior. Runtime SDK2.0.5 copies are not silently relabelled2.0.6. + +Packaging result: two targeted unittest cases PASS in existing reader image +0ed57643ae40 with source read-only, network none, bounded tmpfs, non-root; +disposable qdl-release-sdk-check removed automatically. Deterministic wheel +2.0.6 SHA256 26eede97407a42d8eaa15019cc31e0fe2c8a5545d82bdf7805cf1c9255cf7b2e. +Generated contract digest unchanged2a25a601...624; SDK source digest9930844e...90c. +No runtime SDK pin or running container changed. Public API remains2.0.0. +apply_patch helper failed (mountinfo path is not absolute); exact-match fallback +used perRule32 and four one-line version diffs inspected. Owner plan hunks stay +unstaged. No push/merge/tag or image cleanup in this packaging slice. + +### Release Delta Evidence And Retirement Plan (2026-09-30) +Prepare v2.2.1 certificate/report/notes using exact evidence hashes, per-role +runtime digests and SDK2.0.6; no fresh whole-catalogue/capacity claims. Acceptance +scope is bounded recovery and correct rejection, not always-eligible prices. +Retain observed OKX disconnects and paper cache observer correction explicitly. +Old v2.2.0 report remains inherited historical evidence for unchanged endpoints. +Release publication remains blocked on CI/approved integration ancestry; artifact +status PASS applies to this affected delta, not TS E or broker execution. +After artifact review remove only the ten inventoried stopped legacy containers +by exact IDs (no force, no volume removal), then only unreferenced Data Layer +obsolete images. Preserve active set and native rollbackf204/7fe. Do not touch +TS/Portal artifacts. Archive private inspect metadata, disk pre/post and unchanged +running-role start times. No shared/global BuildKit prune; only own build cache +if attributable, otherwise report deferred cache rather than delete others. + +Release structural checks PASS:128 binding/replica rows,64products,126usable, +2provider-attributed rejections, no p99 for small snapshot groups; endpoint SHA +0f2ce92fd97b3df2b9c5520d1d20ee8b26bd11918824d0ffa5b8ac872ee90465. +Certificate and release notes at upgrade/evidence/releases/v2.2.1/ explicitly +retain observed reconnects, n/denominators, polling limits and inherited gates. +Rust/Cargo/Dockerfile diff0e00c81..release-source is empty; do not rebuild Rust +for journal/SDK metadata. Private packet now APPLIED_TARGETED_ACCEPTANCE_REVIEWED, +publication still PENDING_CI. +Cleanup complete for exact ten stopped legacy containers and their3unreferenced +images37d7f518/56d331db/dd065fdf; no-v/no-force, all running image/start/restart +identities unchanged. Private inspect retained0600. Disk235353243648->235294134272 +bytes; concurrent host traffic means not exact reclaimed amount. Task-only Cargo +cache3,123,045,441logicalbytes removed (initial permission refusal, then bounded +sudo after rechecking no mount). Disk235431378944->232285339648bytes. No volume, +state, evidence, V1/TS/Portal or global BuildKit removal. Shared cache ownership +not attributable, explicitly deferred; native rollbackf204 and active7fe retained. +Remote fetch unchangeddev7ee4af0/main8299838. Next: push this feature/PRdev, CI, +then approvedmain release path. No second C2 and no TS lifecycle work. + +### CI Release Blocker - Dependency Security (2026-09-30) +Featuref460dee pushed, PR25->dev; CI36759351053. SDK Python3.10 PASS. +Unit job110037625827 FAILED before tests in pip-audit:12PyJWT2.13.0 advisories +(fixed2.14.0/2.15.0) +3urllib3 2.7.0 advisories(fixed2.8.0). Full bounded CI log +outsideGit: execution-readiness-20260930/evidence/ci-unit-36759351053.log. +Do not weaken audit or call this a Rust recovery failure. Official release docs: +https://pyjwt.readthedocs.io/en/latest/changelog.html +https://github.com/urllib3/urllib3/security/advisories +Source-only release prerequisite: raise PyJWT floor2.15.1(service+SDK) and +urllib3 floor2.8.0; targeted Poetry lock update onlythese2packages, inspect solver +diff. Test negative/positive JWT realms/SDK and dependency audit, rebuild SDK2.0.6 +(unpublished, replace candidate hash explicitly), update release evidence. +Running Rust/Query/Stream/TS remains unchanged by dependency source edits. +Any additional Python runtime rollout must name affected roles/digest/rollback; +current approval was fiveRustroles, not an implicit unrelated reader recreation. +Do not publish clean whole-runtime security certification for old Python images. + +Additional approved cleanup completed5unusedoldqdl-v2images ba41/579d/ed1e/3897/eec6; +retained3dd54bfd Query predecessor conservatively alongside currentreader and +native rollback. Disk232370102272->232098627584bytes, all runningroles unchanged. +Exact list at evidence/unused-image-cleanup.json; no TS/Portal/cache/volume removal. + +Security source slice: Poetry2.3.4(targeted update) changes ONLYpyjwt2.13.0->2.15.1 +andurllib3 2.7.0->2.8.0 pluslockcontenthash; no transitive dependency churn. +Auth/realm/TLS/SDK/package targeted32/32PASS with bothpatcheddependencies in a +disposableexistingreaderimage testcontainer. SDK2.0.6 unpublished candidatewheel +replaced explicitly: old26eede97...7b2e superseded by +d44fbf559c0c7eb52abf2a40091bcecc9e5ea025fed9c9b0bbbc6db6329395ad. +Certificate overallstatus PENDING_SECURITY_PACKAGING; affectedRustdeltaPASS stays +separate. CIoldrun: nativeintegration/contract/SDKPASS; dependency auditFAILED. +TwoQuery securitypackaging rollout request sent separately; do not recreateuntil +approved. No Rust/Stream/BAR/V1/TS change. Querypredecessor3dd retained conservatively. +Tooling: hostvenv unavailable, no hostpackageinstall; disposablePoetrycontainer +used canonical2.3.4 and hostUID1001. Initialwrongversion/UIDattempt stopped; +noexec tmpfs failure corrected with exec onlyonisolatedtemp. Test/solvercontainers +removed automatically. Partialemptyhostvenv removed, no other cachecleanup. + +Security candidate prepared, NOT deployed: qdl-v2-python:2.2.1-e366f75, +sha256:b40b49c361c7474691ef2d68eba8bcb4a22045cb2d010e2c73a8c0c68cc04a65. +Built standardDockerfile from gitarchivee366f75(noownerhunks/privatefiles). +Actualpackaged32auth/realm/TLS/SDKtestsPASS, networknone; solver/testcontainers +removed. Packet v2.2.1-e366f75-query-security holds candidate/rollbackCompose +validatedconfig--quiet; onlytwoQuery imagefielddiff, noenv/mount/TLS/configchange. +Rollback0ed57643...eb959; packetpreparedpendingownerresponse, noQueryrecreate. +CI36760348660 now passes dependency audit and is executingunit/native/contract +jobs; do not claim finalgreen prematurely. OriginalRust5role packet remains +completed; no repeat300s run. Candidateimageexplicitlyretainedforpendingpacket, +notgarbage. NoadditionalTSwork,mainmerge,tagorpublicationyet. + +### Publication Checkpoint And Tracker Reconciliation (2026-09-30) +Previous goal turn made implementation/rollout/evidence/cleanup progress; current +continuation verified live CIrun36760348660, not a guessedwait. ThreejobsPASS: +contract-tests, kn-native-integration and SDKPython3.10; unit-tests stillrunning +at Run unit tests, dependency audit alreadyPASS. No new C2/marketorder action. +Tracker now reflects DLR02/03 bounded fixes, DLR06 remainingpublication and +actual22sandbox demand versus64sealedentitlements. Earlier incident descriptions +remain historical; this correction does not reduce entitlement coverage. +CandidateQuery imageb40b49c...4a65 packagedversions verified PyJWT2.15.1/urllib32.8.0 +and32packagedtestsPASS. Private packetprepared, noownerresponse recordedyet and +noQueryrecreate. Need that packet outcome plus terminalCI before release closure. +Canonical remains /home/bobby/data_layer, samefeature, onecheckout; dev/main and +publishedv2.2.0 unchanged. All newtestcontainers auto-removed; candidate retained +for the pendingpacket. Source/plan evidence commit only; no runtime mutation. + +### Blocked Audit - Query Runtime Approval (2026-09-30) +The same separate twoQuery approval request remains unanswered across at least +three consecutive goal continuations. Safe independent work is complete: dependency +source fix,32source+32packagedtests, immutable image, exactcandidate/rollbackpacket, +sourcepush and scopedcleanup. No additional runtime permission is inferred from +automatic goalcontinuation. CIjob110041015650/run36760348660 independentlyremains +in_progress at Run unit tests on latestAPIpoll; previousgreen run took24minutes +for thatstep. Waiting is not classified as failure and no rerun/cancellation. +Goal blocked on pendingquerypacket approval (not complete). CI can continue on +GitHub while goal is blocked. On ownerresponse, repoll sameCIrun, apply onlytwo +approvedQueryroles, affectedauth/64productreadback, then finalcertificate/publication. +No newC2, sourcearchitecture, TSorderwork or broaderdeployment is authorized here. + +### Approved Query Security Handoff (2026-09-30) +Ownerexplicitapproval received; CI36760348660 all4jobsPASS. Query1rolled image +b40b49c...4a65 andhealthy. Operationalmeasurementmistake: rawdictionarycomparison +failed solelybecauseDockerreorderedvolumes; orchestration incorrectlycontinued +tothealreadyapprovedQuery1recreate instead ofshortcircuiting. Noextra rolechanged. +Immediatelystoppedfurtherroll, comparedmountsbytarget plusallothereffectivefields: +PASS onlyQuery1imagechanged, Query2unchanged. Thisisnotconfigdrift. Next commands +shortcircuitonfailedprecondition. BeforeQuery2freezeallrunningroles; retainolder +rollout/checkpointinventories separately, do notlabel this newbaselinepreQuery1. +RollonlyQuery2afterQuery1healthy, thenexisting64productx2preflight viaactualTS +identity(noSDKstream/ACK/order). NoC2repeat, noquota/TLS/manifest/statechanges. + +### Query Readback Outcome And Final Release Metadata (2026-09-30) +TwoQueryrolled/healthy,b40b49c...4a65; protectedroles unchanged sinceQuery1baseline. +128typedviews/64productsx2,125usable,0auth/manifestfailures. AllMARK/QUOTE/BOOK/BAR +usable. ThreeTRADEstrictage refusalsretained: BinanceSOL latestagg f/lrangeends +atservedrawtradeID, OKXBNB exactproviderID/timestampmatches. OKXSOL NOTquiet: +providernewtrade228.26msbeforequery,hostreceive27.40mslater,canonicalCreateTime +154.26msbeforequery. CreateTime/normalized/publishedfields are NOT transaction +commit/cacheapplytimestamp; do notpretend exactvisibilityrootcauseproven. Thisis +finitepropagation/correctreject, notalways-freshlasttrade. Tenfollowupreads7usable, +3age-refused thennewertradevisible; notsubstitutedfor125/128 oranotherC2. +BoundedKafkaattribution usedexplicitassign/nocommit; twoinitialACLdenials logged, +correctexistingreaderidentity/group workedwithoutACLmutationorconsumergroupjoin. +Raw1316/canonical1170recordsread over4s historicalbounds, nooffsetcommit. + +CI36760348660 all4jobsPASS. FinalcertificatePASS isaffected-delta qualification, +notcontinuousavailability, arbitrarycapacity or brokerexecution. Querysecurity +readback appended to publicendpointreport, previous300sreceipt remainsunchanged. +UpdateREADME withdatedlatestpatchscope/latency and retainolderload50benchmark as +historical. Finalpublication remainsfeature->dev->mainreleasePR/tag/workflow; +metadata-onlyhead mustpassCI, no newimage orC2 fordocumentation changes. +Authoritativepublicationreceipt will be written outsideGit at +execution-readiness-20260930/evidence/final-publication.json and linked to tag, +main/dev/head, workflowrun and artifact hashes. Untilthen notpublished. + +### Sealed Release Handoff And Publication Authority (2026-09-30) +All in-scope implementation and affectedruntime checks are recorded above; no +wholecatalogue/C2repeat. UpdatedREADME latestmeasurements separateolderload50. +Ninthobsoleteimage3dd removedafter exactQueryrollback0ed frozen; disk219302060032 +->219302334464bytes (concurrentwrites, notclaimedreclaim). Activeimages+named +rollback retained; sharedBuildKit/basebuildercache notgloballypruned. +Finalsourcefunctionalcommit e366f75 isCIgreen; subsequentartifact/README/plan +changes aremetadataonly and must clear normalPRCI beforeintegration. Owner plan +hunks remainunstaged; no owner source rewritten. The final operationalpublication +receipt is externalstate, intentionallynot a recursive newrelease commit: +~/.local/state/qdl-v2/execution-readiness-20260930/evidence/final-publication.json. +It must contain actualPRmerges, main/dev/tagSHAs, terminalreleaseworkflowstatus, +GitHubreleaseURL andasset checks. Its absence meanspublicationnotcomplete. This +samephasecloses onlywhen thatreceipt and immutablev2.2.1tag provepublication; +no futurephase or freshC2 isrequired. Releasecode and imagecomponents remain +immutable; docs commits do notcause redundantimage rebuilds. +TS E work isnotresumed automatically. Handoff permits using measuredbindings +under unchangedpolicy; it does not grant brokerorder authority orcertify TS E09. diff --git a/README.md b/README.md index 6314bbd1..b14cffd3 100644 --- a/README.md +++ b/README.md @@ -17,13 +17,57 @@ It currently serves: - Redis Pub/Sub streams for live consumers. - REST endpoints for warmup, recovery, diagnostics, and health checks. +## Execution Readiness Patch (2026-09-30) + +The v2.2.1 candidate release set is deployed and qualified for the affected +Data Layer scope. Publication is established by the [release tag](https://github.com/BobbyAxerol/quant-data-layer/releases/tag/v2.2.1), not this document alone. +Rust now uses an absolute batch deadline and bounded, generation-fenced L2 +resnapshot feedback. The two Query readers include PyJWT2.15.1/urllib32.8.0 +security fixes. No Kafka offsets, cache contents, TS strategies or order path +were reset or changed. The ten retired SQLite-path containers were removed; +state and named rollback images are preserved. + +SDK2.0.6 is the new reproducible artifact; existing TS SDK pins were not changed. +Final300s observation: paper60 and sandbox22 demanded routes were READY on11 +published heartbeats each. Four OKX disconnects recovered with fail-closed +refusals. This is not a continuous-uptime or always-eligible-price guarantee. +The sandbox manifest has64 entitlements, not64 active subscriptions. + +After the Query security rollout,64 products x2replicas returned128 typed views: +125 usable,3 strict TRADE-age refusals. Two were provider-confirmed quiet last +trades; OKXSOL had a newer trade propagating through the pipeline. It was not +mislabelled quiet. Never use last-trade age or healthy session status alone as +permission to execute; execution prices require the appropriate QUOTE/L2 and +Risk policy. No freshness threshold was relaxed. + +Latest SDK snapshot request-to-validated-usable **observed maximum, milliseconds** +(n8-12 usable samples/feed/venue, **not p99**, metadata resolution separate): + +| Feed | Binance | OKX | +|---|---:|---:| +| QUOTE | 31.78 | 13.69 | +| MARK/INDEX | 49.49 | 20.64 | +| TRADE | 29.22 | 18.73 | +| BOOK_SNAPSHOT | 45.30 | 52.97 | +| BOOK_DELTA | 37.17 | 12.32 | +| BAR latest | 18.02 | 35.12 | + +Full-stack CPU in the original final window: mean4.70cores, sampled peak5.12, +without cap increases. Redis polling measurements include up to1s observation +delay, not exact commit latency. Bounded authentic RF3 replay covered4k/s and +5k/s total ingress windows with catch-up, not indefinite5k/s per consumer. +See [v2.2.1 certificate](./upgrade/evidence/releases/v2.2.1/certificate.json), +[per-binding report](./upgrade/evidence/releases/v2.2.1/endpoint-report.json) and +[repair journal](./DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md#dl-execution-readiness-convergence-20260930). +Unchanged history/batch/reference capacity below is inherited dated evidence. + ## KN Production Status (2026-09-27) The **new Kafka-native read path is serving Trading System**, with SDK2.0.5. Production final load passed: 50 logical alpha sessions, 15,504 requests, 90 streams, 12 reconnects, actual TS60 READY in32/32 samples; no order actions. -Ten old SQLite projector/query/stream roles are stopped, with state and exact -rollback artifacts retained. The v2.2.0 runtime certificate is below; publication +At that dated benchmark, ten old SQLite projector/query/stream roles were stopped; +they were retired on2026-09-30 with state and the current rollback set retained. The v2.2.0 runtime certificate is below; publication is established by green remote CI, the main-line tag and GitHub release, not by a healthy container alone. SDK cursor persistence is bounded and off-loop, with durable acknowledgement after successful fsync. @@ -69,6 +113,7 @@ and [cutover/rollback runbook](./docs/runbooks/kn5-paired-cutover.md). - [Fund-grade implementation tracker](./DATA_LAYER_UNIFIED_IMPLEMENTATION_PLAN.md) - [Fund-grade architecture and migration guide](./upgrade/quant-data-layer-fund-grade-upgrade-architecture.md) - [OKX V5 market-data implementation guide](./upgrade/OKX_MARKET_DATA_V5_GUIDE_QUANT_DATA_LAYER.md) +- [V2.2.1 release notes](./upgrade/evidence/releases/v2.2.1/RELEASE_NOTES.md) - [V2.2.0 release notes](./upgrade/evidence/releases/v2.2.0/RELEASE_NOTES.md) - [V2.2.0 runtime certificate](./upgrade/evidence/releases/v2.2.0/certificate.json) - [Contributing guide](./CONTRIBUTING.md) diff --git a/config/v2/okx-inverse-sandbox-demand.yaml b/config/v2/okx-inverse-sandbox-demand.yaml new file mode 100644 index 00000000..ab005a64 --- /dev/null +++ b/config/v2/okx-inverse-sandbox-demand.yaml @@ -0,0 +1,43 @@ +schema: qdl.v2.production-demand.v1 +revision: 1 +consumers: +- consumer_id: trading-system.sandbox.stable + consumer_grade: EXECUTION + requirements: + - venue: OKX + market: SWAP + product_type: PERPETUAL + native_symbol: BTC-USD-SWAP + feed: QUOTE + interval: null + source_policy_id: crypto_primary_v2 + - venue: OKX + market: SWAP + product_type: PERPETUAL + native_symbol: BTC-USD-SWAP + feed: MARK_INDEX_PRICE + interval: null + source_policy_id: crypto_liquid_v2 + max_freshness_ms: 2000 + require_live: true + index_native_symbol: BTC-USD + - venue: OKX + market: SWAP + product_type: PERPETUAL + native_symbol: BTC-USD-SWAP + feed: BOOK_SNAPSHOT + interval: null + source_policy_id: crypto_liquid_v2 + depth_per_side: 100 + max_freshness_ms: 60000 + require_live: true + - venue: OKX + market: SWAP + product_type: PERPETUAL + native_symbol: BTC-USD-SWAP + feed: BOOK_DELTA + interval: null + source_policy_id: crypto_liquid_v2 + depth_per_side: 100 + max_freshness_ms: 2000 + require_live: true diff --git a/poetry.lock b/poetry.lock index 60109378..c427f78b 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1549,14 +1549,14 @@ typing-extensions = ">=4.14.1" [[package]] name = "pyjwt" -version = "2.13.0" +version = "2.15.1" description = "JSON Web Token implementation in Python" optional = false python-versions = ">=3.9" groups = ["main"] files = [ - {file = "pyjwt-2.13.0-py3-none-any.whl", hash = "sha256:66adcc2aff09b3f1bbd95fc1e1577df8ac8723c978552fd43304c8a290ac5728"}, - {file = "pyjwt-2.13.0.tar.gz", hash = "sha256:41571c89ca91598c79e8ef18a2d07367d4810fbbd6f637794879baf1b7703423"}, + {file = "pyjwt-2.15.1-py3-none-any.whl", hash = "sha256:42d59d631f7768a1028a64c7ff581a9bf7519804daf91fc5b6c56e30eec5e193"}, + {file = "pyjwt-2.15.1.tar.gz", hash = "sha256:4f259e80cdfb6b3fc18a7de51fd1ef9ec79652f25019bae68975ca2468a34df8"}, ] [package.dependencies] @@ -1875,14 +1875,14 @@ files = [ [[package]] name = "urllib3" -version = "2.7.0" +version = "2.8.0" description = "HTTP library with thread-safe connection pooling, file post, and more." optional = false python-versions = ">=3.10" groups = ["main"] files = [ - {file = "urllib3-2.7.0-py3-none-any.whl", hash = "sha256:9fb4c81ebbb1ce9531cce37674bbc6f1360472bc18ca9a553ede278ef7276897"}, - {file = "urllib3-2.7.0.tar.gz", hash = "sha256:231e0ec3b63ceb14667c67be60f2f2c40a518cb38b03af60abc813da26505f4c"}, + {file = "urllib3-2.8.0-py3-none-any.whl", hash = "sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3"}, + {file = "urllib3-2.8.0.tar.gz", hash = "sha256:63bf2ead4c879426ebf22ef2a781eeb4aa3b4ae798a0435506f8687fd5bb9b63"}, ] [package.extras] @@ -2196,4 +2196,4 @@ files = [ [metadata] lock-version = "2.1" python-versions = "^3.10" -content-hash = "d0af8c4e1e251205cfa52599fd30362e0136cdbc0c66ddbe8d342c3faaac1692" +content-hash = "67f16f74f84e421af157a269bcf562af1a31b6d70f940e2d3e973037555b8ddc" diff --git a/pyproject.toml b/pyproject.toml index a33bd2e2..37cde3e6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -30,7 +30,8 @@ dependencies = [ "httpx (>=0.28.0,<1.0.0)", "idna (>=3.15,<4.0)", "pillow (>=12.3.0,<13.0.0)", - "pyjwt[crypto] (>=2.13.0,<3.0.0)", + "pyjwt[crypto] (>=2.15.1,<3.0.0)", + "urllib3 (>=2.8.0,<3.0.0)", "soupsieve (>=2.8.4,<3.0.0)", "starlette (>=1.3.1,<2.0.0)", "tomli (>=2.0.1,<3.0.0) ; python_version < '3.11'" diff --git a/qdl/adapters/binance/bar_edge.py b/qdl/adapters/binance/bar_edge.py index a68513a6..adafdf1f 100644 --- a/qdl/adapters/binance/bar_edge.py +++ b/qdl/adapters/binance/bar_edge.py @@ -7,7 +7,10 @@ from typing import Callable from app.providers.binance.rest import BinanceProviderError, fetch_klines -from qdl.adapters.intervals import BarHistoryGapError, canonical_interval_ms, latest_closed_boundary_ms +from qdl.adapters.intervals import ( + BarHistoryGapError, BarHistoryOverlapError, canonical_interval_ms, + is_valid_bar_open_ms, latest_closed_boundary_ms, +) from qdl.provider.v1 import raw_provider_pb2 from qdl.raw.capture import capture_exact_frame @@ -236,6 +239,43 @@ def fetch_closed_bar_history_raw_envelopes( observed_ms=observed_ms, interval_ms=interval_ms, ) + if len(closed) > page_limit or any(int(row[0]) > page_end_ms for row in closed): + raise RuntimeError("Binance closed-bar history page exceeds requested end boundary") + for row in closed: + open_time = int(row[0]) + previous = selected_by_open.get(open_time) + if previous is not None and previous != row: + raise RuntimeError("Binance history pages conflict for one open time") + selected_by_open[open_time] = row + selected = tuple(selected_by_open[key] for key in sorted(selected_by_open)) + # Walk newest first: only the suffix after the newest discontinuity can + # be certified, including when that discontinuity crosses REST pages. + for index in range(len(selected) - 1, 0, -1): + previous, current = int(selected[index - 1][0]), int(selected[index][0]) + if current - previous == interval_ms: + continue + error = BarHistoryGapError("Binance", binding.native_symbol, binding.interval, + previous, current) + suffix = selected[index:] + latest_open = latest_closed_boundary_ms( + binding.interval, observed_ms, provider="BINANCE" + ) - interval_ms + if (error.kind == "OVERLAPPING_PROVIDER_WINDOWS" and binding.interval == "3d" + and int(suffix[-1][0]) == latest_open + and all(is_valid_bar_open_ms(binding.interval, int(row[0]), provider="BINANCE") + for row in suffix)): + received_at_ns = time.time_ns() + envelopes = tuple(_capture_row( + binding, row, origin="BACKFILLED", received_at_ns=received_at_ns + offset, + test_provenance=test_provenance, + ) for offset, row in enumerate(suffix)) + raise BarHistoryOverlapError( + "Binance", binding.native_symbol, binding.interval, previous, current, + requested_rows=limit, recent_envelopes=envelopes, + ) + raise error + if any(int(row[6]) > page_end_ms for row in closed): + raise RuntimeError("Binance closed-bar history page exceeds requested end boundary") exhausted = False if len(closed) != page_limit: if not (allow_short and len(closed) < page_limit and _provider_exhausted_before( @@ -247,18 +287,6 @@ def fetch_closed_bar_history_raw_envelopes( f"requested={page_limit} observed={len(closed)}" ) exhausted = True - if any(int(row[6]) > page_end_ms for row in closed): - raise RuntimeError( - "Binance closed-bar history page exceeds requested end boundary" - ) - for row in closed: - open_time = int(row[0]) - previous = selected_by_open.get(open_time) - if previous is not None and previous != row: - raise RuntimeError( - "Binance history pages conflict for one open time" - ) - selected_by_open[open_time] = row if exhausted or not closed: break earliest_open_ms = min(int(row[0]) for row in closed) diff --git a/qdl/adapters/intervals.py b/qdl/adapters/intervals.py index ad7ff6a1..49d1e2e6 100644 --- a/qdl/adapters/intervals.py +++ b/qdl/adapters/intervals.py @@ -23,6 +23,30 @@ def __init__(self, provider: str, symbol: str, interval: str, previous: int, cur super().__init__(f"{provider} closed-bar history discontinuity kind={self.kind} " f"symbol={symbol} interval={interval} previous_ms={previous} current_ms={current}") + def retry_delay_seconds(self, attempt: int) -> int: + # Overlapping native windows cannot be repaired by fetching a missing + # bar. Recheck for provider corrections daily, retaining failed coverage. + if self.kind == "OVERLAPPING_PROVIDER_WINDOWS": + return 86_400 + return min(2 ** min(attempt, 6), 30) + + +class BarHistoryOverlapError(BarHistoryGapError): + """Incomplete requested history with an independently validated recent suffix. + + Only history materialization may use these envelopes. Request fetchers still + raise: this is never a successful short response or proof of a listing floor. + """ + + def __init__(self, provider: str, symbol: str, interval: str, previous: int, + current: int, *, requested_rows: int, recent_envelopes: tuple): + super().__init__(provider, symbol, interval, previous, current) + if self.kind != "OVERLAPPING_PROVIDER_WINDOWS" or not recent_envelopes: + raise ValueError("history overlap outcome requires a nonempty recent suffix") + self.requested_rows = requested_rows + self.recent_envelopes = recent_envelopes + self.older_prefix_status = "UNCERTIFIED_PROVIDER_OVERLAP" + _UNIT_MS = { "s": 1_000, diff --git a/qdl/adapters/okx/instruments.py b/qdl/adapters/okx/instruments.py index c106aad4..06762df1 100644 --- a/qdl/adapters/okx/instruments.py +++ b/qdl/adapters/okx/instruments.py @@ -84,7 +84,9 @@ def parse_public_instrument( multiplier_text = format(Decimal(ct_val_text) * Decimal(ct_mult_text), "f") except InvalidOperation as exc: raise ValueError("OKX contract multiplier fields must be exact decimals") from exc - family_parts = identity.canonical_symbol.split("-") + # Contract value currency is not the underlying asset for inverse contracts. + # Dated/option IDs include expiry/strike; currency identity comes from family. + family_parts = str(payload.get("instFamily") or identity.canonical_symbol).split("-") family_base = family_parts[0] if len(family_parts) >= 2 else "" family_quote = family_parts[-1] if len(family_parts) >= 2 else "" record = InstrumentRecord( @@ -93,7 +95,7 @@ def parse_public_instrument( asset_class=_ASSET_CLASSES[inst_type], native_symbol=inst_id, base_asset=str( - payload.get("baseCcy") or payload.get("ctValCcy") or family_base + payload.get("baseCcy") or family_base or payload.get("ctValCcy") ).upper(), quote_asset=str(payload.get("quoteCcy") or family_quote).upper(), settlement_asset=str( diff --git a/qdl/consumer/manifest.py b/qdl/consumer/manifest.py index 8b14decd..33d4c5fa 100644 --- a/qdl/consumer/manifest.py +++ b/qdl/consumer/manifest.py @@ -102,6 +102,8 @@ def __post_init__(self) -> None: raise ValueError("consumer manifest contains an unknown data-plane permission") if self.execution_dependency not in {"FORBIDDEN", "PAPER_ONLY", "ALLOWED"}: raise ValueError("consumer execution dependency policy is invalid") + if self.environment in {"sandbox", "live"} and self.execution_dependency == "PAPER_ONLY": + raise ValueError("sandbox/live consumer cannot declare PAPER_ONLY execution dependency") if not self.requirements: raise ValueError("consumer manifest requires at least one data requirement") if self.rollback_contract not in {"V1", "V2"}: diff --git a/qdl/reference/execution_live.py b/qdl/reference/execution_live.py index 2525e47c..c2ac59a7 100644 --- a/qdl/reference/execution_live.py +++ b/qdl/reference/execution_live.py @@ -208,7 +208,8 @@ async def fetch( request, capability, code, - "execution MARK/INDEX live view did not return a current active record", + "execution MARK/INDEX live view did not return a current active record: " + + ",".join(dict.fromkeys(errors)), ) def stats(self) -> dict[str, int]: @@ -236,8 +237,13 @@ def _bounded_reason(response: httpx.Response) -> str: except (ValueError, AttributeError): return "UNAVAILABLE" normalized = str(detail).upper() - if normalized.endswith(":STALE"): - return "STALE" + # Component cadence and session age are freshness failures, not an + # absent provider. Preserve the bounded reason for consumer diagnosis. + reason = normalized.rsplit(":", 1)[-1] + if reason in {"STALE", "COMPONENT_STALE", "SESSION_LIVENESS"}: + return reason + if reason in {"SESSION_STATE", "QUIET_POLICY_UNAVAILABLE", "LINEAGE_INVALID"}: + return reason if normalized.endswith(":GAP_OR_RESYNC"): return "GAPPED" if normalized.endswith(":IDENTITY_MISMATCH"): @@ -250,7 +256,7 @@ def _bounded_reason(response: httpx.Response) -> str: @staticmethod def _failure_code(errors: list[str]) -> str: - if "STALE" in errors: + if any(reason in {"STALE", "COMPONENT_STALE", "SESSION_LIVENESS"} for reason in errors): return "LIVE_VIEW_STALE" if "GAPPED" in errors: return "LIVE_VIEW_GAPPED" diff --git a/qdl/runtime/kn_query_backend.py b/qdl/runtime/kn_query_backend.py index ad6b27d9..25fafb8a 100644 --- a/qdl/runtime/kn_query_backend.py +++ b/qdl/runtime/kn_query_backend.py @@ -47,7 +47,7 @@ from qdl.adapters.intervals import canonical_interval_ms from qdl.common.v1 import common_pb2 from qdl.marketdata.v2 import market_data_pb2 -from qdl.query import DataRequirement, FeedType, GapRecord, HistoryResult, MarketDataItem, RecoveryPolicy +from qdl.query import CoverageStatus, DataRequirement, FeedType, GapRecord, HistoryResult, MarketDataItem, RecoveryPolicy from qdl.query.contracts import CanonicalErrorCode, QueryProblem from qdl.query.results import NON_REPLAYABLE_STREAM_CURSOR, QueryBackendError, GapScanResult from qdl.query.row_cache import BoundedRowCache @@ -368,6 +368,12 @@ def history_with_envelopes( ) if result is None: return None, () + # Native Binance 3d has overlapping historical grids. A retained recent + # suffix does not prove listing exhaustion or certify an older prefix. + if (binding.feed is FeedType.BAR and binding.interval == "3d" + and binding.instrument.identity.venue == "BINANCE" + and start_ns is None and len(result.items) < requested): + result = replace(result, coverage=CoverageStatus.PARTIAL) selected = records if start_ns is not None: selected = tuple( diff --git a/qdl/runtime/production_catalog.py b/qdl/runtime/production_catalog.py index 31e21ec5..6f16d313 100644 --- a/qdl/runtime/production_catalog.py +++ b/qdl/runtime/production_catalog.py @@ -20,6 +20,7 @@ from qdl.domain.instrument import InstrumentRecord, InstrumentStatus from qdl.query import ConsumerGrade, FeedType from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.execution_l2 import EXECUTION_L2_MATERIALIZATION_INTERVAL_MS from qdl.runtime.stable_deployment import ( V2_REALTIME_RAW_TOPIC, StableAcquisitionPlan, @@ -206,6 +207,8 @@ def load_many(cls, paths: Iterable[str | Path]) -> "ProductionDemandManifest": if current is not None and current.source_policy_id != item.source_policy_id: raise ValueError("one feed requirement cannot use conflicting source policies") deduped.setdefault(key, item) + if current is not None and item.consumer_grade is ConsumerGrade.EXECUTION: + deduped[key] = replace(current, consumer_grade=ConsumerGrade.EXECUTION) consumers_by_key.setdefault(key, set()).add(item.consumer_id) # Consumer IDs are audit inputs but do not alter a canonical source binding. del consumers_by_key @@ -428,6 +431,11 @@ def build_from_records( (item.venue, item.market, item.native_symbol) for item in demand.demands if item.feed is not FeedType.BAR } + execution_books = { + (item.venue, item.market, item.native_symbol) + for item in demand.demands + if item.feed in _BOOK_FEEDS and item.consumer_grade is ConsumerGrade.EXECUTION + } for item in demand.demands: key = (item.venue, item.market, item.native_symbol) try: @@ -443,7 +451,8 @@ def build_from_records( binding_id = self._binding_id(item) bindings.append(self._source_binding(binding_id, item, record)) acquisitions.append(self._acquisition( - binding_id, item, execution_symbol=key in execution_symbols + binding_id, item, execution_symbol=key in execution_symbols, + execution_book=key in execution_books )) source = { "schema": _SOURCE_SCHEMA, @@ -699,7 +708,8 @@ def _book_source_stem(item: ProductionDemand) -> str: @staticmethod def _acquisition( - binding_id: str, item: ProductionDemand, *, execution_symbol: bool = True + binding_id: str, item: ProductionDemand, *, execution_symbol: bool = True, + execution_book: bool = False ) -> dict[str, Any]: if item.venue == "BINANCE": # The provider kind and the stream endpoint both follow the market: @@ -833,6 +843,8 @@ def _acquisition( # it is never polled as if it were a Binance diff-depth book. "snapshot_refresh_seconds": 30, } + if execution_book: + result["l2"]["materialized_snapshot_interval_ms"] = EXECUTION_L2_MATERIALIZATION_INTERVAL_MS if item.feed is FeedType.MARK_INDEX_PRICE: try: component_quiet_after_ms = _MARK_INDEX_COMPONENT_QUIET_AFTER_MS[ diff --git a/qdl/runtime/sandbox_extension.py b/qdl/runtime/sandbox_extension.py new file mode 100644 index 00000000..b2d96286 --- /dev/null +++ b/qdl/runtime/sandbox_extension.py @@ -0,0 +1,150 @@ +"""Pure additive sandbox read packet; activation and certification are separate.""" +from copy import deepcopy +import hashlib +import json +from pathlib import Path +import tempfile + +import yaml + +from qdl.consumer.manifest import ConsumerManifestLoader +from qdl.consumer.realtime_route import requirement_key +from qdl.consumer.universal_release import ConsumerRouteBinding +from qdl.runtime.production_catalog import ProductionCatalogBuilder +from qdl.runtime.stable_catalog import StableSourceCatalog +from qdl.runtime.stable_deployment import StableAcquisitionPlan + + +def _digest(value): + return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + + +def _key(row): + return (row["instrument_uid"], row["feed"], row.get("interval"), row["source_policy_id"]) + + +def prepare_sandbox_extension(*, catalog, acquisition, manifest, binding, demand, + okx_rows, template_uid): + """Compile four reviewed inverse feeds without changing other consumer scopes. + + The old execution template supplies policy, never its instrument identity. + The new generation is a prepared config receipt, not a release certificate. + """ + consumer = ConsumerManifestLoader.from_mapping(manifest) + base = ConsumerRouteBinding.from_canonical_mapping(binding) + current = StableSourceCatalog.from_mapping(catalog) + if (consumer.environment != "sandbox" + or consumer.consumer_id != "trading-system.sandbox.stable" + or consumer.subject != "spiffe://qdl/sandbox/trading-system-stable" + or consumer.execution_dependency != "ALLOWED" + or base.consumer_id != consumer.consumer_id + or base.consumer_manifest_revision != consumer.manifest_revision): + raise ValueError("sandbox identity/revision mismatch") + expected = {"QUOTE", "MARK_INDEX_PRICE", "BOOK_SNAPSHOT", "BOOK_DELTA"} + if (len(demand.demands) != 4 or {r.feed.value for r in demand.demands} != expected + or any(r.consumer_id != consumer.consumer_id or r.consumer_grade.value != "EXECUTION" + or (r.venue, r.market, r.product_type, r.native_symbol) != + ("OKX", "SWAP", "PERPETUAL", "BTC-USD-SWAP") for r in demand.demands)): + raise ValueError("extension outside approved inverse sandbox scope") + requirements = {_key(r): r for r in manifest["spec"]["requirements"]} + products = {_key(r): r for r in binding["products"]} + if (len(requirements) != len(manifest["spec"]["requirements"]) + or len(products) != len(binding["products"]) or products.keys() != requirements.keys()): + raise ValueError("base binding coverage mismatch") + for key, route in products.items(): + record = current.instrument_for(key[0]) + if (route["instrument_id"], route["native_symbol"], route["venue"], + route["market"], route["product_type"]) != ( + record.identity.instrument_id, record.native_symbol, record.identity.venue, + record.identity.market, record.identity.product_type.value): + raise ValueError("base binding/catalog identity mismatch") + template = current.instrument_for(template_uid) + if (template.identity.venue, template.identity.market, template.identity.product_type.value) != ( + "OKX", "SWAP", "PERPETUAL"): + raise ValueError("template venue/product mismatch") + extension = ProductionCatalogBuilder( + catalog_revision=current.catalog_revision + 1, + source_policy_revision=catalog["source_policy_revision"], + authority_revision=catalog["authority_revision"], + canonical_stream=catalog["canonical_stream"], + raw_topic=acquisition["topics"]["raw"], + quarantine_topic=acquisition["topics"]["quarantine"], + ).build(demand=demand, binance_usdm=None, okx_rows=okx_rows) + inverse = extension.source_catalog["instruments"] + if (len(inverse) != 1 or inverse[0]["base_asset"] != "BTC" + or inverse[0]["quote_asset"] != "USD" or inverse[0]["settlement_asset"] != "BTC" + or inverse[0]["attributes"].get("ctType") != "inverse"): + raise ValueError("inverse collateral/identity mismatch") + result_catalog = deepcopy(catalog) + for field, identifier in (("instruments", "instrument_id"), ("bindings", "binding_id")): + old_ids = {r[identifier] for r in catalog[field]} + if any(r[identifier] in old_ids for r in extension.source_catalog[field]): + raise ValueError("extension already exists; explicit revision review required") + result_catalog[field].extend(deepcopy(extension.source_catalog[field])) + result_catalog["catalog_revision"] += 1 + result_acquisition = deepcopy(acquisition) + result_acquisition["revision"] += 1 + result_acquisition["bindings"].extend(extension.acquisition_plan["bindings"]) + parsed_catalog = StableSourceCatalog.from_mapping(result_catalog) + with tempfile.TemporaryDirectory(prefix="qdl-sandbox-validate-") as raw: + path = Path(raw) / "acquisition.yaml" + path.write_text(yaml.safe_dump(result_acquisition)) + StableAcquisitionPlan.load(path, catalog=parsed_catalog) + target = deepcopy(manifest) + target["metadata"]["revision"] += 1 + target_binding = deepcopy(binding) + added = [] + for row in extension.source_catalog["bindings"]: + key = (template_uid, row["feed"], row.get("interval"), row["source"]["source_policy_id"]) + if key not in requirements or key not in products: + raise ValueError("missing exact execution policy template") + req, route = deepcopy(requirements[key]), deepcopy(products[key]) + if (req["consumer_grade"] != "EXECUTION" or route["route"] != "V2_PRIMARY" + or not route["execution_grade"] or route["fallback"] != "BLOCKED" + or req.get("gap_policy", "BLOCK") != "BLOCK" + or req.get("stale_policy", "BLOCK") != "BLOCK" + or not req.get("require_full_coverage", True)): + raise ValueError("template is not fail-closed execution policy") + if (route["max_freshness_ms"] != req["max_freshness_ms"] + or route["require_final_bars"] != req.get("require_final_bars", True)): + raise ValueError("template quality mismatch") + req["instrument_uid"] = row["instrument_uid"] + instrument = parsed_catalog.instrument_for(row["instrument_uid"]) + route.update(instrument_uid=instrument.identity.instrument_uid, + instrument_id=instrument.identity.instrument_id, + native_symbol=instrument.native_symbol) + target["spec"]["requirements"].append(req) + added.append((req, route)) + parsed_manifest = ConsumerManifestLoader.from_mapping(target) + for req, route in added: + parsed = next(r for r in parsed_manifest.requirements if ( + r.instrument_uid == req["instrument_uid"] and r.feed.value == req["feed"])) + parsed_catalog.binding_for(parsed) + route["requirement_id"] = hashlib.sha256( + f"{consumer.consumer_id}:{requirement_key(parsed)}".encode()).hexdigest() + target_binding["products"].append(route) + provenance = { + "schema": "qdl.v2.sandbox-extension-preparation.v1", + "status": "PREPARED_NOT_ACTIVATED_NOT_CERTIFIED", + "base_binding_sha256": base.binding_sha256, + "base_catalog_sha256": _digest(catalog), + "catalog_sha256": _digest(result_catalog), + "acquisition_sha256": _digest(result_acquisition), + "consumer_manifest_sha256": parsed_manifest.manifest_sha256, + "metadata_sha256": _digest(okx_rows), + "demand_sha256": _digest([{"consumer": r.consumer_id, "key": [str(v) for v in r.requirement_key], + "index": r.index_native_symbol, "depth": r.depth_per_side, + "freshness": r.max_freshness_ms, "live": r.require_live} + for r in demand.demands]), + "added_products": len(added), + } + target_binding.update(consumer_manifest_revision=parsed_manifest.manifest_revision, + release_revision=base.release_revision + 1, + universal_manifest_sha256=_digest(provenance), + inventory_sha256=provenance["demand_sha256"]) + target_binding["products"].sort(key=lambda r: r["requirement_id"]) + target_binding.pop("binding_sha256") + target_binding["binding_sha256"] = _digest(target_binding) + ConsumerRouteBinding.from_canonical_mapping(target_binding) + return dict(catalog=result_catalog, acquisition=result_acquisition, + manifest=target, binding=target_binding, provenance=provenance) diff --git a/qdl/runtime/stable.py b/qdl/runtime/stable.py index 33e9e85a..6c66a69e 100644 --- a/qdl/runtime/stable.py +++ b/qdl/runtime/stable.py @@ -526,7 +526,12 @@ def stable_grpc_server_credentials( def load_stable_manifests(config: StableRuntimeConfig) -> ConsumerManifestRegistry: manifests = tuple(ConsumerManifestLoader.load(path) for path in config.manifest_paths) - if any(item.environment != config.environment for item in manifests): + # Public market storage/cursors keep their realm. Consumer realms require + # explicit, separately pinned JWT key/subject bindings. + raw_realms = os.environ.get("QDL_DATA_JWT_KEY_ENVIRONMENTS_JSON") + realms = (DataPlaneSecurityConfig.from_environment().consumer_environments + if raw_realms else frozenset({config.environment})) + if any(item.environment not in realms for item in manifests): raise ValueError("stable consumer manifest environment mismatch") return ConsumerManifestRegistry(manifests) diff --git a/qdl/runtime/stable_bar_edge.py b/qdl/runtime/stable_bar_edge.py index 37abddf5..43a84102 100644 --- a/qdl/runtime/stable_bar_edge.py +++ b/qdl/runtime/stable_bar_edge.py @@ -14,6 +14,7 @@ from qdl.adapters.intervals import ( BarHistoryGapError, + BarHistoryOverlapError, canonical_interval_ms, is_valid_bar_open_ms, latest_closed_boundary_ms, @@ -347,6 +348,8 @@ def __init__( self.history_demand = dict(history_demand) if history_demand is not None else None self.history_gate = history_gate self._history_short: dict[str, tuple[int, int]] = {} + self._history_suspended: dict[str, dict] = {} + self._history_suffix_pending: dict[str, BarHistoryOverlapError] = {} self._history_gate_closed_at: float | None = None # KN-4 D47-4: the history/live join. History holds bars closed before # ``history_end_ms`` (the live log's start, e.g. the mirror start); @@ -487,6 +490,7 @@ def _state_payload(self) -> dict: "connection_generation": self.connection_generation, "last_open_ms": { key: self._last_open_ms[key] for key in sorted(self._last_open_ms) + if key not in getattr(self, "_history_suspended", {}) }, } @@ -825,7 +829,9 @@ def _apply_history_repair_plan( self._assert_canonical_cache_identity() if advance_watermark: # Publish progress: the live watermark may move on the Kafka ACK. - self._last_open_ms[plan.source.binding_id] = max(plan.expected_opens) + self._last_open_ms[plan.source.binding_id] = max( + max(plan.expected_opens), self._last_open_ms.get(plan.source.binding_id, 0) + ) self._persist_state() self._record_serving(plan.source.binding_id, plan.expected_opens, published=bool(current.missing_envelopes)) @@ -1062,6 +1068,8 @@ def _rebase_if_canonical_cache_generation_changed(self) -> bool: previous_cache_id = self.canonical_cache_id self.canonical_cache_id = observed_cache_id self._last_open_ms.clear() + for name in ("_history_suspended", "_history_suffix_pending", "_history_retry"): + getattr(self, name, {}).clear() self._native_recovery_pending.clear() self._native_recovery_verified_open.clear() self._native_gap_first_seen_at.clear() @@ -1315,6 +1323,8 @@ def _rebase_changed_products(self) -> tuple[str, ...]: if previous is not None and generation != previous: rebased.append(binding_id) self._last_open_ms.pop(binding_id, None) + for name in ("_history_suspended", "_history_suffix_pending", "_history_retry"): + getattr(self, name, {}).pop(binding_id, None) for store in (self._native_recovery_pending, self._native_recovery_verified_open, self._native_gap_first_seen_at, self._native_recovery_visible_after, self._retry_attempts, self._next_retry_at, self._serving_pending): @@ -1379,6 +1389,7 @@ def serving_status(self) -> dict[str, int]: "served": len(self._served), "pending": len(self._serving_pending), "overdue": len(self._serve_overdue), "short": len(getattr(self, "_history_short", {})), "history_failed": len(getattr(self, "_history_retry", {})), + "history_suspended": len(getattr(self, "_history_suspended", {})), "gate_closed": int(getattr(self, "_history_gate_closed_at", None) is not None)} def bootstrap_history(self, *, max_bindings: int | None = None) -> int: @@ -1401,8 +1412,13 @@ def bootstrap_history(self, *, max_bindings: int | None = None) -> int: kn_mode = getattr(self, "bar_readback", None) is not None failures = getattr(self, "_history_retry", {}) self._history_retry = failures + suspended = getattr(self, "_history_suspended", {}) + self._history_suspended = suspended + pending = getattr(self, "_history_suffix_pending", {}) + self._history_suffix_pending = pending for source, acquisition in self.history_bindings + self.history_okx_bindings: - if source.binding_id in self._last_open_ms or source.binding_id in self._history_short_empty(): + if ((source.binding_id in self._last_open_ms and source.binding_id not in suspended) + or source.binding_id in self._history_short_empty()): continue if kn_mode and not self._history_gate_open(): return published # resumed on a later turn; live bars keep flowing @@ -1414,15 +1430,53 @@ def bootstrap_history(self, *, max_bindings: int | None = None) -> int: attempted += 1 bootstrap_rows = self._bootstrap_rows_for(source) try: + if source.binding_id in pending: + raise pending[source.binding_id] values = self._fetch_history(source, acquisition, rows=bootstrap_rows, observed_ms=observed_ms, allow_short=kn_mode) except BarHistoryGapError as error: if not kn_mode: raise attempt += 1 - failures[source.binding_id] = (attempt, self.clock() + min(2 ** min(attempt, 6), 30)) - logger.error("stable BAR provider history refused binding=%s attempt=%s error=%s", - source.binding_id, attempt, error) + retry_delay = error.retry_delay_seconds(attempt) + failures[source.binding_id] = (attempt, self.clock() + retry_delay) + logger.error("stable BAR provider history refused binding=%s attempt=%s retry_in_seconds=%s error=%s", + source.binding_id, attempt, retry_delay, error) + if isinstance(error, BarHistoryOverlapError): + values = error.recent_envelopes + opens = tuple(self._open_time_ms(acquisition, item) for item in values) + # Keep failed deep coverage separate from a live watermark. + # The legacy checkpoint excludes suspended bindings, so a + # restart rechecks them rather than claiming full bootstrap. + suspended[source.binding_id] = { + "status": error.older_prefix_status, "requested_rows": error.requested_rows, + "previous_ms": error.previous, "current_ms": error.current, + "recent_from_ms": min(opens), "recent_rows": len(values), + } + # Retain only the outcome, not traceback frames from each + # failed sink retry (which would accumulate indefinitely). + pending[source.binding_id] = BarHistoryOverlapError( + error.provider, error.symbol, error.interval, error.previous, error.current, + requested_rows=error.requested_rows, recent_envelopes=values, + ) + try: + published += self._publish_history( + source, acquisition, values, expected_rows=len(values), + ) + except Exception: + # Retry publication of these same bytes, not provider + # pagination, when the durable sink temporarily fails. + failures[source.binding_id] = (attempt, self.clock() + 30) + raise + pending.pop(source.binding_id, None) + logger.warning("stable BAR recent suffix materialized binding=%s rows=%s " + "first_open_ms=%s older_prefix=%s requested_rows=%s", + source.binding_id, len(values), min(opens), + error.older_prefix_status, error.requested_rows) + continue + if source.binding_id in suspended and not values: + failures[source.binding_id] = (attempt + 1, self.clock() + 86_400) + logger.error("stable BAR suspended history recheck empty binding=%s", source.binding_id) continue failures.pop(source.binding_id, None) if len(values) < bootstrap_rows: @@ -1439,8 +1493,11 @@ def bootstrap_history(self, *, max_bindings: int | None = None) -> int: values, expected_rows=len(values), ) + if source.binding_id in suspended: + suspended.pop(source.binding_id) + self._persist_state() self._history_bootstrapped = ( - set(self._last_open_ms) | self._history_short_empty() == set(self._binding_ids) + (set(self._last_open_ms) - set(suspended)) | self._history_short_empty() == set(self._binding_ids) ) if not self._history_bootstrapped: if kn_mode and failures: @@ -1935,7 +1992,8 @@ def _history_loop_delay(self, now: float) -> float | None: or getattr(self, "_history_bootstrapped", False)): return None retries = getattr(self, "_history_retry", {}) - missing = set(self._binding_ids) - set(self._last_open_ms) - self._history_short_empty() + complete = set(self._last_open_ms) - set(getattr(self, "_history_suspended", {})) + missing = set(self._binding_ids) - complete - self._history_short_empty() if not missing: return None return max(.1, min(max(0., retries.get(key, (0, now))[1] - now) for key in missing)) diff --git a/qdl/security/data_plane.py b/qdl/security/data_plane.py index 69440117..b8636591 100644 --- a/qdl/security/data_plane.py +++ b/qdl/security/data_plane.py @@ -43,6 +43,7 @@ class DataPlaneSecurityConfig: algorithms: tuple[str, ...] max_token_lifetime_seconds: int = 900 subjects_by_key_id: Mapping[str, str] | None = None + environments_by_key_id: Mapping[str, str] | None = None def __post_init__(self) -> None: if not all((self.environment.strip(), self.issuer.strip(), self.audience.strip())): @@ -60,11 +61,26 @@ def __post_init__(self) -> None: ): raise ValueError("data-plane JWT key-subject bindings are invalid") + if self.environments_by_key_id is not None: + if (not isinstance(self.environments_by_key_id, Mapping) + or self.subjects_by_key_id is None + or set(self.environments_by_key_id) != set(self.keys_by_id)): + raise ValueError("JWT key-realm bindings require exact key-subject coverage") + if any(value not in {"paper", "sandbox", "live"} for value in self.environments_by_key_id.values()): + raise ValueError("JWT consumer realm must be paper, sandbox or live") + + @property + def consumer_environments(self) -> frozenset[str]: + if self.environments_by_key_id is None: + return frozenset({self.environment}) + return frozenset(self.environments_by_key_id.values()) + @classmethod def from_environment(cls) -> "DataPlaneSecurityConfig": try: keys = json.loads(os.environ["QDL_DATA_JWT_KEYS_JSON"]) key_subjects = json.loads(os.environ["QDL_DATA_JWT_KEY_SUBJECTS_JSON"]) + realms = json.loads(os.environ.get("QDL_DATA_JWT_KEY_ENVIRONMENTS_JSON", "null")) issuer = os.environ["QDL_DATA_JWT_ISSUER"] audience = os.environ["QDL_DATA_JWT_AUDIENCE"] except (KeyError, json.JSONDecodeError) as error: @@ -82,6 +98,7 @@ def from_environment(cls) -> "DataPlaneSecurityConfig": ) return cls( environment=os.environ.get("QDL_ENVIRONMENT", "paper").lower(), + environments_by_key_id=realms, issuer=issuer, audience=audience, keys_by_id={str(key): str(value) for key, value in keys.items()}, @@ -323,6 +340,7 @@ def __init__( keys_by_id=config.keys_by_id, algorithms=config.algorithms, max_lifetime_seconds=config.max_token_lifetime_seconds, + environments_by_key_id=config.environments_by_key_id, ) def authenticate(self, bearer_token: str, *, consumer_id: str) -> DataPlaneAccess: diff --git a/qdl/security/policy.py b/qdl/security/policy.py index 470e6224..54bfb3f4 100644 --- a/qdl/security/policy.py +++ b/qdl/security/policy.py @@ -60,6 +60,7 @@ def __init__( keys_by_id: Mapping[str, str | bytes], algorithms: Sequence[str] = ("RS256", "ES256"), max_lifetime_seconds: int = 900, + environments_by_key_id: Mapping[str, str] | None = None, ): if not issuer or not audience or not keys_by_id: raise ValueError("issuer, audience and at least one verification key are required") @@ -70,6 +71,12 @@ def __init__( self._keys = dict(keys_by_id) self._algorithms = tuple(algorithms) self._max_lifetime = max_lifetime_seconds + if environments_by_key_id is not None: + if set(environments_by_key_id) != set(keys_by_id) or any( + value not in {"paper", "sandbox", "live"} for value in environments_by_key_id.values() + ): + raise ValueError("JWT key-realm bindings must cover exactly the keyring") + self._environments_by_key_id = environments_by_key_id def verify(self, token: str, *, expected_environment: str) -> Principal: try: @@ -97,7 +104,11 @@ def verify(self, token: str, *, expected_environment: str) -> Principal: if expires_at <= issued_at or expires_at - issued_at > self._max_lifetime: raise PermissionError("workload token lifetime exceeds policy") environment = str(claims["environment"]) - if environment != expected_environment: + trusted_environment = ( + self._environments_by_key_id[key_id] + if self._environments_by_key_id is not None else expected_environment + ) + if environment != trusted_environment: raise PermissionError("workload token environment mismatch") roles = frozenset(str(role) for role in claims.get("roles", [])) unknown = roles - _ROLE_PERMISSIONS.keys() diff --git a/qdl_sdk/__init__.py b/qdl_sdk/__init__.py index ca23f806..c90e7f76 100644 --- a/qdl_sdk/__init__.py +++ b/qdl_sdk/__init__.py @@ -1,6 +1,6 @@ """Stable Python SDK V2 with typed public models and generated transports.""" -__version__ = "2.0.5" +__version__ = "2.0.6" from qdl_sdk.client import AsyncDataLayerClient, DataLayerClientV2, WarmupStreamSession from qdl_sdk.credentials import ( diff --git a/qdl_sdk/client.py b/qdl_sdk/client.py index 28b1ea3f..43e33b24 100644 --- a/qdl_sdk/client.py +++ b/qdl_sdk/client.py @@ -128,6 +128,24 @@ def _allows_quiet_execution_continuity(requirement: DataRequirement, quality) -> ) +def _view_refusal(row, code: str, detail: str) -> ContinuityError: + """Retain the rejected typed view, excluding payload and signed cursor.""" + return ContinuityError(code, detail, diagnostics={ + "origin": "sdk_query_validation", + "instrument_uid": row.instrument_uid, + "instrument_id": row.instrument_id, + "feed": row.feed.value, + "interval": row.interval, + "observed_at_ns": row.observed_at_ns, + "received_at_ns": row.received_at_ns, + "watermark_offset": row.watermark_offset, + "revision": row.revision, + "quality": row.quality.model_dump(mode="json"), + "source": row.source.model_dump(mode="json"), + "contract": row.contract.model_dump(mode="json"), + }) + + def _validate_query_payload( requirement: DataRequirement, payload: dict | WarmupResponse | SnapshotResponse, *, warmup: bool ) -> SnapshotResponse | WarmupResponse: @@ -165,18 +183,18 @@ def _validate_query_payload( for index, row in enumerate(rows): is_tail = index == len(rows) - 1 if row.instrument_uid != requirement.instrument_uid: - raise ContinuityError("CONFLICT", "query response instrument does not match requirement") + raise _view_refusal(row, "CONFLICT", "query response instrument does not match requirement") if row.feed.value != requirement.feed.value: - raise ContinuityError("CONFLICT", "query response feed does not match requirement") + raise _view_refusal(row, "CONFLICT", "query response feed does not match requirement") if requirement.interval is not None and row.interval != requirement.interval: - raise ContinuityError("CONFLICT", "query response interval does not match requirement") + raise _view_refusal(row, "CONFLICT", "query response interval does not match requirement") quality = row.quality if requirement.feed is Feed.BOOK_DELTA and ( not row.payload.sequence_verified or row.payload.book_generation < 1 ): - raise ContinuityError("OPEN_SEQUENCE_GAP", "book delta sequence is not verified") + raise _view_refusal(row, "OPEN_SEQUENCE_GAP", "book delta sequence is not verified") if quality.policy_id != requirement.source_policy_id: - raise ContinuityError( + raise _view_refusal(row, "CONFLICT", "query response source policy does not match requirement" ) state = quality.state.upper() @@ -188,12 +206,12 @@ def _validate_query_payload( and freshness_ms > requirement.max_freshness_ms and requirement.effective_event_recency_policy.value in {"BLOCK", "PAUSE"} ): - raise ContinuityError("DATA_STALE", "query response exceeds freshness policy") + raise _view_refusal(row, "DATA_STALE", "query response exceeds freshness policy") if ( is_tail and quality.provider_session_state in {"STALE", "DISCONNECTED", "UNKNOWN"} ): - raise ContinuityError( + raise _view_refusal(row, "DATA_STALE", "query response provider session is not live" ) if is_tail and requirement.max_session_liveness_ms is not None and ( @@ -202,33 +220,33 @@ def _validate_query_payload( or quality.provider_session_liveness_ms > requirement.max_session_liveness_ms ): - raise ContinuityError( + raise _view_refusal(row, "DATA_STALE", "query response provider session exceeds its policy" ) if quality.gap_open and requirement.gap_policy.value in {"BLOCK", "PAUSE"}: - raise ContinuityError("OPEN_SEQUENCE_GAP", "query response has an open gap") + raise _view_refusal(row, "OPEN_SEQUENCE_GAP", "query response has an open gap") if is_tail and state in { "STALE", "OFFLINE", "UNAVAILABLE" } and requirement.stale_policy.value in { "BLOCK", "PAUSE", }: - raise ContinuityError("DATA_STALE", f"query response quality state is {state}") + raise _view_refusal(row, "DATA_STALE", f"query response quality state is {state}") if ( is_tail and requirement.consumer_grade is Grade.EXECUTION and not quality.execution_eligible and not _allows_quiet_execution_continuity(requirement, quality) ): - raise ContinuityError( + raise _view_refusal(row, "SOURCE_NON_AUTHORITATIVE", "execution-grade response is not execution eligible", ) if requirement.require_full_coverage and not quality.complete: - raise ContinuityError("PARTIAL_RESULT", "query response quality is incomplete") + raise _view_refusal(row, "PARTIAL_RESULT", "query response quality is incomplete") if requirement.feed is Feed.BAR and requirement.require_final_bars: lifecycle = str(getattr(row.payload, "lifecycle", "")).upper() if lifecycle not in {"FINAL", "REVISED"}: - raise ContinuityError("DATA_NOT_READY", "bar response is not final") + raise _view_refusal(row, "DATA_NOT_READY", "bar response is not final") if requirement.feed is Feed.BAR: opens = [int(row.payload.open_time_ns) for row in rows] if opens != sorted(set(opens)): diff --git a/rust/qdl-kafka/Cargo.toml b/rust/qdl-kafka/Cargo.toml index 39d5b790..36ee947f 100644 --- a/rust/qdl-kafka/Cargo.toml +++ b/rust/qdl-kafka/Cargo.toml @@ -41,3 +41,6 @@ path = "src/bin/qdl-phase91-canary-rehearsal.rs" [[bin]] name = "qdl-phase92-primary-rehearsal" path = "src/bin/qdl-phase92-primary-rehearsal.rs" + +[dev-dependencies] +tokio = { workspace = true, features = ["test-util"] } diff --git a/rust/qdl-kafka/src/bin/qdl-native-raw-ingestor.rs b/rust/qdl-kafka/src/bin/qdl-native-raw-ingestor.rs index 72096851..7dd12ea2 100644 --- a/rust/qdl-kafka/src/bin/qdl-native-raw-ingestor.rs +++ b/rust/qdl-kafka/src/bin/qdl-native-raw-ingestor.rs @@ -16,7 +16,8 @@ use futures_util::stream::FuturesUnordered; use futures_util::{SinkExt, StreamExt}; use prost::Message as ProstMessage; use qdl_contracts::qdl::provider::v1::{ - CaptureBoundary, RawProviderEnvelope, TransportCompression, TransportProtocol, + CaptureBoundary, QuarantineReason, QuarantineRecord, RawProviderEnvelope, TransportCompression, + TransportProtocol, }; use qdl_core::backoff::BackoffPolicy; use qdl_core::binance::{ @@ -33,9 +34,10 @@ use qdl_core::okx::{ }; use qdl_core::transport::{DurableRecord, RetryClass}; use qdl_kafka::{ - shutdown_signal, FencedKafkaSink, KafkaTlsConfig, KafkaTransportConfig, KafkaTransportError, - PendingKafkaAppend, + shutdown_signal, FencedKafkaSink, KafkaEventSource, KafkaTlsConfig, KafkaTransportConfig, + KafkaTransportError, PendingKafkaAppend, }; +use qdl_realtime_core::L2_RESNAPSHOT_REQUIRED_V1; use qdl_venue_core::authority::{AuthorityMode, AuthorityRecord, PublicationContext, SinkTarget}; use qdl_venue_core::backpressure::DeliveryClass; use serde::{Deserialize, Serialize}; @@ -205,6 +207,13 @@ impl RawBinding { } } +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct BookResyncConfig { + quarantine_stream: String, + consumer_group_prefix: String, +} + #[derive(Debug, Deserialize)] #[serde(deny_unknown_fields)] struct IngestorConfig { @@ -236,6 +245,8 @@ struct IngestorConfig { latest_state_flush_ms: u64, authority: AuthorityRecord, bindings: Vec, + #[serde(default)] + book_resync: Option, } fn default_max_subscriptions_per_connection() -> usize { @@ -352,6 +363,16 @@ fn partition_okx_bindings( impl IngestorConfig { fn validate(&self) -> Result<(), String> { self.authority.validate()?; + if let Some(resync) = &self.book_resync { + if resync.quarantine_stream.trim().is_empty() + || resync.quarantine_stream == self.raw_stream + || resync.consumer_group_prefix.trim().is_empty() + { + return Err( + "book resync requires a separate quarantine topic and scoped group".into(), + ); + } + } if !matches!( self.authority.mode, AuthorityMode::RustShadow | AuthorityMode::RustPrimary @@ -1304,6 +1325,181 @@ async fn reserve(accepted: &AtomicU64, max_events: u64) -> bool { } } +struct BookResyncListener { + source: KafkaEventSource, + retry_after: Instant, + failures: u32, + next_checkpoint: Instant, + published_anchors: HashMap, +} + +fn book_resync_listener( + config: &IngestorConfig, + lane: &str, + bindings: &HashMap, +) -> Result, KafkaTransportError> { + let Some(resync) = config.book_resync.as_ref() else { + return Ok(None); + }; + if bindings.is_empty() || bindings.values().any(|b| b.feed != RawFeed::Book) { + return Ok(None); + } + let mut transport = kafka_config(lane).map_err(KafkaTransportError::Configuration)?; + transport.group_id = format!("{}-{lane}", resync.consumer_group_prefix); + transport.client_id = format!("{}-resync-{lane}", transport.client_id); + KafkaEventSource::new_live_control(&transport, &[&resync.quarantine_stream]).map(|source| { + Some(BookResyncListener { + source, + retry_after: Instant::now(), + failures: 0, + next_checkpoint: Instant::now(), + published_anchors: HashMap::new(), + }) + }) +} + +fn matches_book_resync( + record: &DurableRecord, + config: &IngestorConfig, + bindings: &HashMap, + session_id: &str, + generation: u64, + now_ns: i64, +) -> bool { + let Ok(request) = QuarantineRecord::decode(record.payload.as_slice()) else { + return false; + }; + if request.safe_summary != L2_RESNAPSHOT_REQUIRED_V1 + || request.reason != QuarantineReason::SequenceGap as i32 + { + return false; + } + let Some(raw) = request.raw.as_ref() else { + return false; + }; + let Some(binding) = bindings + .values() + .find(|b| b.subscription_id == raw.subscription_id) + else { + return false; + }; + let Some(l2) = binding.l2.as_ref() else { + return false; + }; + let max_age_ns = l2 + .snapshot_refresh_seconds + .unwrap_or(0) + .saturating_mul(1_000_000_000) as i64; + if binding.feed != RawFeed::Book + || raw.source_session_id != session_id + || raw.connection_generation != generation + || raw.test_provenance + || raw.provider != binding.provider + || raw.venue != binding.venue + || raw.market != binding.market + || raw.product_type != binding.product_type + || raw.native_symbol != binding.native_symbol + || raw.native_channel != binding.native_channel + || raw.adapter_version != binding.adapter_version + || raw.instrument_catalog_revision != binding.instrument_catalog_revision + || raw.config_revision != config.config_revision + || raw.lease_epoch != config.lease_epoch + || raw.partition_plan_epoch != config.partition_plan_epoch + || raw.authority_revision != config.authority.revision + || raw.received_at_ns <= 0 + || request.quarantined_at_ns < raw.received_at_ns + || now_ns < request.quarantined_at_ns + || now_ns.saturating_sub(raw.received_at_ns) > max_age_ns + { + return false; + } + let mut evidence = Sha256::new(); + evidence.update(raw.encode_to_vec()); + evidence.update(request.reason.to_be_bytes()); + let mut id = Sha256::new(); + id.update(&raw.capture_id); + id.update(L2_RESNAPSHOT_REQUIRED_V1.as_bytes()); + request.evidence_sha256 == evidence.finalize().as_slice() + && record.event_id == id.finalize().as_slice() + && raw.raw_frame_sha256 == Sha256::digest(&raw.raw_frame_bytes).as_slice() +} + +// A request based on a pre-anchor delta is already being repaired. Reconnecting +// here would throw away the just-published REST bootstrap on every generation. +fn snapshot_satisfies_resync( + record: &DurableRecord, + anchors: &HashMap, +) -> bool { + let Ok(request) = QuarantineRecord::decode(record.payload.as_slice()) else { + return false; + }; + let Some(raw) = request.raw else { return false }; + anchors + .get(&raw.subscription_id) + .is_some_and(|(session, generation, receipt)| { + session == &raw.source_session_id + && *generation == raw.connection_generation + && *receipt >= raw.received_at_ns + }) +} + +// Hints are reissued and session-fenced, not market-data acknowledgements. +// Coalesce offset submissions so a quarantine flood cannot queue one broker +// commit request per rejected frame in librdkafka. +fn claim_hint_checkpoint(next: &mut Instant, now: Instant) -> bool { + if now < *next { + return false; + } + *next = now + Duration::from_secs(1); + true +} + +async fn book_resync_requested( + listener: &mut Option, + config: &IngestorConfig, + bindings: &HashMap, + session: &str, + generation: u64, +) -> Result> { + let Some(listener) = listener else { + return std::future::pending().await; + }; + tokio::time::sleep_until(listener.retry_after).await; + let received = listener.source.next().await; + let (record, _) = match received { + Ok(record) => record, + Err(error) => { + listener.failures = listener.failures.saturating_add(1); + listener.retry_after = + Instant::now() + Duration::from_secs(u64::from(listener.failures).min(30)); + eprintln!( + "{}", + json!({"event":"qdl_book_resync_receive_failed", "error":error.to_string()}) + ); + return Ok(false); + } + }; + let requested = matches_book_resync(&record, config, bindings, session, generation, now_ns()?) + && !snapshot_satisfies_resync(&record, &listener.published_anchors); + // A crash before reconnect is recovered by the core's bounded reissue. + // Checkpoint failure may replay this hint, but not across a new WS session. + if !claim_hint_checkpoint(&mut listener.next_checkpoint, Instant::now()) { + return Ok(requested); + } + if let Err(error) = listener.source.checkpoint_hint() { + listener.failures = listener.failures.saturating_add(1); + listener.retry_after = + Instant::now() + Duration::from_secs(u64::from(listener.failures).min(30)); + eprintln!( + "{}", + json!({"event":"qdl_book_resync_checkpoint_failed", "error":error.to_string()}) + ); + } else { + listener.failures = 0; + } + Ok(requested) +} + async fn run_binance_connection( config: Arc, route: BinanceRoute, @@ -1346,6 +1542,7 @@ async fn run_binance_connection( .ok_or("Binance market WebSocket URL is required")?, }; let publisher = RawPublisher::new(&config, &lane)?; + let mut resync_listener = book_resync_listener(&config, &lane, &bindings)?; let snapshot_client = reqwest::Client::builder() .connect_timeout(Duration::from_secs(3)) .timeout(Duration::from_secs(5)) @@ -1471,6 +1668,12 @@ async fn run_binance_connection( .await .map_err(|error| std::io::Error::new(ErrorKind::Other, error))? { + let anchor_key = frame.binding.subscription_id.clone(); + let anchor = ( + frame.session_id.clone(), + frame.generation, + frame.received_at_ns, + ); if !publish_pending_frame( frame, &mut PendingPublishWindow { @@ -1489,6 +1692,9 @@ async fn run_binance_connection( exhausted = true; break; } + if let Some(listener) = resync_listener.as_mut() { + listener.published_anchors.insert(anchor_key, anchor); + } } } while !exhausted && !should_stop(&stopped, &accepted, config.max_events, expires) { @@ -1512,8 +1718,16 @@ async fn run_binance_connection( socket.next(), ); tokio::pin!(read); + // Control backlog must not take priority over socket and publish progress. let outcome = tokio::select! { - biased; + requested = book_resync_requested(&mut resync_listener, &config, &bindings, &session_id, generation) => { + if requested? { + eprintln!("{{\"event\":\"qdl_native_book_resync_requested\",\"session\":\"{}\",\"generation\":{}}}", session_id, generation); + disconnected = true; + break; + } + continue; + } _ = latest_tick.tick(), if !latest.is_empty() => { if !flush_latest_concurrent( &mut latest, @@ -1546,6 +1760,8 @@ async fn run_binance_connection( .await .map_err(|error| std::io::Error::new(ErrorKind::Other, error))? { + let anchor_key = frame.binding.subscription_id.clone(); + let anchor = (frame.session_id.clone(), frame.generation, frame.received_at_ns); if !publish_pending_frame( frame, &mut PendingPublishWindow { @@ -1564,6 +1780,9 @@ async fn run_binance_connection( exhausted = true; break; } + if let Some(listener) = resync_listener.as_mut() { + listener.published_anchors.insert(anchor_key, anchor); + } } failures = 0; continue; @@ -1841,6 +2060,17 @@ async fn run_okx_service( .into_iter() .map(|binding| (binding.key(), binding)) .collect(); + let mut resync_listener = book_resync_listener( + &config, + &format!( + "okx-{}-{shard_index:03}", + match service { + OkxService::Public => "public", + OkxService::Business => "business", + } + ), + &binding_map, + )?; let backoff = BackoffPolicy { initial_ms: 250, maximum_ms: 30_000, @@ -1982,8 +2212,16 @@ async fn run_okx_service( reader.next(), ); tokio::pin!(read); + // Control backlog must not take priority over socket and publish progress. let outcome = tokio::select! { - biased; + requested = book_resync_requested(&mut resync_listener, &config, &binding_map, &session_id, generation) => { + if requested? { + eprintln!("{{\"event\":\"qdl_native_book_resync_requested\",\"session\":\"{}\",\"generation\":{}}}", session_id, generation); + disconnected = true; + break; + } + continue; + } _ = latest_tick.tick(), if !latest.is_empty() => { if !flush_latest_concurrent( &mut latest, @@ -3002,3 +3240,320 @@ mod tests { fs::remove_dir_all(directory).unwrap(); } } + +#[cfg(test)] +mod resync_tests { + use super::*; + + fn fixture( + okx: bool, + ) -> ( + IngestorConfig, + HashMap, + RawProviderEnvelope, + ) { + let b = RawBinding { + provider: if okx { "OKX_DIRECT" } else { "BINANCE_DIRECT" }.into(), + venue: if okx { "OKX" } else { "BINANCE" }.into(), + market: if okx { "SWAP" } else { "USDM" }.into(), + product_type: "PERPETUAL".into(), + native_symbol: if okx { "BNB-USDT-SWAP" } else { "BNBUSDT" }.into(), + native_channel: if okx { "books" } else { "bnbusdt@depth@100ms" }.into(), + subscription_id: "book-test".into(), + adapter_version: "2.0.0".into(), + instrument_catalog_revision: 3, + feed: RawFeed::Book, + delivery_class: DeliveryClass::Lossless, + l2: Some(RawL2Config { + provider_protocol: if okx { + "OKX_PUBLIC_BOOKS" + } else { + "BINANCE_DIFF_DEPTH" + } + .into(), + depth_per_side: 100, + rest_snapshot_url: None, + snapshot_refresh_seconds: Some(30), + }), + mark_index_target: None, + }; + let raw = RawProviderEnvelope { + provider: b.provider.clone(), + venue: b.venue.clone(), + market: b.market.clone(), + product_type: b.product_type.clone(), + native_symbol: b.native_symbol.clone(), + native_channel: b.native_channel.clone(), + subscription_id: b.subscription_id.clone(), + adapter_version: b.adapter_version.clone(), + instrument_catalog_revision: 3, + source_session_id: "test-book-lane-7".into(), + connection_generation: 7, + config_revision: 9, + lease_epoch: 1, + authority_revision: 2, + partition_plan_epoch: 1, + received_at_ns: 1_000_000_000_000, + raw_frame_bytes: b"{}".to_vec(), + raw_frame_sha256: Sha256::digest(b"{}").to_vec(), + capture_id: vec![1; 16], + ..Default::default() + }; + let config = IngestorConfig { + runtime: if okx { + ProviderRuntime::Okx + } else { + ProviderRuntime::Binance + }, + websocket_url: "wss://unit.test".into(), + business_websocket_url: None, + market_websocket_url: None, + raw_stream: "unit.raw".into(), + shard_id: "unit".into(), + lease_epoch: 1, + partition_plan_epoch: 1, + config_revision: 9, + heartbeat_seconds: 10, + max_events: 0, + max_runtime_seconds: 0, + metrics_every_events: 100, + generation_state_path: "/tmp/unit".into(), + session_liveness_dir: "/tmp/unit-session".into(), + session_liveness_write_interval_ms: 1000, + max_inflight_publishes: 16, + max_subscriptions_per_connection: 100, + latest_state_flush_ms: 50, + authority: AuthorityRecord { + schema: "qdl.authority-record.v1".into(), + slice_id: "unit".into(), + revision: 2, + mode: AuthorityMode::RustShadow, + candidate_image_digest: format!("sha256:{}", "0".repeat(64)), + capability_manifest_digest: "0".repeat(64), + contract_digest: "0".repeat(64), + partition_plan_digest: "0".repeat(64), + public_write_allowed: false, + legacy_write_allowed: false, + approved_by: "unit-test".into(), + effective_at_ns: 1, + }, + bindings: vec![b.clone()], + book_resync: None, + }; + (config, HashMap::from([(b.key(), b)]), raw) + } + + fn request(raw: RawProviderEnvelope) -> DurableRecord { + let mut evidence = Sha256::new(); + evidence.update(raw.encode_to_vec()); + evidence.update((QuarantineReason::SequenceGap as i32).to_be_bytes()); + let mut id = Sha256::new(); + id.update(&raw.capture_id); + id.update(L2_RESNAPSHOT_REQUIRED_V1.as_bytes()); + let record = QuarantineRecord { + quarantined_at_ns: raw.received_at_ns, + raw: Some(raw), + reason: QuarantineReason::SequenceGap as i32, + safe_summary: L2_RESNAPSHOT_REQUIRED_V1.into(), + retry_count: 0, + evidence_sha256: evidence.finalize().to_vec(), + }; + DurableRecord { + stream: "unit.quarantine".into(), + partition_key: "unit".into(), + event_id: id.finalize().to_vec(), + payload: record.encode_to_vec(), + accepted_at_ns: record.quarantined_at_ns, + } + } + + #[test] + fn book_resync_accepts_current_lane_and_rejects_superseded_replay() { + for okx in [false, true] { + let (config, bindings, raw) = fixture(okx); + let record = request(raw.clone()); + assert!(matches_book_resync( + &record, + &config, + &bindings, + &raw.source_session_id, + 7, + raw.received_at_ns + 1 + )); + assert!(!matches_book_resync( + &record, + &config, + &bindings, + "next-session", + 8, + raw.received_at_ns + 1 + )); + assert!(!matches_book_resync( + &record, + &config, + &bindings, + &raw.source_session_id, + 8, + raw.received_at_ns + 1 + )); + assert!(!matches_book_resync( + &record, + &config, + &bindings, + &raw.source_session_id, + 7, + raw.received_at_ns - 1 + )); + assert!(!matches_book_resync( + &record, + &config, + &bindings, + &raw.source_session_id, + 7, + raw.received_at_ns + 30_000_000_001 + )); + } + } + + #[test] + fn book_resync_rejects_wrong_identity_fence_and_tampering() { + for okx in [false, true] { + let (config, bindings, raw) = fixture(okx); + for field in [ + "provider", + "venue", + "market", + "product", + "symbol", + "channel", + "subscription", + "adapter", + "catalog", + "config", + "authority", + "lease", + "partition", + "test", + ] { + let mut changed = raw.clone(); + match field { + "provider" => changed.provider.push('x'), + "venue" => changed.venue.push('x'), + "market" => changed.market.push('x'), + "product" => changed.product_type.push('x'), + "symbol" => changed.native_symbol.push('x'), + "channel" => changed.native_channel.push('x'), + "subscription" => changed.subscription_id.push('x'), + "adapter" => changed.adapter_version.push('x'), + "catalog" => changed.instrument_catalog_revision += 1, + "config" => changed.config_revision += 1, + "authority" => changed.authority_revision += 1, + "lease" => changed.lease_epoch += 1, + "partition" => changed.partition_plan_epoch += 1, + "test" => changed.test_provenance = true, + _ => unreachable!(), + } + assert!( + !matches_book_resync( + &request(changed), + &config, + &bindings, + &raw.source_session_id, + 7, + raw.received_at_ns + 1 + ), + "{field}" + ); + } + let mut record = request(raw.clone()); + record.event_id[0] ^= 1; + assert!(!matches_book_resync( + &record, + &config, + &bindings, + &raw.source_session_id, + 7, + raw.received_at_ns + 1 + )); + let mut record = request(raw.clone()); + let mut decoded = QuarantineRecord::decode(record.payload.as_slice()).unwrap(); + decoded.evidence_sha256[0] ^= 1; + record.payload = decoded.encode_to_vec(); + assert!(!matches_book_resync( + &record, + &config, + &bindings, + &raw.source_session_id, + 7, + raw.received_at_ns + 1 + )); + let mut nonbook = bindings.clone(); + for binding in nonbook.values_mut() { + binding.feed = RawFeed::Trade; + } + assert!(!matches_book_resync( + &request(raw.clone()), + &config, + &nonbook, + &raw.source_session_id, + 7, + raw.received_at_ns + 1 + )); + } + } + + #[test] + fn book_resync_late_bootstrap_hint_does_not_reconnect_fresh_anchor() { + let (_, _, raw) = fixture(false); + let record = request(raw.clone()); + let mut anchors = HashMap::new(); + assert!(!snapshot_satisfies_resync(&record, &anchors)); + anchors.insert( + raw.subscription_id.clone(), + ( + raw.source_session_id.clone(), + raw.connection_generation, + raw.received_at_ns + 1, + ), + ); + assert!(snapshot_satisfies_resync(&record, &anchors)); + let mut current = raw.clone(); + current.received_at_ns += 2; + assert!(!snapshot_satisfies_resync(&request(current), &anchors)); + let mut other = raw.clone(); + other.connection_generation += 1; + assert!(!snapshot_satisfies_resync(&request(other), &anchors)); + let mut other = raw.clone(); + other.source_session_id.push('x'); + assert!(!snapshot_satisfies_resync(&request(other), &anchors)); + let mut other = raw; + other.subscription_id.push('x'); + assert!(!snapshot_satisfies_resync(&request(other), &anchors)); + } + + #[tokio::test(start_paused = true)] + async fn book_resync_checkpoint_flood_is_coalesced() { + let mut next = Instant::now(); + assert!(claim_hint_checkpoint(&mut next, Instant::now())); + for _ in 0..40_000 { + assert!(!claim_hint_checkpoint(&mut next, Instant::now())); + } + tokio::time::advance(Duration::from_millis(999)).await; + assert!(!claim_hint_checkpoint(&mut next, Instant::now())); + tokio::time::advance(Duration::from_millis(1)).await; + assert!(claim_hint_checkpoint(&mut next, Instant::now())); + assert!(!claim_hint_checkpoint(&mut next, Instant::now())); + } + + #[tokio::test(start_paused = true)] + async fn book_resync_disabled_has_no_poll_or_connection_side_effect() { + let (config, bindings, raw) = fixture(true); + let mut listener = None; + assert!(tokio::time::timeout( + Duration::from_millis(1), + book_resync_requested(&mut listener, &config, &bindings, &raw.source_session_id, 7) + ) + .await + .is_err()); + } +} diff --git a/rust/qdl-kafka/src/bin/qdl-realtime-core.rs b/rust/qdl-kafka/src/bin/qdl-realtime-core.rs index d686a495..851f78ab 100644 --- a/rust/qdl-kafka/src/bin/qdl-realtime-core.rs +++ b/rust/qdl-kafka/src/bin/qdl-realtime-core.rs @@ -35,6 +35,8 @@ struct RuntimeConfig { metrics_every_batches: u64, #[serde(default)] strict_subscription_scope: bool, + #[serde(default)] + book_resync_quarantine: bool, } impl RuntimeConfig { @@ -213,6 +215,18 @@ fn authority_mode_name(mode: AuthorityMode) -> &'static str { } } +// A ready receive must not extend an expired batch. Tokio timeout alone polls +// the input first, so an always-ready raw queue needs this explicit guard. +async fn receive_batch_input( + deadline: tokio::time::Instant, + next: impl std::future::Future, +) -> Option { + if tokio::time::Instant::now() >= deadline { + return None; + } + tokio::time::timeout_at(deadline, next).await.ok() +} + async fn run_generation( config: &RuntimeConfig, generation: u64, @@ -261,6 +275,9 @@ async fn run_generation( // describes its own interval rather than all of history. let mut raw_age = SpanSummary::default(); let mut commit_latency = SpanSummary::default(); + let mut collect_latency = SpanSummary::default(); + let mut normalize_latency = SpanSummary::default(); + let mut raw_age_by_partition: BTreeMap<(String, i32), SpanSummary> = BTreeMap::new(); let mut ignored_out_of_scope = 0_u64; let mut scope_quarantines = 0_u64; let mut batches = 0_u64; @@ -306,18 +323,25 @@ async fn run_generation( Err(error) => return Err(error.into()), } }; + let collect_started = tokio::time::Instant::now(); + let batch_deadline = collect_started + Duration::from_millis(config.batch_wait_ms); let mut inputs = vec![first]; while inputs.len() < config.batch_size && (config.max_events == 0 || processed + (inputs.len() as u64) < config.max_events) { - match tokio::time::timeout(Duration::from_millis(config.batch_wait_ms), bridge.next()) - .await - { - Ok(Ok(input)) => { + let received = tokio::select! { + biased; + result = &mut shutdown => break 'service result?.as_str(), + result = receive_batch_input(batch_deadline, bridge.next()) => result, + }; + match received { + Some(Ok(input)) => { receive_failures = 0; inputs.push(input); } - Ok(Err(error)) if should_retry_receive_in_generation(&error, receive_failures) => { + Some(Err(error)) + if should_retry_receive_in_generation(&error, receive_failures) => + { receive_failures = receive_failures.saturating_add(1); eprintln!( "{}", @@ -331,11 +355,13 @@ async fn run_generation( ); break; } - Ok(Err(error)) => return Err(error.into()), - Err(_) => break, + Some(Err(error)) => return Err(error.into()), + None => break, } } + collect_latency.observe(collect_started.elapsed().as_nanos().min(i64::MAX as u128) as i64); + let normalize_started = tokio::time::Instant::now(); let normalized_at_ns = now_ns()?; let mut outputs = vec![]; for input in &inputs { @@ -349,8 +375,16 @@ async fn run_generation( // catches. Every four-quantity report before this printed // `received -> published = 0 ms`, which was two copies of one field // subtracting to zero, not a measurement. - raw_age.observe(normalized_at_ns.saturating_sub(raw.received_at_ns)); - let result = if !is_approved_subscription(&raw, &approved_subscriptions) { + let age_ns = normalized_at_ns.saturating_sub(raw.received_at_ns); + raw_age.observe(age_ns); + raw_age_by_partition + .entry(( + input.cursor.stream.clone(), + input.cursor.transport_partition, + )) + .or_default() + .observe(age_ns); + let mut result = if !is_approved_subscription(&raw, &approved_subscriptions) { if config.strict_subscription_scope { scope_quarantines = scope_quarantines.saturating_add(1); core.quarantine_raw( @@ -392,6 +426,14 @@ async fn run_generation( Err(error) => return Err(error.into()), } }; + if config.book_resync_quarantine + && is_approved_subscription(&raw, &approved_subscriptions) + && raw.authority_revision == config.authority.revision + { + if let Some(request) = core.l2_resnapshot_request(&raw, normalized_at_ns) { + result.quarantines.push(request); + } + } canonical += result.canonical.len() as u64; quarantines += result.quarantines.len() as u64; duplicates += result.duplicates as u64; @@ -426,6 +468,8 @@ async fn run_generation( }); } } + normalize_latency + .observe(normalize_started.elapsed().as_nanos().min(i64::MAX as u128) as i64); let commit_started_ns = now_ns()?; bridge.commit(&inputs, &outputs).await?; commit_latency.observe(now_ns()?.saturating_sub(commit_started_ns)); @@ -448,10 +492,18 @@ async fn run_generation( "batches": batches, "raw_age_ms": raw_age.report(), "commit_ms": commit_latency.report(), + "collect_ms": collect_latency.report(), + "normalize_ms": normalize_latency.report(), + "raw_age_by_partition_ms": raw_age_by_partition.iter().map(|((topic, partition), age)| { + json!({"topic": topic, "partition": partition, "age": age.report()}) + }).collect::>(), }))? ); raw_age.reset(); commit_latency.reset(); + collect_latency.reset(); + normalize_latency.reset(); + raw_age_by_partition.clear(); } }; bridge.unsubscribe(); @@ -527,6 +579,85 @@ async fn main() -> Result<(), RuntimeError> { mod tests { use super::*; + #[tokio::test(start_paused = true)] + async fn batch_deadline_does_not_slide_with_continuous_arrivals() { + let started = tokio::time::Instant::now(); + let deadline = started + Duration::from_millis(25); + let mut received = Vec::new(); + for offset in 1..256 { + let value = receive_batch_input(deadline, async { + tokio::time::sleep(Duration::from_millis(10)).await; + offset + }) + .await; + match value { + Some(value) => received.push(value), + None => break, + } + } + assert_eq!(received, vec![1, 2]); + assert!(started.elapsed() <= Duration::from_millis(26)); + } + + #[tokio::test(start_paused = true)] + async fn batch_deadline_idle_wait_uses_remaining_budget() { + let started = tokio::time::Instant::now(); + let deadline = started + Duration::from_millis(25); + tokio::time::sleep(Duration::from_millis(20)).await; + assert_eq!( + receive_batch_input(deadline, std::future::pending::()).await, + None + ); + assert!(started.elapsed() <= Duration::from_millis(26)); + } + + #[tokio::test(start_paused = true)] + async fn batch_deadline_expired_does_not_consume_ready_record() { + let deadline = tokio::time::Instant::now(); + let mut polled = false; + let result = receive_batch_input(deadline, async { + polled = true; + 42 + }) + .await; + assert_eq!(result, None); + assert!(!polled); + } + + #[tokio::test(start_paused = true)] + async fn batch_deadline_preserves_burst_order_and_receive_error() { + let deadline = tokio::time::Instant::now() + Duration::from_millis(25); + let mut offsets = Vec::new(); + for offset in 0..256 { + offsets.push( + receive_batch_input(deadline, std::future::ready(offset)) + .await + .unwrap(), + ); + } + assert_eq!(offsets, (0..256).collect::>()); + assert_eq!( + receive_batch_input(deadline, std::future::ready(Err::<(), _>("receive"))).await, + Some(Err("receive")) + ); + } + + #[tokio::test(start_paused = true)] + async fn batch_deadline_cancellation_does_not_eat_next_record() { + let deadline = tokio::time::Instant::now() + Duration::from_millis(25); + let (tx, mut rx) = tokio::sync::mpsc::channel(1); + tokio::select! { + biased; + _ = tokio::time::sleep(Duration::from_millis(5)) => {}, + _ = receive_batch_input(deadline, rx.recv()) => panic!("input should be waiting"), + } + tx.send(42).await.unwrap(); + assert_eq!( + receive_batch_input(deadline, rx.recv()).await, + Some(Some(42)) + ); + } + #[test] fn runtime_retries_only_retryable_or_capacity_transport_errors() { assert!(should_retry_transport(RetryClass::Retryable)); diff --git a/rust/qdl-kafka/src/lib.rs b/rust/qdl-kafka/src/lib.rs index 307f0dc5..22568d63 100644 --- a/rust/qdl-kafka/src/lib.rs +++ b/rust/qdl-kafka/src/lib.rs @@ -908,6 +908,23 @@ impl KafkaEventSource { pub fn new( config: &KafkaTransportConfig, topics: &[&str], + ) -> Result { + Self::with_offset_reset(config, topics, "earliest") + } + + /// Control notifications are session-fenced and reissued while unresolved. + /// A new control group starts live; an existing group resumes its checkpoint. + pub fn new_live_control( + config: &KafkaTransportConfig, + topics: &[&str], + ) -> Result { + Self::with_offset_reset(config, topics, "latest") + } + + fn with_offset_reset( + config: &KafkaTransportConfig, + topics: &[&str], + reset: &str, ) -> Result { if topics.is_empty() || topics.iter().any(|topic| topic.trim().is_empty()) { return Err(KafkaTransportError::Configuration( @@ -915,7 +932,7 @@ impl KafkaEventSource { )); } let mut client = config.client_config()?; - config.configure_group_consumer(&mut client); + config.configure_group_consumer_with_offset_reset(&mut client, reset); let consumer: StreamConsumer = client.create()?; consumer.subscribe(topics)?; Ok(Self { consumer }) @@ -970,6 +987,13 @@ impl KafkaEventSource { self.consumer.commit_consumer_state(CommitMode::Sync)?; Ok(()) } + + /// Only for idempotent, periodically reissued control hints. Submission is + /// not a durable acknowledgement and must never acknowledge market data. + pub fn checkpoint_hint(&self) -> Result<(), KafkaTransportError> { + self.consumer.commit_consumer_state(CommitMode::Async)?; + Ok(()) + } } #[cfg(test)] @@ -1180,3 +1204,141 @@ mod tests { )); } } + +#[cfg(test)] +mod control_broker_tests { + use super::*; + + async fn send(producer: &FutureProducer, topic: &str, payload: &str) { + producer + .send( + FutureRecord::to(topic) + .partition(0) + .key("test-control") + .payload(payload) + .headers(OwnedHeaders::new().insert(Header { + key: EVENT_ID_HEADER, + value: Some(payload.as_bytes()), + })), + Timeout::After(Duration::from_secs(5)), + ) + .await + .expect("isolated control delivery"); + } + + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] + #[ignore = "requires explicitly provisioned isolated readiness Kafka and disposable TLS"] + async fn live_control_committed_only_resume_and_cancel() { + let bootstrap = + std::env::var("QDL_CONTROL_TEST_BOOTSTRAP").expect("explicit isolated broker"); + assert_eq!(bootstrap, "qdl-readiness-kafka:9094"); + let root = std::env::var("QDL_CONTROL_TEST_TLS").expect("disposable TLS root"); + let topic = "readiness.control-test"; + let config = KafkaTransportConfig { + bootstrap_servers: bootstrap, + client_id: "readiness-control-test".into(), + group_id: "readiness-control-test-group".into(), + request_timeout: Duration::from_secs(10), + tls: KafkaTlsConfig { + ca_location: format!("{root}/ca.crt"), + certificate_location: format!("{root}/client.crt"), + key_location: format!("{root}/client.key"), + key_password: None, + }, + }; + let mut producer_config = config.client_config().unwrap(); + producer_config.set("transactional.id", "readiness-control-test-producer"); + let producer: FutureProducer = producer_config.create().unwrap(); + let timeout = Timeout::After(Duration::from_secs(10)); + producer.init_transactions(timeout).unwrap(); + producer.begin_transaction().unwrap(); + send(&producer, topic, "old-before-subscribe").await; + producer.commit_transaction(timeout).unwrap(); + + let source = KafkaEventSource::new_live_control(&config, &[topic]).unwrap(); + // position() may remain Invalid until the first delivered message. Prove + // startup with a bounded fresh-marker handshake instead of assuming it. + let deadline = tokio::time::Instant::now() + Duration::from_secs(15); + loop { + assert!( + tokio::time::Instant::now() < deadline, + "live startup handshake timeout" + ); + producer.begin_transaction().unwrap(); + send(&producer, topic, "startup-handshake").await; + producer.commit_transaction(timeout).unwrap(); + if let Ok(received) = + tokio::time::timeout(Duration::from_millis(250), source.next()).await + { + assert_eq!( + received.unwrap().0.payload, + b"startup-handshake", + "new control group must skip pre-subscription history" + ); + break; + } + } + // Drain only explicitly generated startup markers, never data under test. + while let Ok(received) = + tokio::time::timeout(Duration::from_millis(250), source.next()).await + { + assert_eq!(received.unwrap().0.payload, b"startup-handshake"); + } + producer.begin_transaction().unwrap(); + send(&producer, topic, "aborted-hint").await; + producer.abort_transaction(timeout).unwrap(); + producer.begin_transaction().unwrap(); + send(&producer, topic, "committed-hint").await; + producer.commit_transaction(timeout).unwrap(); + let (record, cursor) = tokio::time::timeout(Duration::from_secs(10), source.next()) + .await + .unwrap() + .unwrap(); + assert_eq!(record.payload, b"committed-hint"); + source.checkpoint_hint().unwrap(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + loop { + let committed = source.consumer.committed(timeout).unwrap(); + if committed.elements().iter().any(|p| { + p.topic() == topic && p.offset() == Offset::Offset(cursor.offset as i64 + 1) + }) { + break; + } + assert!( + tokio::time::Instant::now() < deadline, + "hint checkpoint not durable yet" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } + drop(source); + // Published while the receiver is down: an existing group must resume, + // not apply latest reset a second time and skip this hint. + producer.begin_transaction().unwrap(); + send(&producer, topic, "resume-hint").await; + producer.commit_transaction(timeout).unwrap(); + let resumed = KafkaEventSource::new_live_control(&config, &[topic]).unwrap(); + let (record, _) = tokio::time::timeout(Duration::from_secs(15), resumed.next()) + .await + .unwrap() + .unwrap(); + assert_eq!(record.payload, b"resume-hint"); + assert!( + tokio::time::timeout(Duration::from_millis(100), resumed.next()) + .await + .is_err() + ); + producer.begin_transaction().unwrap(); + send(&producer, topic, "after-cancelled-receive").await; + producer.commit_transaction(timeout).unwrap(); + let (record, _) = tokio::time::timeout(Duration::from_secs(10), resumed.next()) + .await + .unwrap() + .unwrap(); + assert_eq!(record.payload, b"after-cancelled-receive"); + assert!( + tokio::time::timeout(Duration::from_millis(200), resumed.next()) + .await + .is_err() + ); + } +} diff --git a/rust/qdl-realtime-core/src/lib.rs b/rust/qdl-realtime-core/src/lib.rs index 176e31da..c0818924 100644 --- a/rust/qdl-realtime-core/src/lib.rs +++ b/rust/qdl-realtime-core/src/lib.rs @@ -13,7 +13,7 @@ use qdl_core::canonical::{ TradeContext, TradeFixture, }; use qdl_core::l2_adapter::{BookPublication, BookTransition, L2BookAdapter}; -use qdl_core::l2_book::{BookIdentity, BookOutcome}; +use qdl_core::l2_book::{BookIdentity, BookOutcome, BookStatus}; use qdl_core::okx::expand_data_frame; use qdl_core::transport::DurableRecord; use qdl_provider_envelope::validate as validate_raw; @@ -70,12 +70,6 @@ impl L2Binding { .unwrap_or_else(|| self.snapshot_refresh_seconds.saturating_mul(1_000)) } - fn raw_receipt_is_expired(&self, received_at_ns: i64, processing_at_ns: i64) -> bool { - let max_age_ns = self.snapshot_refresh_seconds.saturating_mul(1_000_000_000); - let max_age_ns = i64::try_from(max_age_ns).unwrap_or(i64::MAX); - processing_at_ns.saturating_sub(received_at_ns) > max_age_ns - } - fn validate(&self, binding: &CoreBinding) -> Result<(), CoreError> { let materialized_snapshot_interval_ms = self.materialized_snapshot_interval_ms(); if !(1..=10_000).contains(&self.depth_per_side) @@ -400,6 +394,8 @@ struct MarkIndexPairState { index: Option, } +pub const L2_RESNAPSHOT_REQUIRED_V1: &str = "L2_RESNAPSHOT_REQUIRED_V1"; + pub struct RealtimeCore { config: RealtimeCoreConfig, bindings: BTreeMap, @@ -411,6 +407,7 @@ pub struct RealtimeCore { /// can be proved. Book frames never reach `ordering`, which is why this /// exists separately; see `process_l2`. l2_sessions: BTreeMap, + l2_resnapshot_requests: BTreeMap, ordering: OrderingTracker, seen_ids: HashSet>, seen_order: VecDeque>, @@ -455,12 +452,80 @@ impl RealtimeCore { mark_index_pairs: BTreeMap::new(), partition_sequences: BTreeMap::new(), l2_sessions: BTreeMap::new(), + l2_resnapshot_requests: BTreeMap::new(), ordering: OrderingTracker::new(4096), seen_ids: HashSet::new(), seen_order: VecDeque::new(), }) } + /// Called after processing a raw frame. The runtime commits this evidence + /// with that frame's offset; only its current acquisition lane may act on it. + pub fn l2_resnapshot_request( + &mut self, + raw: &RawProviderEnvelope, + processing_at_ns: i64, + ) -> Option { + let key = binding_key( + &raw.provider, + &raw.venue, + &raw.market, + &raw.product_type, + &raw.native_symbol, + &raw.native_channel, + ); + let binding = self.bindings.get(&key)?; + let l2 = binding.l2.as_ref()?; + let adapter = self.l2_adapters.get(&key)?; + if raw.subscription_id != binding.source_id + || raw.instrument_catalog_revision != binding.instrument_catalog_revision + || self.l2_sessions.get(&key) != Some(&raw.source_session_id) + || adapter.core().generation() != raw.connection_generation + { + return None; + } + if adapter.core().status() == BookStatus::Ready { + self.l2_resnapshot_requests.remove(&key); + return None; + } + let retry_ns = i64::try_from(l2.snapshot_refresh_seconds) + .ok()? + .checked_mul(1_000_000_000)?; + if self + .l2_resnapshot_requests + .get(&key) + .is_some_and(|(session, generation, sent)| { + session == &raw.source_session_id + && *generation == raw.connection_generation + && raw.received_at_ns.saturating_sub(*sent) < retry_ns + }) + { + return None; + } + self.l2_resnapshot_requests.insert( + key, + ( + raw.source_session_id.clone(), + raw.connection_generation, + raw.received_at_ns, + ), + ); + let mut record = self + .quarantine( + raw, + QuarantineReason::SequenceGap, + L2_RESNAPSHOT_REQUIRED_V1, + processing_at_ns, + ) + .quarantines + .pop()?; + let mut id = Sha256::new(); + id.update(&raw.capture_id); + id.update(L2_RESNAPSHOT_REQUIRED_V1.as_bytes()); + record.event_id = id.finalize().to_vec(); + Some(record) + } + pub fn process_bytes( &mut self, raw_bytes: &[u8], @@ -1020,24 +1085,17 @@ impl RealtimeCore { binding: &CoreBinding, raw: RawProviderEnvelope, payload: Value, - processing_at_ns: i64, + _processing_at_ns: i64, transport_offset: Option, ) -> ProcessBatch { let materialized_at_ns = raw.received_at_ns; let Some(l2) = binding.l2.as_ref() else { unreachable!("process_l2 only accepts an L2 binding") }; - if l2.raw_receipt_is_expired(raw.received_at_ns, processing_at_ns) { - if let Some(adapter) = self.l2_adapters.get_mut(binding_key) { - adapter.request_resync(raw.connection_generation); - } - return self.quarantine( - &raw, - QuarantineReason::SemanticInvalid, - "L2 raw receipt exceeded provider renewal bound; resync required", - processing_at_ns, - ); - } + // Processing backlog is not a provider continuity failure. Replay every + // validated book frame losslessly with its original source/receipt times; + // readers still enforce freshness. Dropping an aged delta and clearing + // the book strands a healthy OKX socket waiting for a new snapshot. let transport = match TransportProtocol::try_from(raw.transport_protocol) { Ok(value) => value, Err(_) => { @@ -1694,6 +1752,7 @@ mod tests { use super::{ CoreBinding, CoreError, L2Binding, L2ProviderProtocol, MarkIndexBinding, MarkIndexComponent, ProcessBatch, RealtimeCore, RealtimeCoreConfig, + L2_RESNAPSHOT_REQUIRED_V1, }; use prost::Message; use qdl_contracts::qdl::common::v1::{QuantityUnit, SourceRole}; @@ -1702,6 +1761,7 @@ mod tests { CaptureBoundary, QuarantineReason, QuarantineRecord, RawProviderEnvelope, TransportCompression, TransportProtocol, }; + use qdl_core::l2_book::BookStatus; use qdl_venue_core::ordering::SequencePolicy; use sha2::{Digest, Sha256}; @@ -2581,48 +2641,298 @@ mod tests { assert_eq!(evidence.reason, QuarantineReason::SequenceGap as i32); } - #[test] - fn stale_l2_raw_is_quarantined_without_redating_and_fresh_snapshot_recovers() { - let binding = binance_book_binding(); - let mut core = core(binding.clone(), true); - let stale = with_transport( - raw( - &binding, - br#"{"lastUpdateId":100,"bids":[["60000","1"]],"asks":[["60001","1"]]}"#, - 1, - ), - TransportProtocol::Http, + // Synthetic protocol fixtures: processing time is deliberately independent + // from provider and receipt time, as during Kafka backlog recovery. + fn delayed_book_frame( + binding: &CoreBinding, + snapshot: bool, + sequence: u64, + previous: u64, + generation: u64, + received_at_ns: i64, + bid_quantity: &str, + ) -> RawProviderEnvelope { + let time_ms = received_at_ns / 1_000_000; + let payload = if binding.venue == "OKX" { + serde_json::json!({ + "arg": {"channel": "books", "instId": binding.native_symbol}, + "action": if snapshot { "snapshot" } else { "update" }, + "data": [{"seqId": sequence.to_string(), + "prevSeqId": if snapshot { "-1".into() } else { previous.to_string() }, + "ts": time_ms.to_string(), + "bids": [["60000", bid_quantity, "0", "1"]], + "asks": [["60001", "1", "0", "1"]]}] + }) + } else if snapshot { + serde_json::json!({"lastUpdateId": sequence, "E": time_ms, + "bids": [["60000", bid_quantity]], "asks": [["60001", "1"]]}) + } else { + serde_json::json!({"s": binding.native_symbol, "U": previous, + "u": sequence, "pu": previous, "E": time_ms, + "b": [["60000", bid_quantity]], "a": [["60001", "1"]]}) + }; + let frame = raw_with_receipt( + binding, + &serde_json::to_vec(&payload).unwrap(), + generation, + received_at_ns, ); - let processing_at_ns = stale.received_at_ns + 30_000_000_001; - let rejected = core.process(stale, processing_at_ns).unwrap(); - assert!(rejected.canonical.is_empty()); - assert_eq!(rejected.quarantines.len(), 1); - let evidence = - QuarantineRecord::decode(rejected.quarantines[0].payload.as_slice()).unwrap(); - assert_eq!(evidence.reason, QuarantineReason::SemanticInvalid as i32); - assert_eq!(evidence.quarantined_at_ns, processing_at_ns); - assert!(evidence.safe_summary.contains("renewal bound")); + if snapshot && binding.venue == "BINANCE" { + with_transport(frame, TransportProtocol::Http) + } else { + frame + } + } - let mut fresh_delta = raw( - &binding, - br#"{"s":"BTCUSDT","U":99,"u":101,"pu":98,"E":1001,"b":[["60000","2"]],"a":[["60001","1"]]}"#, - 2, - ); - fresh_delta.received_at_ns = processing_at_ns; - let buffered = core.process(fresh_delta, processing_at_ns).unwrap(); - assert!(buffered.canonical.is_empty()); + #[test] + fn l2_resnapshot_cold_owner_retry_is_bounded_and_session_fenced() { + for binding in [okx_book_binding(), binance_book_binding()] { + let mut core = core(binding.clone(), true); + let received = 1_786_352_400_000_000_000; + let mut delta = delayed_book_frame(&binding, false, 101, 100, 7, received, "1"); + delta.subscription_id = binding.source_id.clone(); + core.process(delta.clone(), received).unwrap(); + let request = core.l2_resnapshot_request(&delta, received).unwrap(); + let decoded = QuarantineRecord::decode(request.payload.as_slice()).unwrap(); + assert_eq!(decoded.safe_summary, L2_RESNAPSHOT_REQUIRED_V1); + assert_eq!(decoded.raw.as_ref(), Some(&delta)); + assert_ne!(request.event_id, delta.capture_id); + assert!(core.l2_resnapshot_request(&delta, received + 1).is_none()); + delta.received_at_ns += 29_000_000_000; + assert!(core + .l2_resnapshot_request(&delta, delta.received_at_ns) + .is_none()); + delta.received_at_ns += 1_000_000_000; + assert!(core + .l2_resnapshot_request(&delta, delta.received_at_ns) + .is_some()); + delta.connection_generation -= 1; + assert!(core + .l2_resnapshot_request(&delta, delta.received_at_ns) + .is_none()); + delta.connection_generation += 1; + delta.source_session_id.push_str("-foreign"); + assert!(core + .l2_resnapshot_request(&delta, delta.received_at_ns) + .is_none()); + } + } - let mut fresh_snapshot = with_transport( - raw( - &binding, - br#"{"lastUpdateId":100,"bids":[["60000","2"]],"asks":[["60001","1"]]}"#, - 2, - ), - TransportProtocol::Http, - ); - fresh_snapshot.received_at_ns = processing_at_ns; - let recovered = core.process(fresh_snapshot, processing_at_ns).unwrap(); - assert_eq!(recovered.canonical.len(), 1); + #[test] + fn l2_resnapshot_ready_book_suppresses_and_real_gap_rearms_request() { + for binding in [okx_book_binding(), binance_book_binding()] { + let mut core = core(binding.clone(), true); + let received = 1_786_352_400_000_000_000; + let mut delta = delayed_book_frame(&binding, false, 101, 100, 7, received, "1"); + delta.subscription_id = binding.source_id.clone(); + core.process(delta.clone(), received).unwrap(); + assert!(core.l2_resnapshot_request(&delta, received).is_some()); + let mut snapshot = + delayed_book_frame(&binding, true, 100, 0, 7, received + 1_000_000, "1"); + snapshot.subscription_id = binding.source_id.clone(); + core.process(snapshot.clone(), snapshot.received_at_ns) + .unwrap(); + delta.received_at_ns += 1_500_000; + core.process(delta.clone(), delta.received_at_ns).unwrap(); + let mut good = + delayed_book_frame(&binding, false, 102, 101, 7, received + 2_000_000, "2"); + good.subscription_id = binding.source_id.clone(); + core.process(good.clone(), good.received_at_ns).unwrap(); + assert_eq!( + core.l2_adapters[&binding.key()].core().status(), + BookStatus::Ready + ); + assert!(core + .l2_resnapshot_request(&good, good.received_at_ns) + .is_none()); + let mut gap = + delayed_book_frame(&binding, false, 110, 109, 7, received + 3_000_000, "3"); + gap.subscription_id = binding.source_id.clone(); + assert!(!core + .process(gap.clone(), gap.received_at_ns) + .unwrap() + .quarantines + .is_empty()); + assert!(core + .l2_resnapshot_request(&gap, gap.received_at_ns) + .is_some()); + } + } + + #[test] + fn delayed_l2_contiguous_replay_preserves_times_state_and_lossless_publications() { + for binding in [okx_book_binding(), binance_book_binding()] { + let mut delayed = core(binding.clone(), true); + let mut timely = core(binding.clone(), true); + let receipt = 1_786_352_400_000_000_000; + let now = receipt + 60_000_000_000; + let frames = [ + delayed_book_frame(&binding, true, 100, 0, 7, receipt, "1"), + delayed_book_frame(&binding, false, 101, 100, 7, receipt + 1_000_000, "2"), + delayed_book_frame(&binding, false, 102, 101, 7, receipt + 2_000_000, "0"), + delayed_book_frame(&binding, false, 103, 102, 7, now, "3"), + ]; + for (offset, frame) in frames.into_iter().enumerate() { + let original_receipt = frame.received_at_ns; + let expected = timely + .process_at_transport_offset(frame.clone(), original_receipt, offset as u64) + .unwrap(); + let actual = delayed + .process_at_transport_offset(frame.clone(), now, offset as u64) + .unwrap(); + assert!(actual.quarantines.is_empty(), "{}", binding.venue); + assert_eq!(actual.canonical.len(), expected.canonical.len()); + for (record, expected) in actual.canonical.iter().zip(&expected.canonical) { + assert_eq!(record.payload, expected.payload); + assert_eq!(record.accepted_at_ns, original_receipt); + let event = EventEnvelope::decode(record.payload.as_slice()).unwrap(); + assert_eq!(event.received_at_ns, original_receipt); + assert_eq!(event.normalized_at_ns, original_receipt); + assert_eq!(event.published_at_ns, original_receipt); + assert_eq!(event.source_event_time_ns, original_receipt); + } + if offset == 1 || offset == 2 { + assert!( + !actual.canonical.is_empty(), + "delayed deltas cannot be skipped" + ); + } + if offset == 2 { + assert!(delayed.l2_adapters[&binding.key()] + .core() + .view() + .unwrap() + .bids + .is_empty()); + let duplicate = delayed + .process_at_transport_offset(frame, now, offset as u64) + .unwrap(); + assert!(duplicate.canonical.is_empty()); + assert_eq!(duplicate.duplicates, 1); + } + } + let view = delayed.l2_adapters[&binding.key()].core().view().unwrap(); + assert_eq!(view.last_sequence, 103); + assert_eq!(view.snapshot_sequence, 100); + assert_eq!(view.bids.len(), 1); + } + } + + #[test] + fn delayed_l2_real_gap_still_requires_provider_snapshot() { + for binding in [okx_book_binding(), binance_book_binding()] { + let mut core = core(binding.clone(), true); + let receipt = 1_786_352_400_000_000_000; + let now = receipt + 60_000_000_000; + for frame in [ + delayed_book_frame(&binding, true, 100, 0, 7, receipt, "1"), + delayed_book_frame(&binding, false, 101, 100, 7, receipt, "2"), + ] { + assert!(core.process(frame, now).unwrap().quarantines.is_empty()); + } + let gap = core + .process( + delayed_book_frame(&binding, false, 110, 109, 7, receipt, "3"), + now, + ) + .unwrap(); + assert!(gap.canonical.is_empty()); + assert_eq!(gap.quarantines.len(), 1); + assert!(core.l2_adapters[&binding.key()].core().view().is_none()); + let after = core + .process( + delayed_book_frame(&binding, false, 111, 110, 7, now, "4"), + now, + ) + .unwrap(); + assert!(after.canonical.is_empty()); + for frame in [ + delayed_book_frame(&binding, true, 200, 0, 8, now, "5"), + delayed_book_frame(&binding, false, 201, 200, 8, now, "6"), + ] { + assert!(core.process(frame, now).unwrap().quarantines.is_empty()); + } + assert_eq!( + core.l2_adapters[&binding.key()] + .core() + .view() + .unwrap() + .last_sequence, + 201 + ); + } + } + + #[test] + fn delayed_l2_invalid_level_still_invalidates_verified_state() { + for binding in [okx_book_binding(), binance_book_binding()] { + let mut core = core(binding.clone(), true); + let receipt = 1_786_352_400_000_000_000; + let now = receipt + 60_000_000_000; + for frame in [ + delayed_book_frame(&binding, true, 100, 0, 7, receipt, "1"), + delayed_book_frame(&binding, false, 101, 100, 7, receipt, "2"), + ] { + core.process(frame, now).unwrap(); + } + let invalid = core + .process( + delayed_book_frame(&binding, false, 102, 101, 7, receipt, "-1"), + now, + ) + .unwrap(); + assert!(invalid.canonical.is_empty()); + assert_eq!(invalid.quarantines.len(), 1); + assert!(core.l2_adapters[&binding.key()].core().view().is_none()); + let fresh = core + .process( + delayed_book_frame(&binding, false, 103, 102, 7, now, "3"), + now, + ) + .unwrap(); + assert!(fresh.canonical.is_empty()); + } + } + + #[test] + fn delayed_l2_generation_change_requires_anchor_and_rejects_old_generation() { + for binding in [okx_book_binding(), binance_book_binding()] { + let mut core = core(binding.clone(), true); + let receipt = 1_786_352_400_000_000_000; + let now = receipt + 60_000_000_000; + for frame in [ + delayed_book_frame(&binding, true, 100, 0, 7, receipt, "1"), + delayed_book_frame(&binding, false, 101, 100, 7, receipt, "2"), + ] { + core.process(frame, now).unwrap(); + } + let unanchored = core + .process( + delayed_book_frame(&binding, false, 201, 200, 8, receipt, "3"), + now, + ) + .unwrap(); + assert!(unanchored.canonical.is_empty()); + assert!(core.l2_adapters[&binding.key()].core().view().is_none()); + let anchor = core + .process( + delayed_book_frame(&binding, true, 200, 0, 8, receipt, "4"), + now, + ) + .unwrap(); + assert!(!anchor.canonical.is_empty()); + let old = core + .process( + delayed_book_frame(&binding, true, 300, 0, 7, now, "99"), + now, + ) + .unwrap(); + assert!(old.canonical.is_empty()); + assert_eq!(old.filtered_outcome, Some("IGNORED_STALE_GENERATION")); + let view = core.l2_adapters[&binding.key()].core().view().unwrap(); + assert_eq!(view.generation, 8); + assert!(view.last_sequence == 200 || view.last_sequence == 201); + } } #[test] diff --git a/rust/qdl-stream-gateway/examples/kn_trade_burst_gateway.rs b/rust/qdl-stream-gateway/examples/kn_trade_burst_gateway.rs new file mode 100644 index 00000000..600a23f3 --- /dev/null +++ b/rust/qdl-stream-gateway/examples/kn_trade_burst_gateway.rs @@ -0,0 +1,619 @@ +//! Isolated capture-backed gRPC driver, NOT Kafka or live-freshness evidence. +//! Usage: kn_trade_burst_gateway CAPTURE TEMPLATES PRIVATE_DIR RATE REPEAT [PORT] +//! PRIVATE_DIR must exist; publication starts when every product is subscribed. +//! Packet credentials are ephemeral and written mode 0600, never to stdout. +use base64::{engine::general_purpose::STANDARD, Engine}; +use jsonwebtoken::{encode, Algorithm, DecodingKey, EncodingKey, Header}; +use prost::Message; +use qdl_contracts::cursor_v3::{ + CursorV3Claims, CursorV3Codec, CursorV3Expectation, DeliveryRequirement, +}; +use qdl_stream_gateway::{ + auth::{AccessError, JwtConfig, RequestQuota}, + authority::{Authority, AuthorityHandle}, + bundle::{Binding, Bundle, Manifest, ManifestRequirement, Quotas}, + generated::{ + marketdata_v2::{event_envelope, EventEnvelope}, + query_v2 as query, + }, + hub::{Hub, HubConfig, LogSource, RawRecord}, + readview::NotReadyReadView, + replay::{RangeCursor, RangeError, RangeSource, ReplayCoordinator}, + service::{Gateway, GatewayState, StreamLimits}, + subscription::ByteBudget, +}; +use query::market_data_stream_service_server::MarketDataStreamServiceServer; +use ring::{ + rand::{SecureRandom, SystemRandom}, + signature::{EcdsaKeyPair, KeyPair, ECDSA_P256_SHA256_FIXED_SIGNING}, +}; +use serde_json::{json, Value}; +use std::{ + collections::BTreeMap, + fs::{self, OpenOptions}, + io::{BufRead, BufReader, Write}, + os::unix::fs::OpenOptionsExt, + path::{Path, PathBuf}, + sync::{ + atomic::{AtomicU64, Ordering}, + Arc, + }, + time::{Duration, Instant, SystemTime, UNIX_EPOCH}, +}; + +const CONSUMER: &str = "test.capture.trade-burst"; +const SUBJECT: &str = "spiffe://test/capture-trade-burst"; +const STREAM: &str = "test.capture.canonical"; +fn scheduled_seconds(index: u64, total: u64, rate: u64, low_rate: u64) -> f64 { + let quarter = total / 4; + let middle_end = total - quarter; + if index <= quarter { + index as f64 / low_rate as f64 + } else if index <= middle_end { + quarter as f64 / low_rate as f64 + (index - quarter) as f64 / rate as f64 + } else { + quarter as f64 / low_rate as f64 + + (middle_end - quarter) as f64 / rate as f64 + + (index - middle_end) as f64 / low_rate as f64 + } +} + +fn now_ns() -> u64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .unwrap() + .as_nanos() as u64 +} +fn private_json(path: &Path, value: &Value) -> Result<(), Box> { + let temporary = path.with_extension("pending"); + let mut file = OpenOptions::new() + .write(true) + .create_new(true) + .mode(0o600) + .open(&temporary)?; + file.write_all(serde_json::to_string(value)?.as_bytes())?; + file.sync_all()?; + fs::rename(temporary, path)?; + fs::File::open(path.parent().ok_or("output parent missing")?)?.sync_all()?; + Ok(()) +} + +struct Product { + uid: String, + envelope: EventEnvelope, + payloads: Vec>, + source_offsets: Vec<(i64, i64)>, + published: AtomicU64, +} +struct Capture { + products: Vec, +} +impl Capture { + fn record(&self, partition: i32, offset: i64) -> Option { + let product = self.products.get(usize::try_from(partition).ok()?)?; + if offset < 1 || offset as u64 > product.published.load(Ordering::Acquire) { + return None; + } + Some(RawRecord { + partition, + offset, + key: product.uid.as_bytes().to_vec(), + payload: product.payloads[(offset as usize - 1) % product.payloads.len()].clone(), + timestamp_ms: 0, + }) + } +} +struct Live { + capture: Arc, + positions: Vec, + next_partition: usize, +} +impl LogSource for Live { + fn poll(&mut self, timeout: Duration) -> Result, String> { + let deadline = Instant::now() + timeout; + loop { + for _ in 0..self.positions.len() { + let p = self.next_partition; + self.next_partition = (p + 1) % self.positions.len(); + if let Some(row) = self.capture.record(p as i32, self.positions[p]) { + self.positions[p] += 1; + return Ok(Some(row)); + } + } + if Instant::now() >= deadline { + return Ok(None); + } + std::thread::sleep(Duration::from_micros(250)); + } + } + fn positions(&self) -> Vec<(i32, i64)> { + self.positions + .iter() + .enumerate() + .map(|(p, &o)| (p as i32, o)) + .collect() + } +} +struct Ranges(Arc); +struct Reader { + capture: Arc, + partition: i32, + position: i64, +} +impl RangeSource for Ranges { + fn open(&self, partition: i32, from: i64) -> Result, RangeError> { + Ok(Box::new(Reader { + capture: self.0.clone(), + partition, + position: from.max(1), + })) + } +} +impl RangeCursor for Reader { + fn next(&mut self, _: Duration) -> Result, RangeError> { + let record = self.capture.record(self.partition, self.position); + if record.is_some() { + self.position += 1; + } + Ok(record) + } + fn position(&self) -> Option { + Some(self.position) + } +} +// Local fixture only: real JWT/manifest/cursor validation, no shared quota Redis. +struct FixtureQuota; +impl RequestQuota for FixtureQuota { + fn consume(&self, _: &Manifest) -> Result<(), AccessError> { + Ok(()) + } +} + +fn requirement(uid: &str, template: &Value) -> query::DataRequirement { + query::DataRequirement { + instrument_uid: uid.into(), + source_policy_id: template["requirement"]["source_policy_id"] + .as_str() + .unwrap() + .into(), + feed_type: query::FeedType::Trade as i32, + grade: query::ConsumerGrade::Execution as i32, + stale_policy_type: query::StalePolicy::Block as i32, + gap_policy_type: query::GapPolicy::Block as i32, + recovery_policy: query::RecoveryPolicy::SnapshotAndReplay as i32, + revision_policy: query::BarRevisionPolicy::Latest as i32, + require_full_coverage: true, + require_final_bars: false, + event_recency_policy: query::StalePolicy::Observe as i32, + max_freshness_ms: template["requirement"]["max_freshness_ms"] + .as_u64() + .unwrap(), + max_session_liveness_ms: template["requirement"]["max_session_liveness_ms"] + .as_u64() + .unwrap(), + // Actual TS OBSERVE delivery policy; only projection uses a test clock. + ..Default::default() + } +} + +#[tokio::main(flavor = "multi_thread", worker_threads = 2)] +async fn main() -> Result<(), Box> { + let args: Vec = std::env::args().collect(); + if args.len() < 6 || args.len() > 7 { + return Err("CAPTURE TEMPLATES PRIVATE_DIR RATE REPEAT [PORT]".into()); + } + let directory = PathBuf::from(&args[3]); + let rate: u64 = args[4].parse()?; + let repeat: u64 = args[5].parse()?; + if !(100..=5000).contains(&rate) || !(1..=120).contains(&repeat) { + return Err("bounded rate=100..5000 repeat=1..120 required".into()); + } + let low_rate: u64 = std::env::var("QDL_TEST_BURST_LOW_RATE") + .ok() + .map(|v| v.parse()) + .transpose()? + .unwrap_or(rate); + if !(100..=rate).contains(&low_rate) { + return Err("bounded burst low rate=100..offered rate required".into()); + } + if directory.join("start").exists() || directory.join("ready.json").exists() { + return Err("fresh private directory required".into()); + } + let templates: Value = serde_json::from_slice(&fs::read(&args[2])?)?; + let mut products: Vec = Vec::new(); + let mut schedule = Vec::new(); + let mut bytes = 0usize; + for line in BufReader::new(fs::File::open(&args[1])?).lines() { + let row: Value = serde_json::from_str(&line?)?; + let payload = STANDARD.decode(row["payload"].as_str().ok_or("payload missing")?)?; + bytes += payload.len(); + if bytes > 32 << 20 || schedule.len() >= 20000 { + return Err("capture bound exceeded".into()); + } + let envelope = EventEnvelope::decode(payload.as_slice())?; + if !matches!(envelope.payload, Some(event_envelope::Payload::Trade(_))) + || templates.get(&envelope.instrument_uid).is_none() + { + return Err("TRADE and matching template required".into()); + } + let p = match products + .iter() + .position(|p| p.uid == envelope.instrument_uid) + { + Some(p) => p, + None => { + products.push(Product { + uid: envelope.instrument_uid.clone(), + envelope, + payloads: Vec::new(), + source_offsets: Vec::new(), + published: AtomicU64::new(0), + }); + products.len() - 1 + } + }; + let coordinate = ( + row["partition"].as_i64().ok_or("partition missing")?, + row["offset"].as_i64().ok_or("offset missing")?, + ); + if products[p] + .source_offsets + .last() + .is_some_and(|old| old.0 != coordinate.0 || old.1 >= coordinate.1) + { + return Err("capture product offsets must increase in one partition".into()); + } + products[p].payloads.push(payload); + products[p].source_offsets.push(coordinate); + schedule.push(p); + } + if schedule.is_empty() || products.len() > 10 { + return Err("expected 1..10 nonempty products".into()); + } + if scheduled_seconds( + schedule.len() as u64 * repeat, + schedule.len() as u64 * repeat, + rate, + low_rate, + ) > 120.0 + { + return Err("offer window exceeds 120 seconds".into()); + } + let capture = Arc::new(Capture { products }); + let rng = SystemRandom::new(); + let pkcs8 = EcdsaKeyPair::generate_pkcs8(&ECDSA_P256_SHA256_FIXED_SIGNING, &rng) + .map_err(|_| "key generation")?; + let key = EcdsaKeyPair::from_pkcs8(&ECDSA_P256_SHA256_FIXED_SIGNING, pkcs8.as_ref(), &rng) + .map_err(|_| "key parse")?; + let mut secret = vec![0; 32]; + rng.fill(&mut secret).map_err(|_| "random")?; + let codec = CursorV3Codec::new(&BTreeMap::from([("burst".into(), secret)]), "burst")?; + let expectation = CursorV3Expectation { + environment: "paper".into(), + stream: STREAM.into(), + source_topic_id: "capture-test-not-kafka".into(), + partition_plan_epoch: 1, + source_policy_revision: 1, + catalog_revision: 1, + route_generation: "capture-test-only".into(), + schema_major: 2, + }; + let mut bindings = Vec::new(); + let mut requirements = Vec::new(); + let mut routes = Vec::new(); + for (p, product) in capture.products.iter().enumerate() { + let req = requirement(&product.uid, &templates[&product.uid]); + let delivery = DeliveryRequirement::from_proto(&req)?; + let lpk = format!( + "lpk1|paper|{}|{}|{}|TRADE|-", + product.envelope.venue, product.envelope.market, product.uid + ); + bindings.push(Binding { + binding_id: format!("capture-{p}"), + instrument_uid: product.uid.clone(), + venue: product.envelope.venue.clone(), + market: product.envelope.market.clone(), + feed: "TRADE".into(), + interval: None, + source_policy_id: req.source_policy_id.clone(), + physical_key: product.uid.clone(), + product_key: lpk.clone(), + stale_after_ms: 3000, + }); + requirements.push(ManifestRequirement { + instrument_uid: product.uid.clone(), + feed: "TRADE".into(), + interval: None, + consumer_grade: "EXECUTION".into(), + source_policy_id: req.source_policy_id.clone(), + event_recency_policy: Some("OBSERVE".into()), + max_session_liveness_ms: Some(req.max_session_liveness_ms), + }); + let cursor = codec + .encode(&CursorV3Claims { + key_id: "burst".into(), + environment: "paper".into(), + consumer_id: CONSUMER.into(), + requirement_digest: delivery.digest(), + schema_major: 2, + stream: STREAM.into(), + product_key: lpk, + snapshot_id: "capture-start".into(), + source_topic_id: expectation.source_topic_id.clone(), + source_partition: p as u64, + source_offset: 0, + partition_plan_epoch: 1, + source_policy_revision: 1, + catalog_revision: 1, + route_generation: expectation.route_generation.clone(), + issued_at_ns: now_ns(), + expires_at_ns: now_ns() + 900_000_000_000, + }) + .map_err(|e| format!("cursor: {e:?}"))?; + routes.push(json!({"instrument_uid": product.uid, "partition": p, "cursor": cursor, + "requirement_proto_b64": STANDARD.encode(req.encode_to_vec()), "expected_events": product.payloads.len() as u64 * repeat, + "source_coordinates": product.source_offsets})); + } + let bundle = Bundle { + sha256: "capture-test-only".into(), + environment: "paper".into(), + catalog_revision: 1, + source_policy_revision: 1, + canonical_stream: STREAM.into(), + bindings, + manifests: vec![Manifest { + consumer_id: CONSUMER.into(), + subject: SUBJECT.into(), + environment: "paper".into(), + manifest_revision: 1, + allowed_purposes: vec!["INTERNAL_EXECUTION".into()], + allowed_permissions: vec!["stream:read".into(), "history:read".into()], + requirements, + quotas: Quotas { + requests_per_minute: 10000, + max_batch_items: 100, + max_warmup_rows: 5000, + max_streams: 32, + max_buffer_events: 1000, + }, + }], + }; + let authority = AuthorityHandle::new(Authority { + bundle, + expectation, + jwt: JwtConfig { + environment: "paper".into(), + issuer: "https://capture.test".into(), + audience: "capture-test".into(), + keys: BTreeMap::from([( + "burst".into(), + ( + DecodingKey::from_ec_der(key.public_key().as_ref()), + vec![Algorithm::ES256], + ), + )]), + algorithms: vec![Algorithm::ES256], + max_lifetime_seconds: 900, + environments_by_key_id: BTreeMap::new(), + subjects_by_key_id: BTreeMap::from([("burst".into(), SUBJECT.into())]), + }, + }); + let starts: Vec<_> = (0..capture.products.len()).map(|p| (p as i32, 1)).collect(); + let hub = Arc::new(Hub::new( + &starts, + HubConfig { + ring_max_bytes: (48 << 20) / starts.len(), + ring_max_age: Duration::from_secs(120), + }, + )); + let reader = hub.run(Box::new(Live { + capture: capture.clone(), + positions: vec![1; starts.len()], + next_partition: 0, + })); + let limits = StreamLimits::default(); + let state = Arc::new(GatewayState::new( + authority, + Arc::new(FixtureQuota), + codec, + hub.clone(), + ReplayCoordinator::new( + Arc::new(Ranges(capture.clone())), + 2, + limits.replay.clone(), + ByteBudget::new(32 << 20), + ), + Arc::new(NotReadyReadView), + ByteBudget::new(32 << 20), + limits, + )); + let port: u16 = args.get(6).map(String::as_str).unwrap_or("18219").parse()?; + let address: std::net::SocketAddr = format!("127.0.0.1:{port}").parse()?; + let tls_root = directory.join("tls"); + let tls_enabled = tls_root.exists(); + let encrypted = if tls_enabled { + let config = qdl_stream_gateway::tls::server_config( + &fs::read_to_string(tls_root.join("server.pem"))?, + &fs::read_to_string(tls_root.join("server.key"))?, + &fs::read_to_string(tls_root.join("ca.pem"))?, + )?; + Some(qdl_stream_gateway::tls::incoming(address, config).await?) + } else { + None + }; + let plaintext = if tls_enabled { + None + } else { + let listener = tokio::net::TcpListener::bind(address).await?; + let (tx, rx) = tokio::sync::mpsc::channel(8); + tokio::spawn(async move { + loop { + let result = listener.accept().await.map(|(s, _)| s); + if tx.send(result).await.is_err() { + break; + } + } + }); + Some(tokio_stream::wrappers::ReceiverStream::new(rx)) + }; + let tls_packet = if tls_enabled { + json!({"ca_file":"tls/ca.pem", "client_cert_file":"tls/client.pem", "client_key_file":"tls/client.key"}) + } else { + json!(false) + }; + let mut header = Header::new(Algorithm::ES256); + header.kid = Some("burst".into()); + let jwt = encode( + &header, + &json!({"sub": SUBJECT, "iss":"https://capture.test", "aud":"capture-test", + "iat": now_ns()/1_000_000_000, "exp": now_ns()/1_000_000_000+900, "jti": format!("capture-{}", now_ns()), + "environment":"paper", "roles":["stream_consumer","historical_reader","market_data_reader"], "consumer_manifest_revision":1}), + &EncodingKey::from_ec_der(pkcs8.as_ref()), + )?; + let cursors: BTreeMap<_, _> = routes + .iter() + .map(|r| (r["instrument_uid"].as_str().unwrap(), &r["cursor"])) + .collect(); + let totals: BTreeMap<_, _> = routes + .iter() + .map(|r| (r["instrument_uid"].as_str().unwrap(), &r["expected_events"])) + .collect(); + private_json( + &directory.join("ready.json"), + &json!({"target":format!("127.0.0.1:{port}"), "tls":tls_packet, + "test_only":true,"consumer_id":CONSUMER,"purpose":"INTERNAL_EXECUTION","token":jwt,"routes":routes,"cursors":cursors,"total_per_uid":totals, + "offered_rate":rate,"burst_low_rate":low_rate,"repeat":repeat,"capture_clock":"max(received_at_ns,source_event_time_ns)+1000000", + "scope":"test-only capture-backed real gRPC; no Kafka durability or live eligibility"}), + )?; + let stopping = state.clone(); + let publishing = state.clone(); + let output = directory.clone(); + let producer = tokio::spawn(async move { + let wait = Instant::now(); + while publishing.hub.subscriber_count() < capture.products.len() { + if wait.elapsed() > Duration::from_secs(90) { + return Err("start timeout".to_owned()); + } + tokio::time::sleep(Duration::from_millis(20)).await; + } + let began = Instant::now(); + let total = schedule.len() as u64 * repeat; + let started_at_ns = now_ns(); + private_json( + &output.join("offer-start.json"), + &json!({ + "started_at_ns":started_at_ns,"total":total,"rate":rate,"burst_low_rate":low_rate, + "clock":"test scheduled injection wall clock; not original provider time" + }), + ) + .map_err(|e| e.to_string())?; + let mut max_late = 0f64; + let mut max_lag = 0u64; + let mut queue_peak = 0usize; + for index in 0..total { + if index % 16 == 0 { + let due = Duration::from_secs_f64(scheduled_seconds(index, total, rate, low_rate)); + tokio::time::sleep(due.saturating_sub(began.elapsed())).await; + max_late = max_late.max(began.elapsed().saturating_sub(due).as_secs_f64() * 1000.0); + } + capture.products[schedule[index as usize % schedule.len()]] + .published + .fetch_add(1, Ordering::Release); + if index % 16 == 0 { + let lag: u64 = capture + .products + .iter() + .enumerate() + .map(|(p, product)| { + product.published.load(Ordering::Acquire).saturating_sub( + publishing + .hub + .next_offset(p as i32) + .unwrap_or(1) + .saturating_sub(1) as u64, + ) + }) + .sum(); + max_lag = max_lag.max(lag); + queue_peak = queue_peak.max(publishing.budget.used()); + } + } + private_json(&output.join("producer.json"), &json!({"offered":total,"offered_rate":rate,"burst_low_rate":low_rate,"started_at_ns":started_at_ns, + "offer_seconds":began.elapsed().as_secs_f64(),"producer_schedule_late_max_ms":max_late, + "hub_lag_peak_sampled_records":max_lag,"queue_peak_sampled_bytes":queue_peak, + "pass":false,"note":"producer receipt only; TS apply/durable ACK and drain decide acceptance"})).map_err(|e| e.to_string())?; + let drain = Instant::now(); + while !output.join("stop").exists() && drain.elapsed() < Duration::from_secs(60) { + tokio::time::sleep(Duration::from_millis(50)).await; + } + let load = |v: &AtomicU64| v.load(Ordering::Relaxed); + private_json(&output.join("gateway.json"), &json!({"delivered":load(&publishing.metrics.delivered), + "replayed":load(&publishing.metrics.replayed),"overflow":load(&publishing.metrics.overflow), + "reader_errors":load(&publishing.hub.metrics.reader_errors),"decode_errors":load(&publishing.hub.metrics.decode_failures), + "queue_end_bytes":publishing.budget.used(),"drain_wait_seconds":drain.elapsed().as_secs_f64(), + "client_stop_received":output.join("stop").exists()})).map_err(|e| e.to_string())?; + Ok::<(), String>(()) + }); + let server = tonic::transport::Server::builder() + .add_service(MarketDataStreamServiceServer::new(Gateway { state })); + let shutdown = async move { + match producer.await { + Ok(Ok(())) => {} + other => eprintln!("capture producer failed: {other:?}"), + } + stopping.shut_down(); + }; + let result = match encrypted { + Some(incoming) => { + server + .serve_with_incoming_shutdown(incoming, shutdown) + .await + } + None => { + server + .serve_with_incoming_shutdown(plaintext.unwrap(), shutdown) + .await + } + }; + hub.stop(); + reader.join().map_err(|_| "reader panicked")?; + result?; + Ok(()) +} + +#[cfg(test)] +mod scheduling_tests { + use super::{requirement, scheduled_seconds}; + + #[test] + fn trade_requirement_matches_production_facade_cursor_binding() { + let template = serde_json::json!({"requirement": { + "source_policy_id": "crypto_primary_v2", + "max_freshness_ms": 3000, + "max_session_liveness_ms": 45000 + }}); + let request = requirement("11111111-1111-5111-8111-111111111111", &template); + assert!(!request.require_final_bars); + assert!(request.require_full_coverage); + let actual = qdl_contracts::requirement::ValidatedRequirement::from_proto(&request) + .expect("valid production TRADE requirement"); + let mut mismatched = request.clone(); + mismatched.require_final_bars = true; + let old = qdl_contracts::requirement::ValidatedRequirement::from_proto(&mismatched) + .expect("old fixture remains syntactically valid"); + assert_ne!(actual.delivery.digest(), old.delivery.digest()); + } + #[test] + fn constant_and_burst_are_bounded_monotonic_schedules() { + assert_eq!(scheduled_seconds(10000, 10000, 5000, 5000), 2.0); + assert_eq!(scheduled_seconds(2500, 10000, 5000, 1000), 2.5); + assert_eq!(scheduled_seconds(7500, 10000, 5000, 1000), 3.5); + assert_eq!(scheduled_seconds(10000, 10000, 5000, 1000), 6.0); + for index in 1..=10003 { + assert!( + scheduled_seconds(index, 10003, 5000, 1000) + > scheduled_seconds(index - 1, 10003, 5000, 1000) + ); + } + } +} diff --git a/rust/qdl-stream-gateway/src/auth.rs b/rust/qdl-stream-gateway/src/auth.rs index bd684a7c..425f7a57 100644 --- a/rust/qdl-stream-gateway/src/auth.rs +++ b/rust/qdl-stream-gateway/src/auth.rs @@ -61,6 +61,7 @@ pub struct JwtConfig { pub algorithms: Vec, pub max_lifetime_seconds: i64, pub subjects_by_key_id: BTreeMap, + pub environments_by_key_id: BTreeMap, } const OID_RSA_ENCRYPTION: &[u8] = &[0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x01]; @@ -131,6 +132,31 @@ pub fn public_key_from_pem(pem: &str) -> Result<(Algorithm, DecodingKey), String } impl JwtConfig { + /// Explicit key realms share public market storage, never consumer identity. + /// Missing mapping retains the single-realm compatibility behavior. + pub fn with_key_environments_json(mut self, raw: &str) -> Result { + let mapping: Option> = + serde_json::from_str(raw).map_err(|error| error.to_string())?; + if let Some(mapping) = mapping { + if mapping.keys().ne(self.keys.keys()) + || mapping + .values() + .any(|value| !matches!(value.as_str(), "paper" | "sandbox" | "live")) + { + return Err("JWT key-realm bindings must cover exactly the keyring".into()); + } + self.environments_by_key_id = mapping; + } + Ok(self) + } + + pub fn key_environment(&self, key_id: &str) -> &str { + self.environments_by_key_id + .get(key_id) + .map(String::as_str) + .unwrap_or(&self.environment) + } + /// Same inputs as `DataPlaneSecurityConfig.from_environment`: a kid -> /// PEM public key map and a kid -> subject map that must cover it. pub fn from_json( @@ -182,6 +208,7 @@ impl JwtConfig { algorithms, max_lifetime_seconds, subjects_by_key_id: subjects, + environments_by_key_id: BTreeMap::new(), }) } } @@ -299,7 +326,7 @@ pub fn verify_token(config: &JwtConfig, token: &str, now: f64) -> Result = claims["roles"] @@ -565,6 +592,7 @@ mod tests { fn config(decoding: DecodingKey) -> JwtConfig { JwtConfig { + environments_by_key_id: BTreeMap::new(), environment: "paper".into(), issuer: "https://identity.test".into(), audience: "qdl-v2-stable".into(), @@ -589,6 +617,33 @@ mod tests { }) } + #[test] + fn pinned_key_realms_do_not_cross_paper_sandbox_live() { + for realm in ["paper", "sandbox", "live"] { + let (encoding, decoding) = keypair(); + let config = config(decoding) + .with_key_environments_json(&serde_json::json!({"k1": realm}).to_string()) + .unwrap(); + let now = now_seconds().floor() as i64; + for claimed in ["paper", "sandbox", "live"] { + let mut value = claims(now); + value["environment"] = claimed.into(); + let result = verify_token(&config, &token(&encoding, "k1", value), now as f64); + assert_eq!(result.is_ok(), realm == claimed); + } + } + } + + #[test] + fn key_realm_mapping_requires_exact_keys_and_known_realms() { + for raw in ["{}", r#"{"other":"live"}"#, r#"{"k1":"*"}"#, "[]"] { + let (_, decoding) = keypair(); + assert!(config(decoding).with_key_environments_json(raw).is_err()); + } + let (_, decoding) = keypair(); + assert!(config(decoding).with_key_environments_json("null").is_ok()); + } + #[test] fn a_valid_token_verifies_and_every_claim_rule_refuses() { let (encoding, decoding) = keypair(); diff --git a/rust/qdl-stream-gateway/src/authority.rs b/rust/qdl-stream-gateway/src/authority.rs index 1bfccb29..129520a2 100644 --- a/rust/qdl-stream-gateway/src/authority.rs +++ b/rust/qdl-stream-gateway/src/authority.rs @@ -101,7 +101,8 @@ pub fn reauthorize( purpose: &str, entitlement: &Entitlement, ) -> Result<(), AccessError> { - if !authority.jwt.keys.contains_key(&principal.key_id) + if authority.jwt.key_environment(&principal.key_id) != principal.environment + || !authority.jwt.keys.contains_key(&principal.key_id) || authority.jwt.subjects_by_key_id.get(&principal.key_id) != Some(&principal.subject) { return Err(AccessError::Unauthenticated( diff --git a/rust/qdl-stream-gateway/src/main.rs b/rust/qdl-stream-gateway/src/main.rs index afd2ac5e..d82476f3 100644 --- a/rust/qdl-stream-gateway/src/main.rs +++ b/rust/qdl-stream-gateway/src/main.rs @@ -134,7 +134,8 @@ fn jwt_config(environment: &str) -> Result { value["max_lifetime_seconds"] .as_i64() .ok_or("jwt config max_lifetime_seconds must be an integer")?, - ); + )? + .with_key_environments_json(&value["environments"].to_string()); } JwtConfig::from_json( environment, @@ -144,7 +145,8 @@ fn jwt_config(environment: &str) -> Result { &env("QDL_DATA_JWT_KEY_SUBJECTS_JSON")?, &env_or("QDL_DATA_JWT_ALGORITHMS", "RS256,ES256"), parsed("QDL_DATA_JWT_MAX_LIFETIME_SECONDS", "900")?, - ) + )? + .with_key_environments_json(&env_or("QDL_DATA_JWT_KEY_ENVIRONMENTS_JSON", "null")) } fn load_authority() -> Result { diff --git a/rust/qdl-stream-gateway/tests/native_stream.rs b/rust/qdl-stream-gateway/tests/native_stream.rs index 6db82b2f..63f78a79 100644 --- a/rust/qdl-stream-gateway/tests/native_stream.rs +++ b/rust/qdl-stream-gateway/tests/native_stream.rs @@ -453,6 +453,7 @@ fn keypair() -> Keys { fn jwt_config(keys: &[(&str, &Keys)]) -> JwtConfig { JwtConfig { + environments_by_key_id: BTreeMap::new(), environment: "paper".into(), issuer: "https://identity.test".into(), audience: "qdl-v2-stable".into(), diff --git a/scripts/accept_consumer_realms.py b/scripts/accept_consumer_realms.py new file mode 100644 index 00000000..93c5eedc --- /dev/null +++ b/scripts/accept_consumer_realms.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Strict no-order Query/Stream acceptance for separately pinned consumer realms.""" +import argparse +import asyncio +import json +import time +from pathlib import Path + +import yaml +from qdl_sdk import DataRequirement, Feed, Grade, StalePolicy, StreamEvent +from qdl_sdk.client import AsyncDataLayerClient +from qdl_sdk.credentials import RotatingJwtCredentialProvider +from qdl_sdk.errors import DataLayerError +from qdl_sdk.reference import ReferenceRequirement, ReferenceProduct +from qdl_sdk.models import GapPolicy, RecoveryPolicy, BarRevisionPolicy +from qdl_sdk.tls import WorkloadTlsConfig +from qdl_sdk.transport import RestQueryTransport, GrpcStreamTransport + + +def requirement(row): + return DataRequirement(instrument_uid=row["instrument_uid"], feed=Feed(row["feed"]), + consumer_grade=Grade(row["consumer_grade"]), source_policy_id=row["source_policy_id"], + interval=row.get("interval"), warmup_limit=0, max_freshness_ms=row.get("max_freshness_ms"), + event_recency_policy=StalePolicy(row["event_recency_policy"]) if row.get("event_recency_policy") else None, + max_session_liveness_ms=row.get("max_session_liveness_ms"), + require_full_coverage=row.get("require_full_coverage", True), + require_final_bars=row.get("require_final_bars", True), + stale_policy=StalePolicy(row.get("stale_policy", "BLOCK")), + gap_policy=GapPolicy(row.get("gap_policy", "BLOCK")), + recovery=RecoveryPolicy(row.get("recovery", "SNAPSHOT_AND_REPLAY")), + bar_revision_policy=BarRevisionPolicy(row.get("bar_revision_policy", "LATEST"))) + + +def execution_reference(requirement): + if requirement.feed is not Feed.MARK_INDEX_PRICE or requirement.consumer_grade is not Grade.EXECUTION: + raise ValueError("execution MARK_INDEX_PRICE requirement required") + return ReferenceRequirement( + instrument_uid=requirement.instrument_uid, product=ReferenceProduct.MARK_INDEX_PRICE, + consumer_grade=Grade.EXECUTION, source_policy_id=requirement.source_policy_id, + limit=1, page_size=1, max_pages=1, require_full_coverage=True, + max_freshness_ms=requirement.max_freshness_ms, + event_recency_policy=requirement.event_recency_policy, + max_session_liveness_ms=requirement.max_session_liveness_ms, + ) + + +async def read_execution_reference(client, requirement): + # Same one-item path as TS; item errors remain visible, never exempted. + response = await client.reference_batch((execution_reference(requirement),), require_all=False) + from scripts.probe_projector_stage_timing import reference_metadata + return reference_metadata(response.model_dump(mode="json")) + + +async def run(args): + packet = Path(args.packet) + policy = json.loads((packet / "reader-public/jwt-config.json").read_text()) + reads, negatives, streams, reference_reads = [], [], [], [] + for path in sorted((packet / "manifests-crypto").glob("*.yaml")): + payload = yaml.safe_load(path.read_text()) + meta, declared = payload["metadata"], payload["spec"]["requirements"] + cid, realm = meta["id"], meta["environment"] + identity = packet / "identities" / cid + tls = WorkloadTlsConfig(identity / "ca.crt", identity / "client.crt", identity / "client.key") + def credential(**overrides): + values = dict(private_key_file=identity / "private.key", key_id=cid + "-rs256-v1", + algorithm="RS256", issuer=policy["issuer"], audience=policy["audience"], + subject=meta["subject"], environment=realm, + roles=("market_data_reader", "historical_reader", "stream_consumer"), + consumer_manifest_revision=meta["revision"], lifetime_seconds=300, refresh_before_seconds=30) + values.update(overrides) + return RotatingJwtCredentialProvider(**values) + selected = list(declared) if cid.startswith("trading-system.") else [] + if not selected: + for feed in ("TRADE", "QUOTE", "BAR", "MARK_INDEX_PRICE", "BOOK_SNAPSHOT", "BOOK_DELTA"): + row = next((r for r in declared if r["feed"] == feed and (feed != "BAR" or r.get("interval") == "1m")), None) + if row: + selected.append(row) + for replica in (1, 2): + creds = credential() + url = f"https://query_v2_{replica}:8200" + query = RestQueryTransport(url, credential_provider=creds, tls=tls) + transport = GrpcStreamTransport(f"qdl-v2-stream-{'a' if replica == 1 else 'b'}:8210", credential_provider=creds, tls=tls) + client = AsyncDataLayerClient(query_transport=query, stream_transport=transport, + consumer_id=cid, max_warmup_attempts=1) + try: + for raw in selected: + req = requirement(raw) + start = time.perf_counter() + row = dict(consumer_id=cid, realm=realm, replica=replica, + instrument_uid=req.instrument_uid, feed=req.feed.value, interval=req.interval) + try: + response = await client.snapshot(req) + quality = response.data.quality + row.update(status="TYPED_RESPONSE", state=quality.state, + execution_eligible=quality.execution_eligible, + event_recency_state=quality.event_recency_state, + provider_session_state=quality.provider_session_state, + gap_open=quality.gap_open, complete=quality.complete, + sampled_at_ns=time.time_ns(), + observed_at_ns=response.data.observed_at_ns, + received_at_ns=response.data.received_at_ns, + watermark_offset=response.data.watermark_offset, + quality=quality.model_dump(mode="json"), + source=response.data.source.model_dump(mode="json"), + contract=response.data.contract.model_dump(mode="json")) + except Exception as error: + row.update(status="REFUSED", code=getattr(error, "code", type(error).__name__), detail=str(error)[:180], diagnostics=getattr(error, "diagnostics", None)) + row["call_to_result_ms"] = (time.perf_counter() - start) * 1000 + reads.append(row) + if req.feed is Feed.MARK_INDEX_PRICE and req.consumer_grade is Grade.EXECUTION: + reference_row = dict(consumer_id=cid, realm=realm, replica=replica, + instrument_uid=req.instrument_uid, endpoint="reference:batch") + reference_start = time.perf_counter() + try: + evidence = await read_execution_reference(client, req) + reference_row.update(status="TYPED_RESPONSE", evidence=evidence, + item_ok=len(evidence["results"]) == 1 and evidence["results"][0]["status"] == "OK") + except Exception as error: + reference_row.update(status="REFUSED", item_ok=False, + code=getattr(error, "code", type(error).__name__), + diagnostics=getattr(error, "diagnostics", None)) + reference_row["call_to_result_ms"] = (time.perf_counter() - reference_start) * 1000 + reference_reads.append(reference_row) + await asyncio.sleep(.1) + probe = requirement(next(r for r in selected if r["feed"] == "TRADE")) + for name, kwargs, consumer in ( + ("wrong-realm", {"environment": "paper"}, cid), + ("wrong-revision", {"consumer_manifest_revision": meta["revision"] + 1}, cid), + ("wrong-consumer", {}, cid.replace("." + realm + ".", ".paper.")), + ): + bad = RestQueryTransport(url, credential_provider=credential(**kwargs), tls=tls) + try: + await bad.snapshot(probe, consumer_id=consumer) + negatives.append(dict(consumer_id=cid, replica=replica, case=name, refused=False)) + except DataLayerError as error: + negatives.append(dict(consumer_id=cid, replica=replica, case=name, + refused=error.code in {"UNAUTHENTICATED", "CONSUMER_MISMATCH", "PERMISSION_DENIED"}, code=error.code)) + finally: + await bad.close() + try: + async with asyncio.timeout(20): + async with client.warmup_then_stream(probe) as session: + async for event in session: + if isinstance(event, StreamEvent): + await session.acknowledge_async(event) + streams.append(dict(consumer_id=cid, replica=replica, status="STREAM_ACK_PASS")) + break + except Exception as error: + streams.append(dict(consumer_id=cid, replica=replica, status="FAILED", + code=getattr(error, "code", type(error).__name__), detail=str(error)[:180])) + finally: + await client.close() + # No DATA_STALE exemption and no retry seeking a green sample. + failures = [r for r in reads if r["status"] != "TYPED_RESPONSE"] + usable = [r for r in reads if r.get("execution_eligible") is True] + report = dict(schema="qdl.consumer-realm-acceptance.v1", order_actions=0, + snapshots=reads, negatives=negatives, streams=streams, execution_references=reference_reads, + execution_reference_gate=bool(reference_reads) and all(r["item_ok"] for r in reference_reads), + typed_response_gate=bool(reads) and not failures, + execution_usable_count=len(usable), total_reads=len(reads), + negative_gate=bool(negatives) and all(r["refused"] for r in negatives), + stream_gate=len(streams) == 12 and all(r["status"] == "STREAM_ACK_PASS" for r in streams)) + report["all_reads_execution_eligible"] = bool(reads) and len(usable) == len(reads) + Path(args.output).write_text(json.dumps(report, indent=2)) + print(json.dumps({k:v for k,v in report.items() if k not in {"snapshots", "negatives", "streams", "execution_references"}})) + return report["typed_response_gate"] and report["negative_gate"] and report["stream_gate"] and report["execution_reference_gate"] + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--packet", required=True) + parser.add_argument("--output", required=True) + raise SystemExit(0 if asyncio.run(run(parser.parse_args())) else 1) diff --git a/scripts/build_qdl_sdk_release.py b/scripts/build_qdl_sdk_release.py index 59d39d45..708d8bc2 100755 --- a/scripts/build_qdl_sdk_release.py +++ b/scripts/build_qdl_sdk_release.py @@ -15,14 +15,14 @@ ROOT = Path(__file__).resolve().parents[1] NAME = "qdl-sdk" NORMALIZED_NAME = "qdl_sdk" -VERSION = "2.0.5" +VERSION = "2.0.6" DIST_INFO = f"{NORMALIZED_NAME}-{VERSION}.dist-info" DEPENDENCIES = ( "grpcio>=1.70.0,<2.0.0", "httpx>=0.28.0,<1.0.0", "protobuf>=6.31.1,<7.0.0", "pydantic>=2.0.0,<3.0.0", - "PyJWT[crypto]>=2.13.0,<3.0.0", + "PyJWT[crypto]>=2.15.1,<3.0.0", ) FIXED_TIMESTAMP = (2020, 1, 1, 0, 0, 0) diff --git a/scripts/compile_consumer_realms.py b/scripts/compile_consumer_realms.py new file mode 100644 index 00000000..a4b4fcde --- /dev/null +++ b/scripts/compile_consumer_realms.py @@ -0,0 +1,83 @@ +#!/usr/bin/env python3 +"""Compile explicit sandbox/live read manifests from approved stable scopes. + +No credentials, activation or runtime mutation. The source paper manifests stay +unchanged; output belongs in a versioned runtime packet, not the source tree. +""" +from __future__ import annotations + +import argparse +import copy +import json +from pathlib import Path + +import yaml + +from qdl.consumer.manifest import ConsumerManifestLoader + + +def compile_realm(payload: dict, realm: str) -> dict: + source = ConsumerManifestLoader.from_mapping(payload) + if source.environment != "paper" or realm not in {"sandbox", "live"}: + raise ValueError("explicit paper -> sandbox/live public-market mapping required") + approved = { + "trading-system.paper.stable": "PAPER_ONLY", + "alpha.binance.paper.stable": "FORBIDDEN", + "alpha.okx.paper.stable": "FORBIDDEN", + } + if approved.get(source.consumer_id) != source.execution_dependency: + raise ValueError("only the three owner-approved stable consumer scopes may be mapped") + if source.consumer_id.count(".paper.") != 1 or not source.subject.startswith("spiffe://qdl/paper/"): + raise ValueError("stable consumer identity convention mismatch") + result = copy.deepcopy(payload) + metadata = result["metadata"] + metadata.update(id=source.consumer_id.replace(".paper.", "." + realm + "."), + subject=source.subject.replace("spiffe://qdl/paper/", "spiffe://qdl/" + realm + "/", 1), + environment=realm, revision=1) + # Alpha research permission is NOT promoted into direct Risk authority. + result["spec"]["execution_dependency"] = ( + "ALLOWED" if source.execution_dependency == "PAPER_ONLY" else "FORBIDDEN" + ) + # VN stays on its separately deferred V1 path. This packet authorizes only + # the existing crypto policies; future policies require explicit inclusion. + crypto_policies = {"crypto_primary_v2", "crypto_liquid_v2"} + result["spec"]["requirements"] = [ + row for row in result["spec"]["requirements"] + if row["source_policy_id"] in crypto_policies + ] + target = ConsumerManifestLoader.from_mapping(result) + expected = tuple(row for row in source.requirements if row.source_policy_id in crypto_policies) + if target.requirements != expected or target.quotas != source.quotas: + raise AssertionError("realm mapping must preserve approved crypto scope and quotas") + return result + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--source", type=Path, nargs="+", required=True) + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + if args.output_dir.exists(): + raise ValueError("new output directory required; no overwrite of sealed manifests") + outputs = [] + for path in args.source: + payload = yaml.safe_load(path.read_text()) + for realm in ("sandbox", "live"): + mapped = compile_realm(payload, realm) + parsed = ConsumerManifestLoader.from_mapping(mapped) + outputs.append((parsed.consumer_id + ".yaml", mapped, parsed)) + if len({name for name, _, _ in outputs}) != len(outputs): + raise ValueError("duplicate target consumer") + args.output_dir.mkdir(parents=True, mode=0o750) + receipt = [] + for name, mapped, parsed in outputs: + (args.output_dir / name).write_text(yaml.safe_dump(mapped, sort_keys=False)) + receipt.append(dict(consumer_id=parsed.consumer_id, environment=parsed.environment, + subject=parsed.subject, manifest_revision=parsed.manifest_revision, + manifest_sha256=parsed.manifest_sha256, requirements=len(parsed.requirements))) + print(json.dumps({"status": "COMPILED_NOT_ACTIVATED", "market_storage_realm": "paper", + "order_permissions": False, "manifests": receipt}, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/scripts/kn_trade_burst_build.sh b/scripts/kn_trade_burst_build.sh new file mode 100644 index 00000000..b5c3c03d --- /dev/null +++ b/scripts/kn_trade_burst_build.sh @@ -0,0 +1,7 @@ +#!/bin/sh +# Run inside the existing Rust builder with --cpus 2 --memory 2g --memory-swap 2g. +set -eu +export CARGO_BUILD_JOBS=2 +rustfmt --edition 2021 rust/qdl-stream-gateway/examples/kn_trade_burst_gateway.rs +cargo test --locked -j 2 -p qdl-stream-gateway --lib --test native_stream -- --test-threads=2 +cargo build --locked -j 2 -p qdl-stream-gateway --example kn_trade_burst_gateway diff --git a/scripts/kn_trade_burst_tls.py b/scripts/kn_trade_burst_tls.py new file mode 100644 index 00000000..1df72a88 --- /dev/null +++ b/scripts/kn_trade_burst_tls.py @@ -0,0 +1,61 @@ +#!/usr/bin/env python3 +"""Create a fresh two-hour test CA and mTLS identities in PRIVATE_DIR/tls.""" +import argparse +from datetime import datetime, timedelta, timezone +from ipaddress import ip_address +import os +from pathlib import Path + +from cryptography import x509 +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.asymmetric import ec +from cryptography.x509.oid import ExtendedKeyUsageOID, NameOID + + +def create(directory): + root = Path(directory) / "tls" + root.mkdir(mode=0o700) + now = datetime.now(timezone.utc) + ca_key = ec.generate_private_key(ec.SECP256R1()) + ca_name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "capture-test-only-ca")]) + + def builder(subject, key): + return (x509.CertificateBuilder().subject_name(subject).issuer_name(ca_name) + .public_key(key.public_key()).serial_number(x509.random_serial_number()) + .not_valid_before(now - timedelta(minutes=1)) + .not_valid_after(now + timedelta(hours=2))) + + def write(name, data): + descriptor = os.open(root / name, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(descriptor, "wb") as handle: + handle.write(data) + handle.flush() + os.fsync(handle.fileno()) + + ca = (builder(ca_name, ca_key).add_extension(x509.BasicConstraints(ca=True, path_length=0), True) + .add_extension(x509.KeyUsage(digital_signature=True, content_commitment=False, + key_encipherment=False, data_encipherment=False, key_agreement=False, + key_cert_sign=True, crl_sign=True, encipher_only=False, decipher_only=False), True) + .sign(ca_key, hashes.SHA256())) + write("ca.pem", ca.public_bytes(serialization.Encoding.PEM)) + for name, usage in (("server", ExtendedKeyUsageOID.SERVER_AUTH), + ("client", ExtendedKeyUsageOID.CLIENT_AUTH)): + key = ec.generate_private_key(ec.SECP256R1()) + subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "capture-test-" + name)]) + certificate = (builder(subject, key) + .add_extension(x509.BasicConstraints(ca=False, path_length=None), True) + .add_extension(x509.ExtendedKeyUsage([usage]), False)) + if name == "server": + certificate = certificate.add_extension(x509.SubjectAlternativeName( + [x509.IPAddress(ip_address("127.0.0.1")), x509.DNSName("localhost")]), False) + certificate = certificate.sign(ca_key, hashes.SHA256()) + write(name + ".pem", certificate.public_bytes(serialization.Encoding.PEM)) + write(name + ".key", key.private_bytes(serialization.Encoding.PEM, + serialization.PrivateFormat.PKCS8, serialization.NoEncryption())) + # CA private key is deliberately never persisted. + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("directory") + create(parser.parse_args().directory) diff --git a/scripts/kn_trade_burst_wire_smoke.py b/scripts/kn_trade_burst_wire_smoke.py new file mode 100644 index 00000000..3b42e64f --- /dev/null +++ b/scripts/kn_trade_burst_wire_smoke.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 +"""Bounded wire-only driver regression; not SDK/TS/Redis/ACK acceptance.""" +import argparse +import asyncio +import base64 +import json +from pathlib import Path +import tempfile + +import grpc +from qdl.marketdata.v2.market_data_pb2 import EventEnvelope +from qdl.query.v2 import query_pb2 as pb + + +async def run(args): + expected = {} + for line in Path(args.capture).read_text().splitlines(): + payload = base64.b64decode(json.loads(line)["payload"], validate=True) + uid = EventEnvelope.FromString(payload).instrument_uid + expected.setdefault(uid, []).append(payload) + with tempfile.TemporaryDirectory(prefix="kn-trade-wire-") as directory: + root = Path(directory) + process = await asyncio.create_subprocess_exec( + args.driver, args.capture, args.templates, directory, "4000", "1", str(args.port), + stdout=asyncio.subprocess.DEVNULL, stderr=asyncio.subprocess.PIPE) + tasks = [] + try: + async with asyncio.timeout(30): + while not (root / "ready.json").exists(): + if process.returncode is not None: + raise RuntimeError("driver exited before readiness") + await asyncio.sleep(0.02) + packet = json.loads((root / "ready.json").read_text()) + assert set(packet["total_per_uid"]) == set(expected) + ready = {uid: asyncio.Event() for uid in expected} + metadata = (("authorization", "Bearer " + packet["token"]), + ("x-qdl-consumer-id", packet["consumer_id"]), + ("x-qdl-purpose", packet["purpose"])) + async with grpc.aio.insecure_channel(packet["target"]) as channel: + rpc = channel.unary_stream( + "/qdl.query.v2.MarketDataStreamService/Subscribe", + request_serializer=pb.SubscribeRequest.SerializeToString, + response_deserializer=pb.SubscribeResponse.FromString) + + async def consume(route): + uid = route["instrument_uid"] + req = pb.DataRequirement.FromString(base64.b64decode(route["requirement_proto_b64"])) + call = rpc(pb.SubscribeRequest(consumer_id=packet["consumer_id"], requirement=req, + cursor_token=route["cursor"], max_buffer_events=1000), metadata=metadata) + count = 0 + try: + async for response in call: + record = response.record + if record.HasField("control"): + if pb.StreamControlState.Name(record.control.state).endswith("LIVE"): + ready[uid].set() + continue + if not record.HasField("event"): + raise AssertionError("unexpected record") + assert record.logical_offset == count + 1 + assert record.event.SerializeToString(deterministic=True) == expected[uid][count] + count += 1 + if count == len(expected[uid]): + return count + raise AssertionError("early stream end") + finally: + call.cancel() + + tasks = [asyncio.create_task(consume(route)) for route in packet["routes"]] + await asyncio.gather(*(event.wait() for event in ready.values())) + (root / "start").touch() + counts = await asyncio.gather(*tasks) + (root / "stop").touch() + assert await process.wait() == 0 + metrics = json.loads((root / "gateway.json").read_text()) + assert metrics["overflow"] == metrics["reader_errors"] == metrics["decode_errors"] == 0 + print(json.dumps({"pass": True, "scope": "wire-only; no SDK/apply/ACK claim", + "products": len(counts), "events": sum(counts), + "producer": json.loads((root / "producer.json").read_text()), + "gateway": metrics})) + finally: + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + if process.returncode is None: + process.terminate() + await process.wait() + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--driver", required=True) + parser.add_argument("--capture", required=True) + parser.add_argument("--templates", required=True) + parser.add_argument("--port", type=int, default=18229) + asyncio.run(run(parser.parse_args())) diff --git a/scripts/prepare_sandbox_read_extension.py b/scripts/prepare_sandbox_read_extension.py new file mode 100644 index 00000000..4befca75 --- /dev/null +++ b/scripts/prepare_sandbox_read_extension.py @@ -0,0 +1,49 @@ +#!/usr/bin/env python3 +"""Prepare an additive sandbox read packet; never apply runtime or issue credentials.""" +import argparse +import hashlib +import json +from pathlib import Path + +import yaml + +from qdl.runtime.production_catalog import ProductionDemandManifest +from qdl.runtime.sandbox_extension import prepare_sandbox_extension + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + for name in ("catalog", "acquisition", "manifest", "binding", "demand", "metadata"): + parser.add_argument("--" + name, type=Path, required=True) + parser.add_argument("--template-uid", required=True) + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + if args.output_dir.exists(): + raise ValueError("new output directory required, never overwrite a sealed packet") + files = {name: getattr(args, name).read_bytes() + for name in ("catalog", "acquisition", "manifest", "binding", "demand", "metadata")} + packet = prepare_sandbox_extension( + **{name: yaml.safe_load(files[name]) for name in ("catalog", "acquisition", "manifest", "binding")}, + demand=ProductionDemandManifest.load_many([args.demand]), + okx_rows=json.loads(files["metadata"]), template_uid=args.template_uid, + ) + # Build every output before creating the destination; no partial validation. + outputs = {name + ".json": json.dumps(value, indent=2, sort_keys=True).encode() + b"\n" + for name, value in packet.items()} + receipt = {"status": "PREPARED_NOT_ACTIVATED_NOT_CERTIFIED", "runtime_mutations": 0, + "input_sha256": {name: hashlib.sha256(data).hexdigest() for name, data in files.items()}, + "output_sha256": {name: hashlib.sha256(data).hexdigest() for name, data in outputs.items()}, + "binding_sha256": packet["binding"]["binding_sha256"], + "consumer_manifest_revision": packet["manifest"]["metadata"]["revision"], + "requirements": len(packet["manifest"]["spec"]["requirements"])} + args.output_dir.mkdir(mode=0o700, parents=True) + for name, data in outputs.items(): + path = args.output_dir / name + path.write_bytes(data) + path.chmod(0o600) + (args.output_dir / "receipt.json").write_text(json.dumps(receipt, indent=2, sort_keys=True) + "\n") + print(json.dumps(receipt, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/scripts/probe_exact_market_lineage.py b/scripts/probe_exact_market_lineage.py new file mode 100644 index 00000000..4f3d8e69 --- /dev/null +++ b/scripts/probe_exact_market_lineage.py @@ -0,0 +1,314 @@ +#!/usr/bin/env python3 +"""Bounded read-only raw/canonical/exact Query evidence; never an acceptance gate.""" +from __future__ import annotations + +import argparse +import asyncio +import dataclasses +import hashlib +import json +from pathlib import Path +import time + + +VIEW_FIELDS = ("instrument_uid", "instrument_id", "feed", "interval", "quality", + "source", "contract", "observed_at_ns", "received_at_ns", + "watermark_offset", "revision") + + +def exact_view(response): + """Allowlist metadata from the actual response; no payload or signed cursor.""" + row = response.get("data", {}) + return {key: row[key] for key in VIEW_FIELDS if key in row} + + +def trade_state(*, raw_count, canonical_count, query_matches, coverage_complete): + if not coverage_complete: + return "UNASSESSED_INCOMPLETE_CAPTURE" + if raw_count == 0: + return "NO_TRADE_IN_CAPTURE_WINDOW_NOT_SESSION_PROOF" + if canonical_count == 0: + return "RAW_PRESENT_CANONICAL_MISSING_IN_WINDOW" + if not query_matches: + return "CANONICAL_PRESENT_QUERY_JOIN_MISSING" + return "SOURCE_CANONICAL_QUERY_OBSERVED" + + +def exact_join(query, canonical): + """Match original lineage; the public handoff boundary may exceed row offset.""" + view = query.get("exact_view") or {} + event = canonical["event"] + return ( + query["uid"] == event["instrument_uid"] + and query["feed"].lower() == canonical["feed"] + and view.get("watermark_offset", -1) >= canonical["offset"] + and view.get("observed_at_ns") == int(event["source_event_time_ns"]) + and view.get("received_at_ns") == int(event["received_at_ns"]) + and view.get("contract", {}).get("correlation_id") == event["correlation_id"] + ) + + +def newer_before_request(query, selected, candidate): + """Positive lag evidence only: same product/generation, observed before call.""" + old, new = selected["event"], candidate["event"] + return ( + exact_join(query, selected) + and old["instrument_uid"] == new["instrument_uid"] + and selected["feed"] == candidate["feed"] + and selected["partition"] == candidate["partition"] + and candidate["offset"] > selected["offset"] + and candidate["captured_at_ns"] < query["request_started_at_ns"] + and old["source_session_id"] == new["source_session_id"] + and old["connection_generation"] == new["connection_generation"] + and old["authority_revision"] == new["authority_revision"] + and int(new["received_at_ns"]) > int(old["received_at_ns"]) + ) + + +class Evidence: + def __init__(self, path, max_bytes=48 * 1024 * 1024): + self.file = Path(path).open("x") + self.size = 0 + self.max_bytes = max_bytes + self.count = 0 + + def put(self, kind, **row): + line = json.dumps(dict(kind=kind, captured_at_ns=time.time_ns(), **row), + sort_keys=True, separators=(",", ":")) + "\n" + size = len(line.encode()) + if self.size + size > self.max_bytes: + raise RuntimeError("EVIDENCE_BYTE_LIMIT") + self.file.write(line) + self.file.flush() + self.size += size + self.count += 1 + + def close(self): + self.file.close() + + +def raw_metadata(raw): + from qdl.raw.envelope import validate_raw_envelope + validate_raw_envelope(raw) + body = json.loads(raw.raw_frame_bytes) + data = body.get("data", body) + rows = data if isinstance(data, list) else [data] + # Native IDs and clocks only: retain hashes, not provider payloads. + clocks = [{k: item[k] for k in ("ts", "T", "E", "tradeId", "t", "a", "instId", "s") + if k in item} for item in rows if isinstance(item, dict)] + return dict(capture_id=bytes(raw.capture_id).hex(), + raw_frame_sha256=bytes(raw.raw_frame_sha256).hex(), + provider=raw.provider, native_symbol=raw.native_symbol, + native_channel=raw.native_channel, received_at_ns=raw.received_at_ns, + source_session_id=raw.source_session_id, + connection_generation=raw.connection_generation, + authority_revision=raw.authority_revision, + test_provenance=raw.test_provenance, original_clocks=clocks) + + +def canonical_metadata(event): + from google.protobuf.json_format import MessageToDict + from qdl.runtime.mark_index_lineage import paired_mark_index_lineage + result = MessageToDict(event, preserving_proto_field_name=True) + result.pop(event.WhichOneof("payload"), None) + for key in ("event_id", "raw_capture_id", "raw_payload_hash", "canonical_payload_hash"): + result[key] = bytes(getattr(event, key)).hex() + if event.WhichOneof("payload") == "mark_index_price": + try: + pair = dataclasses.asdict(paired_mark_index_lineage(event)) + result["components"] = {k: v.hex() if isinstance(v, bytes) else v + for k, v in pair.items()} + except ValueError: + result["component_lineage"] = "NOT_DERIVED_PAIR" + return result + + +async def run(args): + import yaml + from confluent_kafka import Consumer, KafkaError, TopicPartition + from qdl.marketdata.v2.market_data_pb2 import EventEnvelope + from qdl.provider.v1.raw_provider_pb2 import RawProviderEnvelope + from qdl_sdk.client import AsyncDataLayerClient + from qdl_sdk.credentials import RotatingJwtCredentialProvider + from qdl_sdk.tls import WorkloadTlsConfig + from qdl_sdk.transport import GrpcStreamTransport, RestQueryTransport + from scripts.accept_consumer_realms import requirement + from scripts.kn_canonical_mirror import source_config + + if not 1 <= args.seconds <= 180: + raise ValueError("duration must be 1..180 seconds") + if not 0 <= args.tail_records <= 2000: + raise ValueError("tail must be 0..2000 records per partition") + packet = Path(args.packet) + policy = json.loads((packet / "reader-public/jwt-config.json").read_text()) + catalog = yaml.safe_load(Path(args.catalog).read_text()) + wanted = set(args.uid) + instruments = [r for r in catalog["instruments"] if r["instrument_uid"] in wanted] + if {r["instrument_uid"] for r in instruments} != wanted: + raise ValueError("UID absent from source catalog") + symbols = {r["native_symbol"] for r in instruments} + # OKX's index component is named BTC-USDT, not BTC-USDT-SWAP. + symbols |= {s.removesuffix("-SWAP") for s in symbols} + evidence = Evidence(args.output) + consumer = None + clients = [] + start = time.monotonic() + deadline = start + args.seconds + counters = {"raw": 0, "canonical": 0, "query": 0, "query_refused": 0} + starts, positions, eof = {}, {}, {} + errors = [] + + class ExactTransport(RestQueryTransport): + last = None + + async def snapshot(self, requirement, *, consumer_id): + self.last = None + value = await super().snapshot(requirement, consumer_id=consumer_id) + self.last = exact_view(value) + return value + + try: + evidence.put("scope", schema="qdl.exact-market-lineage.v1", + provenance="production_read_only", test_provenance=False, + instruments=instruments, duration_seconds=args.seconds, + source_catalog_sha256=hashlib.sha256(Path(args.catalog).read_bytes()).hexdigest(), + kafka_commits=False, order_actions=0, + classification_rule="No quiet conclusion from provider_session_state") + config = source_config(argparse.Namespace(source_bootstrap=args.bootstrap, + group_prefix="qdl-r1-reference-parity-", ca=args.ca, + cert=args.cert, key=args.key)) + config["enable.partition.eof"] = True + config["client.id"] = "exact-market-lineage" + consumer = Consumer(config) + assignments = [] + for topic in ("md.raw.realtime.v2", "md.canonical.v2"): + metadata = consumer.list_topics(topic, timeout=5).topics[topic] + if metadata.error: + errors.append(dict(code="TOPIC_METADATA_UNAVAILABLE", topic=topic, kafka_code=metadata.error.code())) + continue + for partition in sorted(metadata.partitions): + tp = TopicPartition(topic, partition) + low, high = consumer.get_watermark_offsets(tp, timeout=5) + offset = max(low, high - args.tail_records) + starts[f"{topic}:{partition}"] = dict(low=low, high=high, start=offset) + assignments.append(TopicPartition(topic, partition, offset)) + consumer.assign(assignments) + evidence.put("kafka_start", partitions=starts, group_id=config["group.id"], + isolation="read_committed", manual_assignment=True) + for realm in ("live", "sandbox"): + cid = f"trading-system.{realm}.stable" + path = packet / "manifests-crypto" / f"{cid}.yaml" + manifest = yaml.safe_load(path.read_text()) + meta = manifest["metadata"] + identity = Path(args.identities) / cid + creds = RotatingJwtCredentialProvider(private_key_file=identity / "private.key", + key_id=cid + "-rs256-v1", algorithm="RS256", issuer=policy["issuer"], + audience=policy["audience"], subject=meta["subject"], environment=realm, + roles=("market_data_reader", "historical_reader", "stream_consumer"), + consumer_manifest_revision=meta["revision"], lifetime_seconds=300, + refresh_before_seconds=30) + tls = WorkloadTlsConfig(identity / "ca.crt", identity / "client.crt", identity / "client.key") + selected = [requirement(r) for r in manifest["spec"]["requirements"] + if r["instrument_uid"] in wanted and r["feed"] in {"TRADE", "MARK_INDEX_PRICE"}] + for replica in (1, 2): + transport = ExactTransport(f"https://query_v2_{replica}:8200", + credential_provider=creds, tls=tls, timeout_seconds=3) + stream = GrpcStreamTransport("unused.invalid:8210", credential_provider=creds, tls=tls) + client = AsyncDataLayerClient(query_transport=transport, stream_transport=stream, consumer_id=cid) + clients.append((cid, replica, selected, transport, client)) + + async def queries(): + while time.monotonic() < deadline: + for cid, replica, selected, transport, client in clients: + for req in selected: + remaining = deadline - time.monotonic() + if remaining <= 0: + return + row = dict(consumer_id=cid, replica=replica, uid=req.instrument_uid, + feed=req.feed.value, request_started_at_ns=time.time_ns()) + try: + async with asyncio.timeout(min(3.5, remaining)): + await client.snapshot(req) + row["status"] = "TYPED_RESPONSE" + except Exception as error: + row.update(status="REFUSED", code=getattr(error, "code", type(error).__name__), + diagnostics=getattr(error, "diagnostics", None)) + counters["query_refused"] += 1 + row["exact_view"] = transport.last + evidence.put("query", **row) + counters["query"] += 1 + await asyncio.sleep(.025) + await asyncio.sleep(.5) + + async def kafka(): + while time.monotonic() < deadline: + # Bounded microbatches yield to Query; no unbounded queue/list. + for _ in range(100): + msg = consumer.poll(0) + if msg is None: + break + key = f"{msg.topic()}:{msg.partition()}" + if msg.error(): + if msg.error().code() == KafkaError._PARTITION_EOF: + positions[key] = msg.offset() + eof[key] = dict(offset=msg.offset(), observed_at_ns=time.time_ns()) + continue + errors.append(dict(code=msg.error().code(), topic=msg.topic())) + raise RuntimeError("KAFKA_READ_ERROR") + positions[key] = msg.offset() + 1 + where = dict(topic=msg.topic(), partition=msg.partition(), offset=msg.offset(), + kafka_timestamp_ms=msg.timestamp()[1]) + if msg.topic() == "md.raw.realtime.v2": + raw = RawProviderEnvelope.FromString(msg.value()) + if raw.native_symbol not in symbols or not any( + word in raw.native_channel.lower() for word in ("trade", "mark", "index")): + continue + evidence.put("raw", **where, **raw_metadata(raw)) + counters["raw"] += 1 + else: + event = EventEnvelope.FromString(msg.value()) + if event.instrument_uid not in wanted or event.WhichOneof("payload") not in {"trade", "mark_index_price"}: + continue + record = canonical_metadata(event) + headers = dict(msg.headers() or ()) + inline = headers.get("qdl-raw-provider-envelope") + if inline: + record["inline_raw"] = raw_metadata(RawProviderEnvelope.FromString(inline)) + evidence.put("canonical", **where, feed=event.WhichOneof("payload"), event=record) + counters["canonical"] += 1 + await asyncio.sleep(.01) + + tasks = [asyncio.create_task(queries()), asyncio.create_task(kafka())] + try: + await asyncio.gather(*tasks) + finally: + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + except Exception as error: + errors.append(dict(code=type(error).__name__)) + finally: + for _, _, _, _, client in clients: + await client.close() + if consumer is not None: + consumer.close() + try: + evidence.put("summary", counts=counters, errors=errors, start_offsets=starts, + next_positions=positions, last_partition_eof=eof, elapsed_seconds=time.monotonic() - start, + historical_receipt_reconstructed=False, + quiet_certified=False, production_writes=0) + finally: + evidence.close() + print(json.dumps(dict(counts=counters, errors=errors, evidence_bytes=evidence.size))) + return not errors + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + for name in ("packet", "identities", "catalog", "output", "ca", "cert", "key"): + parser.add_argument("--" + name, required=True) + parser.add_argument("--uid", action="append", required=True) + parser.add_argument("--seconds", type=int, default=180) + parser.add_argument("--tail-records", type=int, default=400) + parser.add_argument("--bootstrap", default="kafka1:9092,kafka2:9092,kafka3:9092") + raise SystemExit(0 if asyncio.run(run(parser.parse_args())) else 1) diff --git a/scripts/probe_projector_stage_timing.py b/scripts/probe_projector_stage_timing.py new file mode 100644 index 00000000..5f7bc450 --- /dev/null +++ b/scripts/probe_projector_stage_timing.py @@ -0,0 +1,305 @@ +#!/usr/bin/env python3 +"""Read-only bounded canonical/state/Redis timing; never change runtime tuning.""" +from __future__ import annotations + +import argparse +import asyncio +import hashlib +import json +from pathlib import Path +import time + +from scripts.probe_exact_market_lineage import Evidence + + +def admitted_query_timeout(remaining_seconds): + # Stop admitting at the observation boundary, but do not shorten an RPC's + # normal timeout and misclassify window cancellation as service failure. + return 2.5 if remaining_seconds > 0 else None + + +def reference_metadata(response): + results = [] + for item in response.get("results", []): + out = {k: item[k] for k in ("instrument_uid", "product", "status", "problem") if k in item} + data = item.get("data") + if data: + out["data"] = {k: data[k] for k in ("status", "received_at_ns", "coverage", "lineage", + "error_code", "error_detail", "cache_hit", "coalesced", "quality") if k in data} + allowed = {"native_symbol", "execution_view", "freshness_basis", "source_event_time_ns", + "provider_confirmation_ns", "connection_generation", "gateway_lease_epoch", "delivery_stage", + "spool_watermark_offset", "event_recency_policy", "recency_mode", "provider_session_state", + "provider_session_liveness_ms", "provider_session_checked_at_ns", + "component_mark_received_at_ns", "component_index_received_at_ns", + "component_mark_quiet_after_ms", "component_index_quiet_after_ms"} + out["data"]["observations"] = [dict(observed_at_ns=o.get("observed_at_ns"), + labels={k:v for k,v in o.get("labels", {}).items() if k in allowed}, + field_names=[f["name"] for f in o.get("fields", [])]) for o in data.get("observations", [])] + results.append(out) + return dict(partial=response.get("partial"), results=results) + + +async def query_probe(args, evidence, deadline): + import yaml + from scripts.accept_consumer_realms import requirement + from scripts.probe_exact_market_lineage import exact_view + from qdl_sdk.client import AsyncDataLayerClient + from qdl_sdk.credentials import RotatingJwtCredentialProvider + from qdl_sdk.reference import ReferenceRequirement, ReferenceProduct + from qdl_sdk.models import Grade + from qdl_sdk.tls import WorkloadTlsConfig + from qdl_sdk.transport import RestQueryTransport, GrpcStreamTransport + packet = Path(args.packet) + cid = "trading-system.live.stable" + policy = json.loads((packet / "reader-public/jwt-config.json").read_text()) + manifest = yaml.safe_load((packet / "manifests-crypto" / (cid + ".yaml")).read_text()) + req = requirement(next(r for r in manifest["spec"]["requirements"] + if r["instrument_uid"] == args.uid and r["feed"] == "MARK_INDEX_PRICE")) + ref = ReferenceRequirement(instrument_uid=req.instrument_uid, product=ReferenceProduct.MARK_INDEX_PRICE, + consumer_grade=Grade.EXECUTION, source_policy_id=req.source_policy_id, limit=1, page_size=1, max_pages=1, + max_freshness_ms=req.max_freshness_ms, event_recency_policy=req.event_recency_policy, + max_session_liveness_ms=req.max_session_liveness_ms, require_full_coverage=True) + evidence.put("query_scope", reference_requirement=ref.model_dump(mode="json"), require_all=False, + generic_snapshot_is_not_ts_reference_path=True) + identity_path = Path(args.identity) + creds = RotatingJwtCredentialProvider(private_key_file=identity_path / "private.key", + key_id=cid + "-rs256-v1", algorithm="RS256", issuer=policy["issuer"], audience=policy["audience"], + subject=manifest["metadata"]["subject"], environment="live", + roles=("market_data_reader", "historical_reader", "stream_consumer"), + consumer_manifest_revision=manifest["metadata"]["revision"], lifetime_seconds=300, refresh_before_seconds=30) + tls = WorkloadTlsConfig(identity_path / "ca.crt", identity_path / "client.crt", identity_path / "client.key") + class ExactTransport(RestQueryTransport): + last = None + async def snapshot(self, requirement, *, consumer_id): + self.last = None + value = await super().snapshot(requirement, consumer_id=consumer_id) + self.last = exact_view(value) + return value + async def reference_batch(self, requirements, *, consumer_id, require_all): + self.last = None + value = await super().reference_batch(requirements, consumer_id=consumer_id, require_all=require_all) + self.last = reference_metadata(value) + return value + clients = [] + try: + for replica in (1, 2): + transport = ExactTransport(f"https://query_v2_{replica}:8200", credential_provider=creds, tls=tls, timeout_seconds=2) + stream = GrpcStreamTransport("unused.invalid:8210", credential_provider=creds, tls=tls) + clients.append((replica, transport, AsyncDataLayerClient(query_transport=transport, stream_transport=stream, consumer_id=cid))) + pair_id = 0 + while time.monotonic() < deadline: + pair_id += 1 + for replica, transport, client in clients: + for path in ("generic_snapshot", "ts_reference_batch"): + remaining = deadline - time.monotonic() + request_timeout = admitted_query_timeout(remaining) + if request_timeout is None: + return + row = dict(pair_id=pair_id, replica=replica, path=path, + request_started_at_ns=time.time_ns(), request_timeout_seconds=request_timeout, + observation_remaining_seconds=remaining) + try: + async with asyncio.timeout(request_timeout): + if path == "generic_snapshot": + await client.snapshot(req) + else: + await client.reference_batch((ref,), require_all=False) + row["status"] = "TYPED_RESPONSE" + except Exception as error: + row.update(status="REFUSED", code=getattr(error, "code", type(error).__name__), diagnostics=getattr(error, "diagnostics", None)) + evidence.put("query", **row, response_at_ns=time.time_ns(), exact_view=transport.last, + completed_after_observation_window=time.monotonic() > deadline) + await asyncio.sleep(.5) + finally: + for _, _, client in clients: + await client.close() + + +def classify_stage(canonical_ns, state_ns, redis_ns): + if state_ns is None: + return "CANONICAL_TO_REDIS_ONLY_STATE_UNAVAILABLE" + if state_ns < canonical_ns or redis_ns < state_ns: + return "OBSERVER_ORDER_INVERSION_NOT_NEGATIVE_LATENCY" + return "OBSERVED_STAGE_INTERVALS_NOT_BROKER_COMMIT_TIMES" + + +def counter_rate(before, after, name): + elapsed = after["at_ms"] - before["at_ms"] + delta = after["stage_a"][name] - before["stage_a"][name] + if elapsed <= 0 or delta < 0 or before["started_ms"] != after["started_ms"]: + return None + return delta * 1000 / elapsed + + +def identity(event, payload): + return dict(uid=event.instrument_uid, event_id=bytes(event.event_id).hex(), + canonical_sha256=hashlib.sha256(payload).hexdigest(), + source_event_time_ns=event.source_event_time_ns, + received_at_ns=event.received_at_ns, + source_session_id=event.source_session_id, + connection_generation=event.connection_generation, + correlation_id=event.correlation_id) + + +async def run(args): + from confluent_kafka import Consumer, KafkaError, TopicPartition + import redis.asyncio as redis + from qdl.marketdata.v2.market_data_pb2 import EventEnvelope + from qdl.projection.kn_state_codec import decode_frame, decode_latest_value, state_partition + from qdl.projection.state_contract import LogicalProductKey + from scripts.kn_canonical_mirror import source_config + + if not 1 <= args.seconds <= 180: + raise ValueError("duration must be 1..180 seconds") + lpk = LogicalProductKey("paper", "OKX", "SWAP", args.uid, "MARK_INDEX_PRICE") + product = lpk.encode() + partition = state_partition(lpk, 6) + prefix = "kn3:paper:" + pointer = prefix + "ptr:" + product + ckpt = prefix + "ckpt:md.latest.v2:" + str(partition) + evidence = Evidence(args.output, max_bytes=24 * 1024 * 1024) + config = source_config(argparse.Namespace(source_bootstrap=args.bootstrap, + group_prefix="qdl-r1-reference-parity-", ca=args.ca, cert=args.cert, key=args.key)) + # These settings affect only this observer, never a native runtime consumer. + config.update({"fetch.wait.max.ms": 20, "fetch.queue.backoff.ms": 10, + "enable.partition.eof": True, "client.id": "projector-stage-observer"}) + consumer = Consumer(config) + cache = redis.Redis.from_url(args.redis_url, socket_timeout=3, socket_connect_timeout=3) + errors = [] + counts = {"canonical": 0, "state": 0, "redis": 0, "status": 0} + started = time.monotonic() + deadline = started + args.seconds + eof = {} + try: + evidence.put("scope", schema="qdl.projector-stage-timing.v1", product=product, + duration_seconds=args.seconds, test_provenance=False, production_writes=0, + kafka_commits=False, state_partition=partition, + script_sha256=hashlib.sha256(Path(__file__).read_bytes()).hexdigest(), + observer_fetch_wait_ms=20, observer_fetch_queue_backoff_ms=10, + timing_semantics="local observations, not exact broker commit/apply timestamps") + assignments = [] + for topic in ("md.canonical.v2", "md.latest.v2"): + metadata = consumer.list_topics(topic, timeout=5).topics[topic] + if metadata.error: + error = dict(code="TOPIC_AUTHORIZATION_FAILED" if metadata.error.code() == 29 else "TOPIC_UNAVAILABLE", + kafka_code=metadata.error.code(), topic=topic) + errors.append(error) + evidence.put("access", **error) + continue + partitions = sorted(metadata.partitions) if topic == "md.canonical.v2" else [partition] + for p in partitions: + low, high = consumer.get_watermark_offsets(TopicPartition(topic, p), timeout=5) + offset = max(low, high - 100) + assignments.append(TopicPartition(topic, p, offset)) + evidence.put("start", topic=topic, partition=p, low=low, high=high, offset=offset) + consumer.assign(assignments) + + async def kafka_loop(): + while time.monotonic() < deadline: + for _ in range(100): + msg = consumer.poll(0) + if msg is None: + break + if msg.error(): + if msg.error().code() == KafkaError._PARTITION_EOF: + eof[f"{msg.topic()}:{msg.partition()}"] = dict(offset=msg.offset(), observed_at_ns=time.time_ns()) + continue + raise RuntimeError("KAFKA_READ_ERROR") + observed = time.time_ns() + if msg.topic() == "md.canonical.v2": + body = msg.value() + event = EventEnvelope.FromString(body) + if event.instrument_uid != args.uid or event.WhichOneof("payload") != "mark_index_price": + continue + kind = "canonical" + source = dict(partition=msg.partition(), offset=msg.offset()) + else: + if msg.key() != product.encode() or not msg.value(): + continue + frame = decode_frame(msg.value()) + body = frame.envelope + event = EventEnvelope.FromString(body) + source = dict(topic_id=frame.source.topic_id, partition=frame.source.partition, + offset=frame.source.offset) + kind = "state" + evidence.put(kind, observed_at_ns=observed, topic=msg.topic(), + partition=msg.partition(), offset=msg.offset(), source=source, + kafka_timestamp_type=msg.timestamp()[0], kafka_timestamp_ms=msg.timestamp()[1], + **identity(event, body)) + counts[kind] += 1 + await asyncio.sleep(.005) + + async def redis_loop(): + while time.monotonic() < deadline: + begin = time.time_ns() + ready = await cache.hmget(pointer, "ready", "fence") + if ready[0] is None: + raise RuntimeError("REDIS_NO_GENERATION") + key = prefix + "l:" + ready[0].decode() + ":" + product + async with cache.pipeline(transaction=False) as pipe: + pipe.hmget(key, "v", "t", "p", "o") + pipe.hmget(pointer, "ready", "fence") + pipe.hmget(ckpt, "next", "at_ms", "fence") + values, after, checkpoint = await pipe.execute() + end = time.time_ns() + if ready != after: + evidence.put("redis_generation_changed", request_started_at_ns=begin) + continue + if values[0] is None: + raise RuntimeError("REDIS_LATEST_MISSING") + decoded = decode_latest_value(values[0]) + if decoded.source_offset != int(values[3]): + raise RuntimeError("REDIS_SOURCE_OFFSET_MISMATCH") + event = EventEnvelope.FromString(decoded.canonical) + evidence.put("redis", request_started_at_ns=begin, response_at_ns=end, + generation=int(ready[0]), fence=int(ready[1] or 0), + source=dict(topic_id=values[1].decode(), partition=int(values[2]), offset=int(values[3])), + state_checkpoint_next_offset=int(checkpoint[0]) if checkpoint[0] else None, + state_checkpoint_at_ms=int(checkpoint[1]) if checkpoint[1] else None, + state_checkpoint_fence=int(checkpoint[2]) if checkpoint[2] else None, + **identity(event, decoded.canonical)) + counts["redis"] += 1 + await asyncio.sleep(.025) + + async def status_loop(): + previous = {} + while time.monotonic() < deadline: + for i, path in enumerate(args.status): + status = json.loads(Path(path).read_text()) + if status["at_ms"] != previous.get(i): + evidence.put("status", replica=i + 1, status=status) + counts["status"] += 1 + previous[i] = status["at_ms"] + await asyncio.sleep(.5) + + tasks = [asyncio.create_task(query_probe(args, evidence, deadline))] + tasks.extend(asyncio.create_task(fn()) for fn in (kafka_loop, redis_loop, status_loop)) + try: + await asyncio.gather(*tasks) + finally: + for task in tasks: + task.cancel() + await asyncio.gather(*tasks, return_exceptions=True) + except Exception as error: + errors.append(dict(code=type(error).__name__)) + finally: + consumer.close() + await cache.aclose() + evidence.put("summary", counts=counts, errors=errors, last_eof=eof, + elapsed_seconds=time.monotonic() - started, production_writes=0, + stage_split_proven=False) + evidence.close() + print(json.dumps(dict(counts=counts, errors=errors))) + return not errors + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + for name in ("ca", "cert", "key", "output", "packet", "identity"): + parser.add_argument("--" + name, required=True) + parser.add_argument("--status", action="append", default=[]) + parser.add_argument("--seconds", type=int, default=180) + parser.add_argument("--uid", default="fb26214c-7b9b-5961-95b2-55154755af0f") + parser.add_argument("--redis-url", default="redis://market_cache:6379/0") + parser.add_argument("--bootstrap", default="kafka1:9092,kafka2:9092,kafka3:9092") + raise SystemExit(0 if asyncio.run(run(parser.parse_args())) else 1) diff --git a/tests/test_binance_history_overlap.py b/tests/test_binance_history_overlap.py new file mode 100644 index 00000000..ec836c70 --- /dev/null +++ b/tests/test_binance_history_overlap.py @@ -0,0 +1,245 @@ +"""Test-only synthetic OHLCV; provider-observed 3d transition timestamps.""" +from dataclasses import dataclass, replace +import json +from types import SimpleNamespace +import unittest +from unittest.mock import Mock, patch + +from qdl.adapters.binance.bar_edge import fetch_closed_bar_history_raw_envelopes +from qdl.adapters.intervals import BarHistoryGapError, BarHistoryOverlapError +from qdl.query import ConsumerGrade, CoverageStatus, DataRequirement, FeedType, HistoryResult +from qdl.query.contracts import CanonicalErrorCode, evaluate_requirement +from qdl.runtime.kn_query_backend import KnMarketCacheQueryBackend +from qdl.runtime.stable_bar_edge import StableBinanceBarEdge +import tests.test_kn_history_fill as fixtures + +DURATION = 259_200_000 +START = 1_692_144_000_000 + + +def row(open_ms): + value = fixtures.ShortProviderHistoryTests._row(open_ms) + value[6] = open_ms + DURATION - 1 + return value + + +def binding(): + return replace(fixtures.ShortProviderHistoryTests()._binding(), + native_symbol="ETHUSDT", interval="3d") + + +def provider(rows): + def fetch(_symbol, *, end_time, limit, **kwargs): + return {"data": [value for value in rows if value[0] <= end_time][-limit:]} + return Mock(side_effect=fetch) + + +def overlapping_rows(): + return [row(START - 86_400_000 * 2)] + [row(START + i * DURATION) for i in range(3)] + + +def fetch(fetcher, **kwargs): + return fetch_closed_bar_history_raw_envelopes( + binding(), now_ms=START + 3 * DURATION + 1, attempts=1, + fetcher=fetcher, test_provenance=True, **kwargs, + ) + + +class SuffixOutcomeTests(unittest.TestCase): + def test_same_page_suffix_raises_and_preserves_exact_native_rows(self): + rows = overlapping_rows() + call = provider(rows) + with self.assertRaises(BarHistoryOverlapError) as caught: + fetch(call, limit=10000, allow_short=True) + error = caught.exception + self.assertEqual(error.requested_rows, 10000) + self.assertEqual(error.older_prefix_status, "UNCERTIFIED_PROVIDER_OVERLAP") + self.assertEqual([json.loads(v.raw_frame_bytes)["row"] for v in error.recent_envelopes], rows[1:]) + self.assertEqual(call.call_count, 1, "do not fetch an older page or probe exhaustion after overlap") + self.assertEqual(len(fetch(provider(rows), limit=3)), 3) + + def test_cross_page_overlap_retains_all_verified_newer_rows(self): + call = provider(overlapping_rows()) + with patch("qdl.adapters.binance.bar_edge._HISTORY_PAGE_ROWS", 3): + with self.assertRaises(BarHistoryOverlapError) as caught: + fetch(call, limit=10000, allow_short=True) + self.assertEqual(len(caught.exception.recent_envelopes), 3) + self.assertEqual(call.call_count, 2) + + def test_only_newest_contiguous_segment_can_be_materialized(self): + rows = [row(START - 86_400_000 * 4), row(START - 86_400_000 * 2)] + overlapping_rows()[1:] + with self.assertRaises(BarHistoryOverlapError) as caught: + fetch(provider(rows), limit=5) + self.assertEqual(caught.exception.previous, START - 86_400_000 * 2) + self.assertEqual(len(caught.exception.recent_envelopes), 3) + + def test_off_grid_stale_and_missing_suffix_never_offer_materialization(self): + for rows in ( + [row(value[0] - 1) for value in overlapping_rows()], + overlapping_rows()[:-1], + [row(START - DURATION * 2), row(START), row(START + 2 * DURATION)], + ): + with self.subTest(opens=[r[0] for r in rows]): + with self.assertRaises(BarHistoryGapError) as caught: + fetch(provider(rows), limit=len(rows)) + self.assertNotIsInstance(caught.exception, BarHistoryOverlapError) + + def test_invalid_close_rejected_before_any_suffix_is_offered(self): + rows = overlapping_rows() + rows[-1][6] -= 1 + with self.assertRaises(ValueError): + fetch(provider(rows), limit=4) + + +class SuffixMaterializationTests(unittest.TestCase): + def edge(self): + fixture = fixtures.EdgeFillTests() + edge = fixture._edge(demand=None, gate=None) + source = edge.history_bindings[0][0] + source.interval = "3d" + acquisition = SimpleNamespace(runtime="BINANCE") + edge.history_bindings = ((source, acquisition),) + edge._history_short = {"b": (1, 0), "c": (1, 0)} + edge._settled_observed_ms = lambda: START + 3 * DURATION + 1 + edge._assert_repair_writer_current = lambda: None + edge._assert_canonical_cache_identity = lambda: None + edge._persist_state = Mock() + edge._record_serving = Mock() + edge._publish_history = StableBinanceBarEdge._publish_history.__get__(edge) + durable = {START + 2 * DURATION} + edge._durable_final_bar_opens = lambda _source, opens: frozenset(durable & opens) + emitted = [] + + def publish(values): + values = tuple(values) + emitted.extend(values) + durable.update(json.loads(value.raw_frame_bytes)["row"][0] for value in values) + return tuple(object() for _ in values) + + edge.publisher = SimpleNamespace(publish_many=Mock(side_effect=publish)) + call = provider(overlapping_rows()) + edge._fetch_history = lambda _source, _acquisition, rows, **kwargs: fetch( + call, limit=rows, allow_short=True) + return edge, call, durable, emitted + + def test_bootstrap_materializes_recent_three_but_never_completes_deep_history(self): + edge, call, durable, emitted = self.edge() + self.assertEqual(edge.bootstrap_history(), 2) + self.assertEqual(durable, {START + i * DURATION for i in range(3)}) + self.assertEqual(len(emitted), 2, "existing newest bar is not republished") + self.assertTrue(edge._live_history_ready("a")) + self.assertEqual(edge._last_open_ms["a"], START + 2 * DURATION) + self.assertFalse(edge._history_bootstrapped) + self.assertNotIn("a", edge._history_short) + self.assertEqual(edge._history_suspended["a"]["status"], "UNCERTIFIED_PROVIDER_OVERLAP") + for now in (31, 3601, 86400): + edge.clock = lambda: now + edge.bootstrap_history() + self.assertEqual(call.call_count, 1) + self.assertEqual(edge.publisher.publish_many.call_count, 1) + self.assertEqual(edge._history_loop_delay(86400), 1) + edge.clock = lambda: 86401 + self.assertEqual(edge.bootstrap_history(), 0) + self.assertEqual(call.call_count, 2) + self.assertFalse(edge._history_bootstrapped) + edge.canonical_cache_id = "test-cache" + edge.connection_generation = 1 + edge._state_identity_payload = lambda **kwargs: {"schema": kwargs["schema"]} + self.assertEqual(edge._state_payload()["last_open_ms"], {}, "restart cannot certify suffix as full history") + + def test_next_final_bar_advances_while_deep_history_remains_suspended(self): + edge, call, durable, emitted = self.edge() + edge.bootstrap_history() + edge.bindings = edge.history_bindings + edge.okx_bindings = () + edge._rest_fallback_active = True + edge.max_catchup_rows = 10000 + edge._binding_is_due = lambda *args, **kwargs: True + latest = fetch_closed_bar_history_raw_envelopes( + binding(), now_ms=START + 4 * DURATION + 1, limit=1, attempts=1, + fetcher=provider([row(START + 3 * DURATION)]), test_provenance=True)[0] + edge._fetch_latest = Mock(return_value=latest) + edge._settled_observed_ms = lambda: START + 4 * DURATION + 1 + self.assertEqual(edge.run_cycle(), 1) + self.assertEqual(edge._last_open_ms["a"], START + 3 * DURATION) + self.assertIn(START + 3 * DURATION, durable) + self.assertFalse(edge._history_bootstrapped) + self.assertIn("a", edge._history_retry) + self.assertIn("a", edge._history_suspended) + self.assertEqual(call.call_count, 1) + + def test_empty_recheck_cannot_certify_previously_rejected_history(self): + edge, call, durable, emitted = self.edge() + edge.bootstrap_history() + edge.clock = lambda: 86401 + edge._fetch_history = lambda *args, **kwargs: () + self.assertEqual(edge.bootstrap_history(), 0) + self.assertFalse(edge._history_bootstrapped) + self.assertIn("a", edge._history_suspended) + self.assertIn("a", edge._history_retry) + self.assertNotIn("a", edge._history_short) + + def test_sink_failure_reuses_pending_provider_bytes_without_refetch(self): + edge, call, durable, emitted = self.edge() + publish = edge.publisher.publish_many.side_effect + edge.publisher.publish_many.side_effect = OSError("test-only sink failure") + with self.assertRaises(OSError): + edge.bootstrap_history() + self.assertNotIn("a", edge._last_open_ms) + self.assertFalse(edge._history_bootstrapped) + self.assertEqual(call.call_count, 1) + edge.clock = lambda: 31 + edge.publisher.publish_many.side_effect = publish + self.assertEqual(edge.bootstrap_history(), 2) + self.assertEqual(call.call_count, 1) + self.assertEqual(len(durable), 3) + self.assertFalse(edge._history_suffix_pending) + + def test_verified_provider_correction_can_complete_without_rewinding_live(self): + edge, call, durable, emitted = self.edge() + edge.bootstrap_history() + edge.clock = lambda: 86401 + edge._last_open_ms["a"] = START + 3 * DURATION + edge._fetch_history = lambda *args, **kwargs: fetch(provider(overlapping_rows()), limit=3) + edge.bootstrap_history() + self.assertTrue(edge._history_bootstrapped) + self.assertFalse(edge._history_suspended) + self.assertFalse(edge._history_retry) + self.assertEqual(edge._last_open_ms["a"], START + 3 * DURATION) + + +@dataclass(frozen=True) +class Item: + snapshot_id: str = "test-only" + + +class SuffixQueryCoverageTests(unittest.TestCase): + def result(self, requested, *, venue="BINANCE", interval="3d", start_ns=None): + backend = KnMarketCacheQueryBackend.__new__(KnMarketCacheQueryBackend) + source = SimpleNamespace(feed=FeedType.BAR, interval=interval, + instrument=SimpleNamespace(identity=SimpleNamespace(venue=venue))) + backend.catalog = SimpleNamespace(binding_for=lambda _: source) + backend._requested_window = lambda _: (requested, start_ns, 4, None) + records = tuple(SimpleNamespace(envelope=SimpleNamespace(bar=SimpleNamespace(open_time_ns=i))) for i in (1, 2, 3)) + view = SimpleNamespace(boundary=SimpleNamespace(offset=3)) + backend._history_view = lambda *args: (view, records) + backend._history_from_records = lambda *args, **kwargs: HistoryResult( + items=tuple(Item() for _ in records), coverage=CoverageStatus.FULL, + snapshot_id="test-only", stream_cursor="test-only", watermark_offset=3, data_as_of_ns=3) + with patch("qdl.runtime.kn_query_backend.view_snapshot_id", return_value="test-only"), patch( + "qdl.runtime.kn_query_backend.source_placeholder", return_value="test-only"): + return backend.history_with_envelopes(None)[0] + + def test_exact_recent_request_full_but_uncertified_older_prefix_is_partial(self): + self.assertEqual(self.result(3).coverage, CoverageStatus.FULL) + result = self.result(10000) + self.assertEqual(result.coverage, CoverageStatus.PARTIAL) + requirement = DataRequirement(instrument_uid="test-only", feed=FeedType.BAR, + interval="3d", consumer_grade=ConsumerGrade.ALPHA, + source_policy_id="test-only", warmup_limit=10000) + problem = evaluate_requirement(requirement, coverage=result.coverage, entitled=True, + available=True, fresh=True, authoritative=True, gap_open=False) + self.assertEqual(problem.code, CanonicalErrorCode.PARTIAL_RESULT) + self.assertEqual(self.result(10000, venue="OKX").coverage, CoverageStatus.FULL) + self.assertEqual(self.result(10000, interval="1m").coverage, CoverageStatus.FULL) + self.assertEqual(self.result(10000, start_ns=1).coverage, CoverageStatus.FULL) diff --git a/tests/test_consumer_key_realms.py b/tests/test_consumer_key_realms.py new file mode 100644 index 00000000..3cc247ea --- /dev/null +++ b/tests/test_consumer_key_realms.py @@ -0,0 +1,115 @@ +"""Public market storage is shared; workload keys/subjects/realms are not.""" +import unittest +from dataclasses import replace + +from qdl.consumer.manifest import ConsumerManifestLoader, ConsumerManifestRegistry +from qdl.security.data_plane import DataPlaneAccessError, DataPlaneIdentityService, DataPlaneSecurityConfig +from tests.phase7_support import TEST_AUDIENCE, TEST_ISSUER, make_token, manifest_mapping + + +class ConsumerKeyRealmTests(unittest.TestCase): + def setUp(self): + self.realms = ("paper", "sandbox", "live") + self.manifests = {} + for realm in self.realms: + mapping = manifest_mapping(consumer_id="ts." + realm, + subject="spiffe://qdl/" + realm + "/ts", instrument_uid="instrument-test") + mapping["metadata"]["environment"] = realm + mapping["spec"]["execution_dependency"] = "ALLOWED" + self.manifests[realm] = ConsumerManifestLoader.from_mapping(mapping) + self.config = DataPlaneSecurityConfig(environment="paper", issuer=TEST_ISSUER, + audience=TEST_AUDIENCE, algorithms=("HS256",), + keys_by_id={r: ("test-only-" + r).encode() * 8 for r in self.realms}, + subjects_by_key_id={r: m.subject for r, m in self.manifests.items()}, + environments_by_key_id={r: r for r in self.realms}) + self.identity = DataPlaneIdentityService(self.config, ConsumerManifestRegistry(tuple(self.manifests.values()))) + + def token(self, realm, **kwargs): + defaults = dict(environment=realm, key_id=realm, secret=self.config.keys_by_id[realm]) + defaults.update(kwargs) + return make_token(self.manifests[realm].subject, **defaults) + + def test_each_mode_authenticates_without_changing_storage_realm(self): + for realm, manifest in self.manifests.items(): + with self.subTest(realm=realm): + access = self.identity.authenticate(self.token(realm), consumer_id=manifest.consumer_id) + self.assertEqual(access.principal.environment, realm) + self.assertEqual(access.manifest, manifest) + self.assertEqual(self.config.environment, "paper") + + def test_key_cannot_assert_another_realm_or_consumer(self): + for realm in self.realms: + for other in set(self.realms) - {realm}: + with self.subTest(realm=realm, other=other): + with self.assertRaises(DataPlaneAccessError): + self.identity.authenticate(self.token(realm, environment=other), consumer_id="ts." + other) + with self.assertRaises(DataPlaneAccessError): + self.identity.authenticate(self.token(realm), consumer_id="ts." + other) + with self.assertRaises(DataPlaneAccessError): + self.identity.authenticate(self.token(realm, key_id=other), consumer_id="ts." + realm) + + def test_subject_and_revision_stay_bound(self): + for realm in self.realms: + with self.assertRaises(DataPlaneAccessError): + self.identity.authenticate(self.token(realm, manifest_revision=2), consumer_id="ts." + realm) + forged = make_token("spiffe://qdl/" + realm + "/other", environment=realm, + key_id=realm, secret=self.config.keys_by_id[realm]) + with self.assertRaises(DataPlaneAccessError): + self.identity.authenticate(forged, consumer_id="ts." + realm) + + def test_unconfigured_legacy_single_realm_stays_closed(self): + legacy = DataPlaneIdentityService(replace(self.config, environments_by_key_id=None), + ConsumerManifestRegistry(tuple(self.manifests.values()))) + legacy.authenticate(self.token("paper"), consumer_id="ts.paper") + for realm in ("sandbox", "live"): + with self.assertRaises(DataPlaneAccessError): + legacy.authenticate(self.token(realm), consumer_id="ts." + realm) + + def test_mapping_is_exact_and_has_no_wildcard(self): + for value in ({}, {"live": "live"}, {r: "*" for r in self.realms}, [], "live"): + with self.subTest(value=value), self.assertRaises(ValueError): + replace(self.config, environments_by_key_id=value) + with self.assertRaises(ValueError): + replace(self.config, subjects_by_key_id=None) + + def test_live_and_sandbox_do_not_accept_paper_only_contract(self): + for realm in ("sandbox", "live"): + with self.assertRaises(ValueError): + replace(self.manifests[realm], execution_dependency="PAPER_ONLY") + + +class ConsumerRealmCompilerTests(unittest.TestCase): + def test_stable_scopes_preserved_for_both_additional_modes(self): + from pathlib import Path + import yaml + from scripts.compile_consumer_realms import compile_realm + root = Path(__file__).resolve().parents[1] + for name in ("trading-system-paper", "alpha-binance-paper", "alpha-okx-paper"): + payload = yaml.safe_load((root / "consumers/stable" / (name + ".yaml")).read_text()) + original = ConsumerManifestLoader.from_mapping(payload) + for realm in ("sandbox", "live"): + result = ConsumerManifestLoader.from_mapping(compile_realm(payload, realm)) + self.assertEqual(result.requirements, tuple(r for r in original.requirements + if r.source_policy_id in {"crypto_primary_v2", "crypto_liquid_v2"})) + self.assertFalse(any(r.source_policy_id == "vn_primary_v2" for r in result.requirements)) + if name == "trading-system-paper": + self.assertEqual(len(result.requirements), 60) + self.assertEqual(result.quotas, original.quotas) + self.assertEqual(result.allowed_permissions, original.allowed_permissions) + self.assertEqual(result.allowed_purposes, original.allowed_purposes) + self.assertNotEqual(result.subject, original.subject) + self.assertNotEqual(result.consumer_id, original.consumer_id) + self.assertEqual(result.environment, realm) + self.assertEqual(payload["metadata"]["environment"], "paper") + self.assertEqual(payload["spec"]["execution_dependency"], original.execution_dependency) + self.assertEqual(result.execution_dependency, + "ALLOWED" if original.execution_dependency == "PAPER_ONLY" else "FORBIDDEN") + + def test_other_consumer_is_not_implicitly_promoted(self): + from scripts.compile_consumer_realms import compile_realm + payload = manifest_mapping(consumer_id="unapproved.paper.stable", + subject="spiffe://qdl/paper/unapproved", instrument_uid="instrument-test") + for policy in ("FORBIDDEN", "PAPER_ONLY", "ALLOWED"): + payload["spec"]["execution_dependency"] = policy + with self.assertRaises(ValueError): + compile_realm(payload, "live") diff --git a/tests/test_consumer_realm_execution_path.py b/tests/test_consumer_realm_execution_path.py new file mode 100644 index 00000000..21cbb735 --- /dev/null +++ b/tests/test_consumer_realm_execution_path.py @@ -0,0 +1,42 @@ +import unittest +from types import SimpleNamespace +from unittest.mock import AsyncMock + +from qdl_sdk import DataRequirement, Feed, Grade, StalePolicy +from scripts.accept_consumer_realms import execution_reference, read_execution_reference + + +class ExecutionPathTests(unittest.IsolatedAsyncioTestCase): + def requirement(self, feed=Feed.MARK_INDEX_PRICE): + return DataRequirement(instrument_uid="test-only", feed=feed, + consumer_grade=Grade.EXECUTION, source_policy_id="test-policy", + max_freshness_ms=2000, event_recency_policy=StalePolicy.OBSERVE, + max_session_liveness_ms=2000) + + def test_ts_reference_contract_preserves_signed_limits(self): + ref = execution_reference(self.requirement()) + self.assertEqual((ref.limit, ref.page_size, ref.max_pages), (1, 1, 1)) + self.assertTrue(ref.require_full_coverage) + self.assertEqual(ref.max_freshness_ms, 2000) + self.assertEqual(ref.max_session_liveness_ms, 2000) + self.assertEqual(ref.event_recency_policy, StalePolicy.OBSERVE) + self.assertEqual(ref.source_policy_id, "test-policy") + + def test_trade_is_not_silently_replaced(self): + with self.assertRaises(ValueError): + execution_reference(self.requirement(Feed.TRADE)) + + async def test_item_refusal_is_preserved_not_accepted(self): + payload = {"partial": True, "results": [{"instrument_uid": "test-only", + "product": "MARK_INDEX_PRICE", "status": "ERROR", + "problem": {"code": "DATA_STALE"}}]} + response = SimpleNamespace(model_dump=lambda **kw: payload) + client = SimpleNamespace(reference_batch=AsyncMock(return_value=response)) + evidence = await read_execution_reference(client, self.requirement()) + self.assertEqual(evidence, payload) + self.assertIs(client.reference_batch.call_args.kwargs["require_all"], False) + self.assertEqual(client.reference_batch.call_count, 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_execution_mark_index_live_view.py b/tests/test_execution_mark_index_live_view.py index 30328ef9..b74227c8 100644 --- a/tests/test_execution_mark_index_live_view.py +++ b/tests/test_execution_mark_index_live_view.py @@ -1239,6 +1239,43 @@ async def handler(request: httpx.Request) -> httpx.Response: self.assertEqual(result.status, ReferenceStatus.ERROR) self.assertEqual(result.error_code, "LIVE_VIEW_PROTOCOL") + async def test_reader_preserves_typed_quiet_rejection_without_retry_or_rest(self): + cases = { + "STALE": "LIVE_VIEW_STALE", + "COMPONENT_STALE": "LIVE_VIEW_STALE", + "SESSION_LIVENESS": "LIVE_VIEW_STALE", + "SESSION_STATE": "LIVE_VIEW_UNAVAILABLE", + "QUIET_POLICY_UNAVAILABLE": "LIVE_VIEW_UNAVAILABLE", + "LINEAGE_INVALID": "LIVE_VIEW_UNAVAILABLE", + "GAP_OR_RESYNC": "LIVE_VIEW_GAPPED", + "IDENTITY_MISMATCH": "LIVE_VIEW_IDENTITY", + "FENCED": "LIVE_VIEW_UNAVAILABLE", + } + for reason, expected in cases.items(): + with self.subTest(reason=reason): + calls = [] + async def handler(request): + calls.append(request.url.host) + return httpx.Response(409, json={ + "detail": "execution MARK/INDEX live view unavailable:" + reason, + }, request=request) + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client: + reader = HttpExecutionMarkIndexReader( + ("https://stream_v2_active:8200",), SECRET, client=client, + ) + result = await reader.fetch( + ReferenceRequest(self.record, ReferenceProduct.MARK_INDEX_PRICE), + max_freshness_ms=2000, source_policy_id="crypto_liquid_v2", + event_recency_policy=StalePolicy.OBSERVE, + max_session_liveness_ms=45000, + ) + self.assertEqual(result.status, ReferenceStatus.ERROR) + self.assertEqual(result.error_code, expected) + self.assertEqual(calls, ["stream_v2_active"]) + self.assertTrue(result.error_detail.endswith(reader._bounded_reason( + httpx.Response(409, json={"detail": "unavailable:" + reason}) + ))) + async def test_reader_uses_one_deadline_across_active_passive_urls(self): envelope = _envelope(self.binding, sequence=50) payload = { diff --git a/tests/test_kn_history_fill.py b/tests/test_kn_history_fill.py index 624320ba..f025d1a4 100644 --- a/tests/test_kn_history_fill.py +++ b/tests/test_kn_history_fill.py @@ -31,6 +31,55 @@ def _bundle(): return {"catalog": {"bindings": bindings}, "manifests": manifests} +class BinanceNativeThreeDayTests(unittest.TestCase): + """Synthetic OHLCV with provider-observed timestamps; never provider replay.""" + + def test_recent_exact_window_and_overlap_never_become_truncated_success(self): + from dataclasses import replace + import json + from unittest.mock import patch + from qdl.adapters.binance.bar_edge import fetch_closed_bar_history_raw_envelopes + from qdl.adapters.intervals import BarHistoryGapError + + duration = 259200000 + for symbol, previous, current in ( + ("BTCUSDT", 1569110400000, 1569283200000), + ("ETHUSDT", 1691971200000, 1692144000000), + ("BNBUSDT", 1691971200000, 1692144000000), + ("SOLUSDT", 1692057600000, 1692144000000), + ("DOGEUSDT", 1692057600000, 1692144000000), + ): + binding = replace(ShortProviderHistoryTests()._binding(), + native_symbol=symbol, interval="3d") + rows = [ShortProviderHistoryTests._row(t) for t in + (previous - duration, previous, current, current + duration)] + for row in rows: + row[6] = row[0] + duration - 1 + saved = [list(row) for row in rows] + calls = [] + + def fetch(_symbol, *, limit, end_time, **kwargs): + calls.append((limit, end_time)) + return {"data": [row for row in rows if row[0] <= end_time][-limit:]} + + kwargs = dict(now_ms=current + 3 * duration, attempts=1, + fetcher=fetch, test_provenance=True) + with self.subTest(symbol=symbol, window="recent"): + values = fetch_closed_bar_history_raw_envelopes(binding, limit=2, **kwargs) + self.assertEqual([json.loads(v.raw_frame_bytes)["row"] for v in values], rows[-2:]) + for page_size, limit, allow_short in ((1000, 4, False), (2, 4, False), (1000, 10, True)): + with self.subTest(symbol=symbol, page_size=page_size, allow_short=allow_short): + with patch("qdl.adapters.binance.bar_edge._HISTORY_PAGE_ROWS", page_size): + with self.assertRaises(BarHistoryGapError) as caught: + fetch_closed_bar_history_raw_envelopes( + binding, limit=limit, allow_short=allow_short, **kwargs) + self.assertEqual(caught.exception.kind, "OVERLAPPING_PROVIDER_WINDOWS") + self.assertEqual((caught.exception.previous, caught.exception.current), + (previous, current)) + self.assertEqual(rows, saved) + self.assertLessEqual(len(calls), 6) + + class DemandTests(unittest.TestCase): def test_the_largest_demand_of_a_requiring_manifest_per_bar_binding(self): self.assertEqual(demanded_history_rows(_bundle()), {"b-1m": 10000, "b-1h": 500, "b-nobody": 0}) @@ -231,6 +280,59 @@ def publish(source, _acquisition, values, *, expected_rows): edge._publish_history = publish return edge + def test_overlap_rechecks_daily_without_completing_or_starving_other_history(self): + from qdl.adapters.intervals import BarHistoryGapError + edge = self._edge(demand=None, gate=None) + original = edge._fetch_history + calls = [] + + def refused(source, acquisition, **kwargs): + if source.binding_id == "a": + calls.append(source.binding_id) + raise BarHistoryGapError("Binance", "TEST_ONLY", "3d", + 1691971200000, 1692144000000) + return original(source, acquisition, **kwargs) + + edge._fetch_history = refused + edge.bootstrap_history() + self.assertEqual(edge._history_retry, {"a": (1, 86401.0)}) + self.assertNotIn("a", edge._last_open_ms) + self.assertNotIn("a", edge._history_short) + self.assertFalse(edge._history_bootstrapped) + self.assertEqual(self.published, [("b", 7)]) + edge._rest_fallback_active = False + self.assertEqual(edge._loop_sleep_seconds(1), 60) + for now in (31, 61, 3601, 86400): + edge.clock = lambda: now + edge.bootstrap_history() + self.assertEqual(calls, ["a"]) + edge.clock = lambda: 86401.0 + edge.bootstrap_history() + self.assertEqual(calls, ["a", "a"]) + self.assertEqual(edge._history_retry, {"a": (2, 172801.0)}) + edge.clock = lambda: 172801.0 + edge._fetch_history = original + edge.bootstrap_history() + self.assertEqual(edge._history_retry, {}) + self.assertTrue(edge._history_bootstrapped) + + def test_missing_window_keeps_bounded_transient_retry_and_legacy_raises(self): + from qdl.adapters.intervals import BarHistoryGapError + error = BarHistoryGapError("Binance", "TEST_ONLY", "1m", 60000, 180000) + self.assertEqual([error.retry_delay_seconds(n) for n in (1, 2, 5, 1000)], + [2, 4, 30, 30]) + edge = self._edge(demand=None, gate=None) + + def refused(*args, **kwargs): + raise error + + edge._fetch_history = refused + edge.bootstrap_history(max_bindings=1) + self.assertEqual(edge._history_retry, {"a": (1, 3.0)}) + edge.bar_readback = None + with self.assertRaises(BarHistoryGapError): + edge.bootstrap_history(max_bindings=1) + def test_depth_by_demand_and_short_or_empty_history_are_reported(self): edge = self._edge(demand={"a": 5000, "b": 10_000, "c": 0}, gate=None) edge.bootstrap_history() diff --git a/tests/test_kn_query_backend.py b/tests/test_kn_query_backend.py index ed1b15a3..0edfb082 100644 --- a/tests/test_kn_query_backend.py +++ b/tests/test_kn_query_backend.py @@ -395,6 +395,29 @@ def test_bar_history_matches_the_spool_semantics_at_the_view_boundary(self): self.assertEqual(item.payload, spool.items[-1].payload) self.assertEqual(parse_placeholder(item.cursor), (TOPIC_ID, 3, 4_321)) + def test_binance_3d_suffix_does_not_certify_the_uncaptured_older_prefix(self): + from qdl.query import CoverageStatus + from qdl.query.contracts import evaluate_requirement + + pk, real_bar = next((pk, raw) for pk, raw in golden_records(bar=True) + if pk.endswith("binance-usdm-dogeusdt-bar-3d-primary-v2")) + binding = binding_of(self.catalog, pk, real_bar) + rows = [derived_bar(real_bar, -shift) for shift in range(3)] + self.now_ns = (open_ms_of(real_bar) + canonical_interval_ms("3d")) * 1_000_000 + 5_000_000_000 + self.put_bars(binding, rows) + backend = self.backend() + recent = backend.history(requirement(binding, 3)) + self.assertEqual(len(recent.items), 3) + self.assertEqual(recent.coverage, CoverageStatus.FULL) + full_request = requirement(binding, 10000) + incomplete = backend.history(full_request) + self.assertEqual(len(incomplete.items), 3) + self.assertEqual(incomplete.coverage, CoverageStatus.PARTIAL) + problem = evaluate_requirement(full_request, coverage=incomplete.coverage, + entitled=True, available=True, fresh=True, + authoritative=True, gap_open=False) + self.assertEqual(problem.code, CanonicalErrorCode.PARTIAL_RESULT) + def test_a_time_range_reads_exactly_its_opens(self): payloads = self.history_rows(300) self.put_bars(self.bar_binding, payloads, offsets=[10 + index for index in range(300)], mark=400) diff --git a/tests/test_okx_inverse_sandbox_demand.py b/tests/test_okx_inverse_sandbox_demand.py new file mode 100644 index 00000000..6301d412 --- /dev/null +++ b/tests/test_okx_inverse_sandbox_demand.py @@ -0,0 +1,93 @@ +"""Contract fixtures only; these tests do not activate a broker or consumer.""" +from pathlib import Path +from dataclasses import replace +import tempfile +import yaml + +from qdl.query import ConsumerGrade, FeedType +import unittest + +from qdl.runtime.production_catalog import ProductionCatalogBuilder, ProductionDemandManifest + +ROOT = Path(__file__).resolve().parents[1] +DEMAND = ROOT / "config/v2/okx-inverse-sandbox-demand.yaml" + + +class InverseSandboxDemandTests(unittest.TestCase): + def test_additive_scope_preserves_existing_demand(self): + path = ROOT / "config/v2/stable-crypto-demand.yaml" + old = ProductionDemandManifest.load_many([path]) + inverse = ProductionDemandManifest.load_many([DEMAND]) + merged = ProductionDemandManifest.load_many([path, DEMAND]) + self.assertEqual(len(inverse.demands), 4) + self.assertEqual(len(merged.demands), len(old.demands) + 4) + self.assertTrue(all(row in merged.demands for row in old.demands)) + for row in inverse.demands: + self.assertEqual(row.consumer_id, "trading-system.sandbox.stable") + self.assertEqual(row.native_symbol, "BTC-USD-SWAP") + self.assertIsNone(row.interval) + self.assertEqual({r.feed.value for r in inverse.demands}, + {"QUOTE", "MARK_INDEX_PRICE", "BOOK_SNAPSHOT", "BOOK_DELTA"}) + + def _compile(self, demand): + return ProductionCatalogBuilder( + catalog_revision=1, source_policy_revision=1, authority_revision=1, + ).build(demand=demand, binance_usdm=None, okx_rows=[{ + "instType": "SWAP", "instId": "BTC-USD-SWAP", "instFamily": "BTC-USD", + "baseCcy": "", "quoteCcy": "", "ctValCcy": "USD", "ctType": "inverse", + "settleCcy": "BTC", "ctVal": "100", "ctMult": "1", "state": "live", + "tickSz": "0.1", "lotSz": "0.1", + }]) + + def test_native_compilation_keeps_inverse_identity_and_index(self): + bundle = self._compile(ProductionDemandManifest.load_many([DEMAND])) + record = bundle.source_catalog["instruments"][0] + self.assertEqual(record["native_symbol"], "BTC-USD-SWAP") + self.assertEqual(record["base_asset"], "BTC") + self.assertEqual(record["quote_asset"], "USD") + self.assertEqual(record["settlement_asset"], "BTC") + rows = bundle.acquisition_plan["bindings"] + self.assertEqual(len(rows), 4) + self.assertTrue(all(r["mode"] == "RUST_NATIVE" for r in rows)) + mark = next(r for r in rows if r["provider_kind"] == "okx_mark_index") + self.assertEqual(mark["mark_index"]["index_native_symbol"], "BTC-USD") + books = [r for r in rows if r["provider_kind"] == "okx_book"] + self.assertEqual(len(books), 2) + self.assertTrue(all(r["native_channel"] == "books" for r in books)) + self.assertTrue(all(r["sequence_policy"] == "CONTIGUOUS" for r in books)) + + self.assertTrue(all(r["l2"]["materialized_snapshot_interval_ms"] == 1000 for r in books)) + self.assertTrue(all(r["l2"]["snapshot_refresh_seconds"] == 30 for r in books)) + + def test_research_only_keeps_provider_default_without_hot_snapshot(self): + demand = ProductionDemandManifest.load_many([DEMAND]) + research = replace(demand, demands=tuple(replace(r, consumer_grade=ConsumerGrade.ALPHA) for r in demand.demands)) + bundle = self._compile(research) + books = [r for r in bundle.acquisition_plan["bindings"] if "l2" in r] + self.assertEqual(len(books), 2) + self.assertTrue(all("materialized_snapshot_interval_ms" not in r["l2"] for r in books)) + current = self._compile(demand) + for old, new in zip(bundle.acquisition_plan["bindings"], current.acquisition_plan["bindings"]): + if "l2" not in old: + self.assertEqual(old, new) + + def test_mixed_alias_grade_uses_one_physical_execution_book(self): + demand = ProductionDemandManifest.load_many([DEMAND]) + mixed = replace(demand, demands=tuple(replace(r, consumer_grade=ConsumerGrade.ALPHA) + if r.feed is FeedType.BOOK_SNAPSHOT else r for r in demand.demands)) + books = [r["l2"] for r in self._compile(mixed).acquisition_plan["bindings"] if "l2" in r] + self.assertEqual(books[0], books[1]) + self.assertEqual(books[0]["materialized_snapshot_interval_ms"], 1000) + + def test_duplicate_research_consumer_cannot_hide_execution_grade(self): + payload = yaml.safe_load(DEMAND.read_text()) + research = dict(payload["consumers"][0], consumer_id="a-research", consumer_grade="ALPHA") + payload["consumers"].insert(0, research) + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "demand.yaml" + path.write_text(yaml.safe_dump(payload)) + merged = ProductionDemandManifest.load_many([path]) + self.assertEqual(len(merged.demands), 4) + self.assertTrue(all(r.consumer_grade is ConsumerGrade.EXECUTION for r in merged.demands)) + books = [r["l2"] for r in self._compile(merged).acquisition_plan["bindings"] if "l2" in r] + self.assertTrue(all(r["materialized_snapshot_interval_ms"] == 1000 for r in books)) diff --git a/tests/test_phase1_instrument_domain.py b/tests/test_phase1_instrument_domain.py index 503ca588..e2bb46b6 100644 --- a/tests/test_phase1_instrument_domain.py +++ b/tests/test_phase1_instrument_domain.py @@ -116,6 +116,47 @@ def test_swap_uses_registry_inst_family_and_preserves_exact_inst_id(self): self.assertEqual(record.attributes["ctVal"], "0.01") self.assertEqual(record.attributes["ctMult"], "10") + def test_inverse_contract_value_currency_is_not_underlying_asset(self): + for kind, native, family in ( + ("SWAP", "BTC-USD-SWAP", "BTC-USD"), + ("FUTURES", "BTC-USD-261225", "BTC-USD"), + ): + with self.subTest(kind=kind): + record, alias = parse_public_instrument( + {"instType": kind, "instId": native, "instFamily": family, + "baseCcy": "", "quoteCcy": "", "ctType": "inverse", + "ctValCcy": "USD", "settleCcy": "BTC", "ctVal": "100", + "ctMult": "1", "tickSz": "0.1", "lotSz": "1", + "expTime": "1798185600000" if kind == "FUTURES" else "", + "state": "live"}, + metadata_revision=1, valid_from_ns=100, + ) + self.assertEqual(record.base_asset, "BTC") + self.assertEqual(record.quote_asset, "USD") + self.assertEqual(record.settlement_asset, "BTC") + self.assertEqual(record.attributes["ctValCcy"], "USD") + self.assertEqual(record.attributes["ctType"], "inverse") + self.assertEqual(record.contract_multiplier.as_decimal(), CanonicalDecimal.from_text("100").as_decimal()) + self.assertEqual(alias.native_symbol, native) + + def test_dated_and_option_quote_currency_comes_from_family_not_expiry_or_side(self): + for kind, native, extra in ( + ("FUTURES", "BTC-USDT-261225", {}), + ("OPTION", "BTC-USD-261225-50000-P", {"stk": "50000", "optType": "P"}), + ): + with self.subTest(kind=kind): + family = "BTC-USDT" if kind == "FUTURES" else "BTC-USD" + record, _ = parse_public_instrument( + {"instType": kind, "instId": native, "instFamily": family, + "ctValCcy": "BTC", "settleCcy": "USDT" if kind == "FUTURES" else "BTC", + "ctVal": "0.01", "tickSz": "0.1", "lotSz": "1", + "expTime": "1798185600000", **extra}, + metadata_revision=1, valid_from_ns=100, + ) + self.assertEqual(record.base_asset, "BTC") + self.assertEqual(record.quote_asset, family.split("-")[1]) + self.assertEqual(record.native_symbol, native) + def test_option_preserves_registry_identity_and_required_fields(self): record, _ = parse_public_instrument( { diff --git a/tests/test_phaseb_stable_release.py b/tests/test_phaseb_stable_release.py index b9acd716..5dcaf019 100644 --- a/tests/test_phaseb_stable_release.py +++ b/tests/test_phaseb_stable_release.py @@ -317,7 +317,7 @@ def test_service_openapi_and_sdk_versions_are_explicit(self): ) generated = build_openapi() self.assertEqual(package["project"]["version"], "2.0.0") - self.assertEqual(qdl_sdk.__version__, "2.0.5") + self.assertEqual(qdl_sdk.__version__, "2.0.6") self.assertEqual(generated["info"]["version"], "2.0.0") self.assertEqual(snapshot, generated) # ``reference:batch`` is a governed V2 public path in the checked-in diff --git a/tests/test_probe_exact_market_lineage.py b/tests/test_probe_exact_market_lineage.py new file mode 100644 index 00000000..dcb314e9 --- /dev/null +++ b/tests/test_probe_exact_market_lineage.py @@ -0,0 +1,81 @@ +"""Synthetic test-only fixtures; not provider acceptance evidence.""" +import json +from pathlib import Path +import tempfile +import unittest + +from scripts.probe_exact_market_lineage import Evidence, exact_view, trade_state, exact_join, newer_before_request + + +class ExactMarketLineageTests(unittest.TestCase): + def test_exact_view_preserves_flags_and_original_clocks_without_secrets(self): + row = {"quality": {"flags": ["INDEX_STALE"], "execution_eligible": False}, + "observed_at_ns": 123, "received_at_ns": 124, + "watermark_offset": 9, "contract": {"connection_generation": 7}, + "cursor": "secret", "payload": {"price": "42"}} + result = exact_view({"data": row, "cursor": "secret"}) + self.assertEqual(result["quality"], row["quality"]) + self.assertEqual(result["observed_at_ns"], 123) + self.assertEqual(result["contract"], row["contract"]) + self.assertNotIn("secret", json.dumps(result)) + self.assertNotIn("payload", result) + + def test_missing_raw_never_claims_quiet_from_session(self): + self.assertEqual(trade_state(raw_count=0, canonical_count=0, + query_matches=False, coverage_complete=True), + "NO_TRADE_IN_CAPTURE_WINDOW_NOT_SESSION_PROOF") + + def test_incomplete_capture_fails_closed(self): + self.assertEqual(trade_state(raw_count=12, canonical_count=12, + query_matches=True, coverage_complete=False), "UNASSESSED_INCOMPLETE_CAPTURE") + + def test_distinguish_pipeline_stages(self): + self.assertEqual(trade_state(raw_count=1, canonical_count=0, + query_matches=False, coverage_complete=True), "RAW_PRESENT_CANONICAL_MISSING_IN_WINDOW") + self.assertEqual(trade_state(raw_count=1, canonical_count=1, + query_matches=False, coverage_complete=True), "CANONICAL_PRESENT_QUERY_JOIN_MISSING") + self.assertEqual(trade_state(raw_count=1, canonical_count=1, + query_matches=True, coverage_complete=True), "SOURCE_CANONICAL_QUERY_OBSERVED") + + def test_join_and_positive_lag_require_original_identity_and_pre_request_capture(self): + import copy + event = dict(instrument_uid="test-only", source_event_time_ns="10", + received_at_ns="11", correlation_id="capture", source_session_id="s", + connection_generation="7", authority_revision="1") + selected = dict(event=event, feed="trade", partition=2, offset=4, captured_at_ns=20) + query = dict(uid="test-only", feed="TRADE", request_started_at_ns=30, + exact_view=dict(observed_at_ns=10, received_at_ns=11, watermark_offset=4, + contract=dict(correlation_id="capture"))) + newer = copy.deepcopy(selected) + newer.update(offset=5, captured_at_ns=29) + newer["event"]["received_at_ns"] = "12" + self.assertTrue(exact_join(query, selected)) + query["exact_view"]["watermark_offset"] = 8 + self.assertTrue(exact_join(query, selected)) + query["exact_view"]["watermark_offset"] = 3 + self.assertFalse(exact_join(query, selected)) + query["exact_view"]["watermark_offset"] = 4 + self.assertTrue(newer_before_request(query, selected, newer)) + newer["captured_at_ns"] = 31 + self.assertFalse(newer_before_request(query, selected, newer)) + newer["captured_at_ns"] = 29 + newer["event"]["connection_generation"] = "8" + self.assertFalse(newer_before_request(query, selected, newer)) + query["exact_view"]["contract"]["correlation_id"] = "different" + self.assertFalse(exact_join(query, selected)) + + def test_bounded_evidence_exclusive_and_no_overwrite(self): + with tempfile.TemporaryDirectory() as directory: + path = Path(directory) / "test-only.jsonl" + writer = Evidence(path, max_bytes=200) + writer.put("test", test_provenance=True) + with self.assertRaises(RuntimeError): + writer.put("test", value="x" * 200) + writer.close() + self.assertLessEqual(path.stat().st_size, 200) + with self.assertRaises(FileExistsError): + Evidence(path) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_probe_projector_stage_timing.py b/tests/test_probe_projector_stage_timing.py new file mode 100644 index 00000000..9b85abae --- /dev/null +++ b/tests/test_probe_projector_stage_timing.py @@ -0,0 +1,53 @@ +"""Synthetic unit evidence only, not production latency evidence.""" +import unittest + +from scripts.probe_projector_stage_timing import classify_stage, counter_rate, reference_metadata + + +class StageTimingTests(unittest.TestCase): + def test_window_end_does_not_shorten_admitted_rpc_timeout(self): + from scripts.probe_projector_stage_timing import admitted_query_timeout + self.assertEqual(admitted_query_timeout(0.021), 2.5) + self.assertEqual(admitted_query_timeout(180), 2.5) + self.assertIsNone(admitted_query_timeout(0)) + self.assertIsNone(admitted_query_timeout(-1)) + + def test_reference_preserves_problem_components_without_price_or_secrets(self): + import json + response = {"partial": True, "results": [{"status": "ERROR", "problem": {"code": "DATA_STALE"}}, + {"status": "OK", "data": {"received_at_ns": 5, "observations": [{"observed_at_ns": 4, + "fields": [{"name": "mark_price", "value": "SECRET_PRICE"}], + "labels": {"component_mark_received_at_ns": "3", "recency_mode": "COMPONENT_SESSION_LIVE", + "cursor": "SECRET_CURSOR"}}]}}]} + result = reference_metadata(response) + self.assertEqual(result["results"][0]["problem"], {"code": "DATA_STALE"}) + self.assertEqual(result["results"][1]["data"]["observations"][0]["labels"]["component_mark_received_at_ns"], "3") + self.assertNotIn("SECRET", json.dumps(result)) + + def test_missing_state_never_localizes_stage(self): + self.assertEqual(classify_stage(10, None, 30), + "CANONICAL_TO_REDIS_ONLY_STATE_UNAVAILABLE") + + def test_observer_order_is_not_negative_pipeline_latency(self): + self.assertEqual(classify_stage(20, 10, 30), + "OBSERVER_ORDER_INVERSION_NOT_NEGATIVE_LATENCY") + self.assertEqual(classify_stage(10, 30, 20), + "OBSERVER_ORDER_INVERSION_NOT_NEGATIVE_LATENCY") + + def test_ordered_observations_are_not_broker_commit_times(self): + self.assertEqual(classify_stage(10, 20, 30), + "OBSERVED_STAGE_INTERVALS_NOT_BROKER_COMMIT_TIMES") + + def test_counter_rates_reject_restarts_and_resets(self): + first = dict(at_ms=1000, started_ms=1, stage_a=dict(transactions=10)) + last = dict(at_ms=6000, started_ms=1, stage_a=dict(transactions=35)) + self.assertEqual(counter_rate(first, last, "transactions"), 5) + last["started_ms"] = 2 + self.assertIsNone(counter_rate(first, last, "transactions")) + last["started_ms"] = 1 + last["stage_a"]["transactions"] = 9 + self.assertIsNone(counter_rate(first, last, "transactions")) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_qdl_sdk_release.py b/tests/test_qdl_sdk_release.py index 6b9fe360..41585be2 100644 --- a/tests/test_qdl_sdk_release.py +++ b/tests/test_qdl_sdk_release.py @@ -33,7 +33,7 @@ def test_wheel_is_reproducible_self_contained_and_recorded(self): self.assertEqual({row[0] for row in rows}, names) release = json.loads(Path(first_result["manifest"]).read_text()) self.assertFalse(release["contains_service_internals"]) - self.assertEqual(release["version"], "2.0.5") + self.assertEqual(release["version"], "2.0.6") self.assertEqual(len(release["generated_contract_digest"]), 64) sbom = json.loads(Path(first_result["sbom"]).read_text()) self.assertEqual(sbom["bomFormat"], "CycloneDX") diff --git a/tests/test_qdl_sdk_stream_projection.py b/tests/test_qdl_sdk_stream_projection.py index e89f86e7..edfbfb74 100644 --- a/tests/test_qdl_sdk_stream_projection.py +++ b/tests/test_qdl_sdk_stream_projection.py @@ -187,6 +187,53 @@ def template(feed: Feed) -> MarketDataView: ) +class ExactQueryRefusalTests(unittest.TestCase): + def test_rejection_keeps_original_quality_not_later_view(self): + for feed in (Feed.TRADE, Feed.QUOTE, Feed.MARK_INDEX_PRICE, Feed.BOOK_DELTA): + with self.subTest(feed=feed): + view = template(feed) + view.quality.execution_eligible = False + view.quality.flags = ["component:index:stale", "generation:7"] + view.received_at_ns = NOW + 1000 + view.cursor = "signed-private-cursor-not-for-logs" + requirement = DataRequirement(instrument_uid=view.instrument_uid, + feed=feed, consumer_grade=Grade.EXECUTION, + source_policy_id="crypto_primary_v2", max_freshness_ms=2000) + with self.assertRaises(ContinuityError) as caught: + _validate_query_payload(requirement, + {"request_id": "exact", "data": view}, warmup=False) + evidence = caught.exception.diagnostics + self.assertEqual(caught.exception.code, "SOURCE_NON_AUTHORITATIVE") + self.assertEqual(evidence["quality"], view.quality.model_dump(mode="json")) + self.assertEqual(evidence["watermark_offset"], 10) + self.assertEqual(evidence["received_at_ns"], NOW + 1000) + self.assertNotIn("cursor", evidence) + self.assertNotIn("payload", evidence) + view.quality.flags.clear() + view.quality.execution_eligible = True + self.assertFalse(evidence["quality"]["execution_eligible"]) + self.assertEqual(len(evidence["quality"]["flags"]), 2) + + def test_stale_disconnected_and_gap_diagnostics_keep_error_codes(self): + cases = ( + ({"freshness_ms": 3001}, "DATA_STALE"), + ({"provider_session_state": "DISCONNECTED"}, "DATA_STALE"), + ({"gap_open": True}, "OPEN_SEQUENCE_GAP"), + ) + for changes, code in cases: + view = template(Feed.TRADE) + view.quality = view.quality.model_copy(update=changes) + requirement = DataRequirement(instrument_uid=view.instrument_uid, + feed=Feed.TRADE, consumer_grade=Grade.EXECUTION, + source_policy_id="crypto_primary_v2", max_freshness_ms=2000) + with self.assertRaises(ContinuityError) as caught: + _validate_query_payload(requirement, + {"request_id": "negative", "data": view}, warmup=False) + self.assertEqual(caught.exception.code, code) + self.assertEqual(caught.exception.diagnostics["quality"], + view.quality.model_dump(mode="json")) + + def envelope(feed: Feed) -> market_data_pb2.EventEnvelope: result = market_data_pb2.EventEnvelope( schema_name="qdl.marketdata.v2.EventEnvelope", diff --git a/tests/test_sandbox_extension.py b/tests/test_sandbox_extension.py new file mode 100644 index 00000000..da2bb1f2 --- /dev/null +++ b/tests/test_sandbox_extension.py @@ -0,0 +1,102 @@ +from copy import deepcopy +from pathlib import Path +import unittest + +import yaml + +from qdl.runtime.production_catalog import ProductionDemandManifest +from qdl.runtime.sandbox_extension import _digest, prepare_sandbox_extension +from scripts.compile_consumer_realms import compile_realm +from scripts.phase115_render_consumer_route_binding import binding_from_stable_release + +ROOT = Path(__file__).resolve().parents[1] + + +class SandboxExtensionTests(unittest.TestCase): + @classmethod + def setUpClass(cls): + manifest = compile_realm(yaml.safe_load( + (ROOT / "consumers/stable/trading-system-paper.yaml").read_text()), "sandbox") + binding = binding_from_stable_release( + ROOT / "config/v2/stable-v2-release-routing.yaml", ROOT, + "trading-system.paper.stable").canonical_mapping() + binding["consumer_id"] = manifest["metadata"]["id"] + binding["consumer_manifest_revision"] = 1 + for row in binding["products"]: + row["consumer_id"] = binding["consumer_id"] + binding.pop("binding_sha256") + binding["binding_sha256"] = _digest(binding) + cls.inputs = dict( + catalog=yaml.safe_load((ROOT / "config/v2/stable-source-bindings.yaml").read_text()), + acquisition=yaml.safe_load((ROOT / "config/v2/stable-acquisition-bindings.yaml").read_text()), + manifest=manifest, binding=binding, + demand=ProductionDemandManifest.load_many([ROOT / "config/v2/okx-inverse-sandbox-demand.yaml"]), + template_uid="fb26214c-7b9b-5961-95b2-55154755af0f", + okx_rows=[dict(instId="BTC-USD-SWAP", instType="SWAP", instFamily="BTC-USD", + ctType="inverse", ctValCcy="USD", settleCcy="BTC", ctVal="100", + ctMult="1", lotSz="0.1", tickSz="0.1", state="live")], + ) + + def test_preserves_old_scope_and_seals_new_generation(self): + source = deepcopy(self.inputs) + out = prepare_sandbox_extension(**source) + self.assertEqual(source, self.inputs) + self.assertEqual(len(out["binding"]["products"]), 64) + self.assertEqual(out["manifest"]["metadata"]["revision"], 2) + self.assertEqual(out["manifest"]["spec"]["quotas"], source["manifest"]["spec"]["quotas"]) + for product in source["binding"]["products"]: + self.assertIn(product, out["binding"]["products"]) + for row in source["catalog"]["bindings"]: + self.assertIn(row, out["catalog"]["bindings"]) + self.assertNotEqual(out["binding"]["universal_manifest_sha256"], source["binding"]["universal_manifest_sha256"]) + self.assertEqual(out["provenance"]["status"], "PREPARED_NOT_ACTIVATED_NOT_CERTIFIED") + + def test_rejects_paper_and_live_scope(self): + for realm in ("paper", "live"): + with self.subTest(realm=realm): + source = deepcopy(self.inputs) + source["manifest"]["metadata"]["environment"] = realm + with self.assertRaises(ValueError): + prepare_sandbox_extension(**source) + + def test_rejects_wrong_collateral_or_non_active_metadata(self): + for field, value in (("settleCcy", "USDT"), ("state", "suspend"), ("ctType", "linear")): + with self.subTest(field=field): + source = deepcopy(self.inputs) + source["okx_rows"][0][field] = value + with self.assertRaises(ValueError): + prepare_sandbox_extension(**source) + + def test_rejects_missing_template_and_quality_drift(self): + for change in ("missing", "freshness", "fallback", "identity"): + with self.subTest(change=change): + source = deepcopy(self.inputs) + if change == "missing": + source["template_uid"] = "a953e16e-7138-5562-b5e8-c337a44d0b65" + else: + row = next(r for r in source["binding"]["products"] + if r["instrument_uid"] == source["template_uid"] and r["feed"] == "QUOTE") + if change == "freshness": + row["max_freshness_ms"] += 1 + elif change == "fallback": + row["fallback"] = "V1" + else: + row["native_symbol"] = "BTC-USD-SWAP" + source["binding"].pop("binding_sha256") + source["binding"]["binding_sha256"] = _digest(source["binding"]) + with self.assertRaises(ValueError): + prepare_sandbox_extension(**source) + + def test_rejects_repeat_application(self): + out = prepare_sandbox_extension(**deepcopy(self.inputs)) + source = {**self.inputs, **{k: out[k] for k in ("catalog", "acquisition", "manifest", "binding")}} + with self.assertRaisesRegex(ValueError, "already exists"): + prepare_sandbox_extension(**source) + + def test_rejects_subject_and_manifest_revision_mismatch(self): + for field, value in (("subject", "spiffe://qdl/sandbox/other"), ("revision", 2)): + with self.subTest(field=field): + source = deepcopy(self.inputs) + source["manifest"]["metadata"][field] = value + with self.assertRaises(ValueError): + prepare_sandbox_extension(**source) diff --git a/upgrade/DATA_LAYER_V2_KAFKA_NATIVE_ARCHITECTURE_REVIEW.md b/upgrade/DATA_LAYER_V2_KAFKA_NATIVE_ARCHITECTURE_REVIEW.md index 67f48c2c..0e4491fe 100644 --- a/upgrade/DATA_LAYER_V2_KAFKA_NATIVE_ARCHITECTURE_REVIEW.md +++ b/upgrade/DATA_LAYER_V2_KAFKA_NATIVE_ARCHITECTURE_REVIEW.md @@ -2996,3 +2996,50 @@ fault tests và một final acceptance; không bỏ tests hoặc nới SLA cho k Trạng thái thực thi ban đầu của cả năm phase là NOT STARTED. Source ở thời điểm soạn: `74337e71ce8ce7611ca424d33f7a0ab29535733a`; facts runtime trong mục 0-17 là lịch sử, không có health/latency certification mới từ việc soạn guide. + +### Post-release consumer realms (owner approved 2026-09-27) + +`paper`, `sandbox`, and `live` are workload authorization realms, not different +copies of public mainnet market data. The existing `paper` cache/product/cursor +namespace is retained as the physical market-data realm for compatibility; +it does not certify an execution account mode. Every additional identity has a +separate TLS client identity and JWT signing key. Explicit key-to-realm and +key-to-subject maps must cover the entire keyring. A token cannot select its own +realm independently of its pinned key. Consumer ID, manifest revision, venue/feed +entitlements, purposes, quotas and consumer-bound signed cursor remain enforced. +Old single-realm configuration keeps its existing rejection behavior. + +No extra network authorization hop or service is introduced. Query loads a sealed +configuration on startup; Rust Stream atomically reloads its authority and revokes +an admitted stream if its key realm changes. Configuration is public-key material +only; all private material stays in protected runtime directories outside Git. +`compile_consumer_realms.py` maps only the three approved TS/Binance-alpha/OKX-alpha +scopes. TS PAPER_ONLY becomes ALLOWED for its explicitly approved sandbox/live +read identity. Alpha FORBIDDEN direct execution dependency stays FORBIDDEN: alpha +uses data to construct intent, while TS Risk remains the admission authority. +Read activation never enables orders, changes broker mode, or starts an alpha. + +#### Three-realm operation and consumer handoff + +Adding an approved reader is configuration materialization, not a new service or +per-symbol image. Compile the versioned manifest with compile_consumer_realms.py, +provision a separate TLS identity and JWT key, and include the public key/subject/ +realm and manifest in the reader packet. Current Query startup requires a bounded +reader reload/recreate for configuration changes; do not promise zero reload. +No code/image rebuild is needed merely for another binding within supported +contracts. New feed implementations still require source tests and release. + +Consumer identity, sealed binding, manifest revision and credential must change +together. TS settings use DATA_LAYER_V2_JWT_KEY_ID, JWT_SUBJECT, JWT_ENVIRONMENT, +JWT_MANIFEST_REVISION and JWT_PRIVATE_KEY_FILE (all prefixed DATA_LAYER_V2_), plus +TLS_CA_FILE/TLS_CERT_FILE/TLS_KEY_FILE and CONSUMER_BINDING_FILE. Use the real +configuration field names in each consumer; do not rename a paper key to live. +Keep separate cursor/audit paths by consumer ID and preserve source-policy gates. +The runtime packet under consumer-realms contains private credentials outside Git; +mount only the selected consumer identity, never the whole identity directory. + +The realm labels workload permission only: sandbox/live still read the same +public mainnet market products. They do not select demo exchange books, change +account mode, establish broker execution parity, or permit order submission. +Current TS process remains on its existing paper read identity until a consumer +config handoff; the additional server mappings are active and tested no-order. diff --git a/upgrade/evidence/releases/v2.2.1/RELEASE_NOTES.md b/upgrade/evidence/releases/v2.2.1/RELEASE_NOTES.md new file mode 100644 index 00000000..da9170de --- /dev/null +++ b/upgrade/evidence/releases/v2.2.1/RELEASE_NOTES.md @@ -0,0 +1,26 @@ +# Quant Data Layer v2.2.1 + +## Scope +Kafka-native execution-readiness repair, retaining V2 primary and policy-limited V1 fallback. No broker orders, TS/alpha rollout, Kafka offset reset, cache flush, or history deletion. + +- Rust core flushes against an absolute batch deadline; arrivals cannot extend the deadline indefinitely. +- Existing transactional quarantine carries bounded, generation-fenced L2 resnapshot feedback to the existing venue ingestor. BOOK recovery does not restart TRADE/QUOTE lanes. +- Fair socket polling and coalesced control checkpoints prevent feedback pressure from starving live input. Genuine gaps still fail closed. +- Includes additive OKX inverse sandbox binding and previously integrated realm/diagnostic fixes; not a grant of new alpha execution rights. +- SDK2.0.6 gives existing typed refusal diagnostics a distinct reproducible artifact identity. API schema remains2.0.0; existing consumer pins are unchanged. + +## Evidence +64 entitlement products x two Query replicas:128 typed views,126 usable, two OKX TRADE refusals matched to latest public trade ID/time at observation. All MARK/QUOTE/BOOK/BAR preflight reads usable. Request-to-validated-usable observed maxima are below50ms in these small groups; they are not p99. + +One300s production observation: paper60/60 and sandbox22/22 READY on11 distinct published heartbeats each. Four OKX transport disconnects produced12paper/8sandbox session refusals followed by recovery. This is fail-closed recovery, not continuous availability. Paper cache was measured176s after correcting the observer Redis selection; sandbox300s. Polling ages include up to1s observation delay and are not exact Redis commit latency. + +Isolated authentic RF3 replay at unchanged core caps:330000 records across6partitions, bounded4k/s and5k/s windows, all final offsets caught up. Not indefinite5000/s per consumer or unlimited fanout. Whole runtime observation averaged4.70cores, sampled peak5.12, no cap increase. + +## Deployment And Rollback +Three Rust cores and two native ingestors use immutable658a9570c5fc...54023. Two Query readers additionally use b40b49c361c7...4a65 (sourcee366f75) with PyJWT2.15.1/urllib32.8.0 security fixes;32packagedauth/SDKtests pass. Query rollback is0ed57643...eb959 with identical mounts/config. Stream, KN market projectors, BAR edge, V1 and TS images stay unchanged. Certificate contains full per-role digests. Native rollback is coref2040ac9...ca79 and ingestor7fe34806...f69f with original per-role config;7fe remains active in KN market projectors. No reset needed. Historical authority handoff digest is provenance, not a false claim of current executable identity. + +## Limits +Binance3d and DNSE/VN V2 are excluded by owner. Quiet last trades can legitimately fail execution eligibility; no stale price is made fresh. Unchanged warmup/batch/reference/diagnostics capacity inherits dated v2.2.0 evidence, not a new full-catalogue benchmark. TS E lifecycle/order certification remains separate. See certificate and endpoint report for denominators, sampling limitations and observed reconnects. + +## Query Security Readback +After the separately approved twoQuery roll,128/128 typed views returned,125 usable and3 strict TRADE-age refusals; no auth/manifest failure. BinanceSOL andOKXBNB matched provider quiet-last-trade evidence. OKXSOL had a newer trade228ms before the request, present in raw/canonical; this is not called quiet. Canonical CreateTime is not commit visibility. Ten follow-up reads yielded7usable/3age-refused then a newer trade, retained separately rather than replacing the128-read denominator. No additional C2 or threshold relaxation. Latest per-feed observed request maximum8-12usable samples is in the endpoint report; no fabricated p99 or exact Redis commit latency. diff --git a/upgrade/evidence/releases/v2.2.1/certificate.json b/upgrade/evidence/releases/v2.2.1/certificate.json new file mode 100644 index 00000000..08e00846 --- /dev/null +++ b/upgrade/evidence/releases/v2.2.1/certificate.json @@ -0,0 +1,435 @@ +{ + "affected_runtime_delta_status": "PASS", + "ci": { + "jobs": [ + "contract-tests", + "unit-tests", + "sdk-python310", + "kn-native-integration" + ], + "release_metadata_head_requires_checks": true, + "run_id": 36760348660, + "status": "PASS", + "tested_source_sha": "e366f752914db8d1d43cb03a970b0278635ba88d" + }, + "classification": "Bounded affected Data Layer execution-readiness repair; production no-order consumer acceptance, not broker execution certification", + "cleanup": { + "legacy-retirement-result.json": { + "build_cache": "Shared BuildKit cache not pruned; no per-task ownership identified.", + "disk_note": "Concurrent host changes mean difference is not claimed as exact reclaimed bytes.", + "disk_used_after": 235294134272, + "disk_used_before": 235353243648, + "removed_containers": [ + "query_v2_1", + "query_v2_2", + "stream_v2_active", + "stream_v2_passive", + "projector_v2", + "projector_v2_2", + "projector_v2_3", + "projector_v2_4", + "projector_v2_5", + "projector_v2_6" + ], + "removed_images": [ + "sha256:37d7f5182ea170f5cc35967aabba0d7f0c1b4b934c12aa25a51ac79664b181f9", + "sha256:56d331db87d9c5ab8bfa8690491d7e69dfbb1661d824073d32bf3750cebac5f1", + "sha256:dd065fdf8c439ce9034bca47817e1e88ce0a69bbdbd24a2d1f6b0d2f6f693951" + ], + "retained_images": [], + "running_roles_changed": [], + "state": "EXACT_LEGACY_RETIREMENT_COMPLETE", + "timestamp_ns": 1790792967199966546, + "volumes_removed": false + }, + "query-predecessor-cleanup.json": { + "disk_used_after": 219302334464, + "disk_used_before": 219302060032, + "reason": "Superseded unreferenced Query predecessor; exact active Query b40/rollback0ed now frozen by appliedpacket.", + "removed_image": "sha256:3dd54bfd8b9de0b143d317798ab9c63cebc502d01fe3cf6f513032d6484b5b11", + "state_or_container_mutation": false + }, + "task-build-cache-cleanup.json": { + "disk_used_after": 232285339648, + "disk_used_before": 235431378944, + "logical_bytes_at_privileged_retry": 3123045441, + "note": "Unprivileged attempt stopped on root-owned compiler file. Exact task-only cache removed after checking no container mounts; no shared BuildKit prune.", + "removed_path": "execution-readiness-20260930/target", + "runtime_mutation": false + }, + "unused-image-cleanup.json": { + "disk_used_after": 232098627584, + "disk_used_before": 232370102272, + "removed_images": [ + "sha256:ba41b1f279d42998ecb6abf6cae32e55ef6d275e59c37498077b774edb7f01b4", + "sha256:579d578e30814192ae5d20c4f29b653b5c5bc173cfaad73f86e9c192dcb6aa6c", + "sha256:ed1ef88bb400bc262ffdec40589e210657757dc7f963c2f727cb86d4b8c4378a", + "sha256:389753b37c4f3935193acc739687c4a18c5b307f3ea6a427153f5689d31c9762", + "sha256:eec6388421845ef570cb3878145d0c5e0398fc44cc2a662805298295cfb0976a" + ], + "retained_query_predecessor": "sha256:3dd54bfd8b9de0b143d317798ab9c63cebc502d01fe3cf6f513032d6484b5b11", + "retention_reason": "Conservative existing Query predecessor retention; current repair does not change Query. Do not remove a potential named reader rollback while finishing release provenance.", + "running_roles_changed": [], + "timestamp_ns": 1790793275717377902, + "volume_or_shared_cache_changes": false + } + }, + "component_tree_identity": { + "Cargo.lock": "8a193f6da07f4742d03043beff2a34f1062ccfff", + "Dockerfile.qdl-rust-runtime": "96e1a310e40903792d68d9252285b99f18c7460a", + "rust": "3f8bc79be7212377442218f231a95bc4e6d0bb73" + }, + "date_utc": "2026-09-30", + "endpoint_report_sha256": "a7fa7ff9883a1a91f59315c3a5d1e3b08650e23c78a448f98fe92e16ed75b63b", + "evidence_sha256": { + "evidence/batch-result.json": "442a0d251e41abe3dc2bf2bbff6235cba80e2149df84a10f18f66386b7b1432c", + "evidence/candidate-image-attestation.json": "c9ef6963ab78717ca0f0fa10fef78ea8cc04a4715822c4967006ea74aa2628b1", + "evidence/consumer-final-300s/receipt.json": "a87ce6c156915b453291566ab95ed5a96f45ed7a4daf8198198210236d467499", + "evidence/consumer-final-300s/reviewed-summary.json": "2222f0945f9da6bb56e6c752e344aeed3bec0d5a9c302b2e3bad4ee37b5d898c", + "evidence/l2-source-result.json": "09f7b61120342ef6bcee0c333e6790922519e4602ee28eb5f73126ad96e2f2a1", + "evidence/legacy-retirement-result.json": "9ee640812502bdfa576b2ebf772b93e65237c7ba36a55216155781d53447cade", + "evidence/query-predecessor-cleanup.json": "787fd2d62d37371d077c7b8e515dad0314cc4a10011f4105d2e98583396eabb7", + "evidence/query-security-preflight.jsonl": "0e9e319a88bc735f02ab7944ee200e9b74b1ddcbebc26855e696a2e2a89b7454", + "evidence/query-security-reviewed.json": "74d4d8687b1559bf909dd3aa0239128d35f727b55e68f9950197d16ad2f9e1ea", + "evidence/query-security-sol-canonical.json": "9fb63a28225dec4960d89081f81cb68518cd47ea00fa558225c005840704254c", + "evidence/query-security-sol-followup.jsonl": "b53143b563572cd41849a9e5ed1cafe7b5ba33cc50eca44486add228f224fd09", + "evidence/query-security-sol-lineage-v2.jsonl": "7a83084af9a58699d1142ba7081ee38115dcf6eeb2debf6a67b6df19b61dc76e", + "evidence/query-security-trade-attribution.json": "af7444c5deef7fbd6b28368d17305b132c1ca702a3efb65aaa7d526ba27bd40a", + "evidence/rollout-state.json": "4d77647f4e54009dc855fda665434eec9297dacf61b473ed791299ce9dd8101e", + "evidence/task-build-cache-cleanup.json": "0183a57a4a984ffd97809c7925196c3a3a2c85f8c25242ab21b7b6ba39d153a7", + "evidence/trade-rejection-provider-attribution.json": "b61671093444985c054a21309a7651a6d980a17b1526321d19ef8a3b756bbc56", + "evidence/typed-preflight-post-roll.jsonl": "e8187f085f12c073570b9d01868641a8239b4d90e74c5695d5f64464aeeebf68", + "owner-move-r1/receipt.json": "09a265f94fc38543a13afd49461643ac6e25909f74b5dd088d47d239fdac2326", + "throughput-rf3-matched-r1/receipt.json": "0c1165c87a79d6d53c6f8d5f23d4abf8094bb28b075faac53d505f6ebd8f9d57" + }, + "gates": { + "R1": "PASS exact aliases/role digests and protected start times", + "R2": "PASS absolute batch deadline regressions; existing transaction ordering retained", + "R3": "PASS authentic four-book cold/rebalance recovery; real Kafka committed/resume/cancel; hostile hint pressure bounded", + "R4": "PASS bounded RF3 production-cap replay4k/s30s +5k/s30s +drain,330000 offsets accounted; not indefinite sustained/fanout claim", + "R5": "PASS128 typed reads:126eligible,2valid last-trade refusals matched provider history; all22MARK reads usable", + "R6": "PASS bounded fail-closed recovery; paper60/sandbox22 READY on11distinct heartbeats each; observed disconnects retained", + "R7": "PASS packaged binary attestation, candidate/rollback Compose validation, actual rollout. No live downgrade-return in this repair; unchanged state schema and prior KN paired rollback retained." + }, + "inherited": { + "certificate_sha256": "262f0123901501f7b8cdea669684b350c5dac96c97c8002551143a48deeef59d", + "current_post_v220_readers": "fba8863 realm/diagnostic corrections and inverse entitlement tracked by this release-set map and R5; not old reader binary", + "release": "v2.2.0", + "scope": "Unchanged public schemas, KN read/storage architecture, history/reference catalogue and historical workload envelope only; does not override failures or describe current binary as v2.2.0." + }, + "limits": [ + "No p99 for snapshot groups with fewer than100 samples.", + "Paper cache observed176s after Redis selection correction; heartbeat300s.", + "No 24/7 uptime, arbitrary50-alpha or indefinite5000/s certification.", + "Two OKX TRADE refusals attributed by exact public history ID/time; no stale price accepted.", + "Binance3d and VN/DNSE V2 explicitly excluded by owner.", + "No exact per-event Redis commit timestamp distribution from polling.", + "Current native socket disconnect reason not logged; four recovery incidents not mislabelled zero interruption.", + "Inherited ignored Redis50000ns test timing race is not counted PASS; provider admission unchanged.", + "Sandbox64entitlements but22active demand, not64active subscriptions.", + "Querysecurityreadback125/128usable:2quietlasttrade,1newtradepropagation. CanonicalCreateTime is not commitvisibility. Follow-up7/10usable retained separately; not always-eligible execution pricing." + ], + "publication_requires": "feature->dev CI->main ancestry and immutable release workflow; this file alone is not publication", + "recovery": { + "market_cache_flush": false, + "new_topics": false, + "owner_move_ms_including_5000ms_stability": 7830.931890988722, + "production_offsets_reset": false, + "trade_or_order_mutation": false + }, + "release": "v2.2.1", + "rollback": { + "cores": "sha256:f2040ac9e9380d067e8bcb4c965b1d7ea39c181c08993b7dba1b096337f7ca79", + "ingestors": "sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f", + "packet": "v2.2.1-0e00c81-recovery/rollback.compose.json", + "preserve": "original per-role config/TLS/state/group/offsets; disable feedback clients before removing only added feedback ACLs", + "query": "sha256:0ed57643ae40087fe7d35ec04a778b2c8847d520f02aa0394a87b71dcdbeb959", + "query_packet": "v2.2.1-e366f75-query-security/rollback.compose.json" + }, + "runtime": [ + { + "before_image": "sha256:3b4ee780f2d41625feb2ee3337338e1f89717942822685e6caaf08206c309d72", + "before_started_at": "2026-09-30T14:39:42.813261759Z", + "image": "sha256:3b4ee780f2d41625feb2ee3337338e1f89717942822685e6caaf08206c309d72", + "name": "/ts-p183-p183_market_data-1", + "oom": false, + "restarts": 0, + "role": "p183_market_data", + "started_at": "2026-09-30T14:39:42.813261759Z", + "status": "running" + }, + { + "before_image": "sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f", + "before_started_at": "2026-09-30T05:00:36.758628771Z", + "image": "sha256:658a9570c5fc23f4906413aa2460f4d82e21772cc63c9ed900363d4d13d54023", + "name": "/qdl_v2_stable_candidate-ingestor_okx_swap-1", + "oom": false, + "restarts": 0, + "role": "ingestor_okx_swap", + "started_at": "2026-09-30T17:45:03.984169624Z", + "status": "running" + }, + { + "before_image": "sha256:f2040ac9e9380d067e8bcb4c965b1d7ea39c181c08993b7dba1b096337f7ca79", + "before_started_at": "2026-09-30T04:56:09.10107135Z", + "image": "sha256:658a9570c5fc23f4906413aa2460f4d82e21772cc63c9ed900363d4d13d54023", + "name": "/qdl_v2_stable_candidate-rust_core_3-1", + "oom": false, + "restarts": 0, + "role": "rust_core_3", + "started_at": "2026-09-30T17:45:38.534537186Z", + "status": "running" + }, + { + "before_image": "sha256:f2040ac9e9380d067e8bcb4c965b1d7ea39c181c08993b7dba1b096337f7ca79", + "before_started_at": "2026-09-30T04:55:59.964837196Z", + "image": "sha256:658a9570c5fc23f4906413aa2460f4d82e21772cc63c9ed900363d4d13d54023", + "name": "/qdl_v2_stable_candidate-rust_core_2-1", + "oom": false, + "restarts": 0, + "role": "rust_core_2", + "started_at": "2026-09-30T17:46:23.058376424Z", + "status": "running" + }, + { + "before_image": "sha256:f2040ac9e9380d067e8bcb4c965b1d7ea39c181c08993b7dba1b096337f7ca79", + "before_started_at": "2026-09-30T04:55:50.317257108Z", + "image": "sha256:658a9570c5fc23f4906413aa2460f4d82e21772cc63c9ed900363d4d13d54023", + "name": "/qdl_v2_stable_candidate-rust_core-1", + "oom": false, + "restarts": 0, + "role": "rust_core", + "started_at": "2026-09-30T17:46:56.312111099Z", + "status": "running" + }, + { + "before_image": "sha256:65918a656dcb92ab92b687a92195caa5642a185dfdc60c4bd3ebbc702609c09a", + "before_started_at": "2026-09-30T02:41:30.438040199Z", + "image": "sha256:65918a656dcb92ab92b687a92195caa5642a185dfdc60c4bd3ebbc702609c09a", + "name": "/qdl_v2_stable_candidate-stream_kn_2-1", + "oom": false, + "restarts": 0, + "role": "stream_kn_2", + "started_at": "2026-09-30T02:41:30.438040199Z", + "status": "running" + }, + { + "before_image": "sha256:65918a656dcb92ab92b687a92195caa5642a185dfdc60c4bd3ebbc702609c09a", + "before_started_at": "2026-09-30T02:41:27.075445103Z", + "image": "sha256:65918a656dcb92ab92b687a92195caa5642a185dfdc60c4bd3ebbc702609c09a", + "name": "/qdl_v2_stable_candidate-stream_kn_1-1", + "oom": false, + "restarts": 0, + "role": "stream_kn_1", + "started_at": "2026-09-30T02:41:27.075445103Z", + "status": "running" + }, + { + "before_image": "sha256:0ed57643ae40087fe7d35ec04a778b2c8847d520f02aa0394a87b71dcdbeb959", + "before_started_at": "2026-09-30T02:39:32.257181634Z", + "image": "sha256:b40b49c361c7474691ef2d68eba8bcb4a22045cb2d010e2c73a8c0c68cc04a65", + "name": "/qdl_v2_stable_candidate-query_kn_2-1", + "oom": false, + "restarts": 0, + "role": "query_kn_2", + "started_at": "2026-09-30T23:29:17.340927437Z", + "status": "running" + }, + { + "before_image": "sha256:0ed57643ae40087fe7d35ec04a778b2c8847d520f02aa0394a87b71dcdbeb959", + "before_started_at": "2026-09-30T02:38:12.148941229Z", + "image": "sha256:b40b49c361c7474691ef2d68eba8bcb4a22045cb2d010e2c73a8c0c68cc04a65", + "name": "/qdl_v2_stable_candidate-query_kn_1-1", + "oom": false, + "restarts": 0, + "role": "query_kn_1", + "started_at": "2026-09-30T23:27:49.661572186Z", + "status": "running" + }, + { + "before_image": "sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f", + "before_started_at": "2026-09-30T02:37:27.760443157Z", + "image": "sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f", + "name": "/qdl_v2_stable_candidate-market_projector_2-1", + "oom": false, + "restarts": 0, + "role": "market_projector_2", + "started_at": "2026-09-30T02:37:27.760443157Z", + "status": "running" + }, + { + "before_image": "sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f", + "before_started_at": "2026-09-30T02:36:59.050411241Z", + "image": "sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f", + "name": "/qdl_v2_stable_candidate-market_projector_1-1", + "oom": false, + "restarts": 0, + "role": "market_projector_1", + "started_at": "2026-09-30T02:36:59.050411241Z", + "status": "running" + }, + { + "before_image": "sha256:1037c56a48d65156bb64136db78325678e4940b51bc12dbee050162cf9549217", + "before_started_at": "2026-09-30T12:53:58.890843635Z", + "image": "sha256:1037c56a48d65156bb64136db78325678e4940b51bc12dbee050162cf9549217", + "name": "/market_data_service", + "oom": false, + "restarts": 1, + "role": "market_data", + "started_at": "2026-09-30T12:53:58.890843635Z", + "status": "running" + }, + { + "before_image": "sha256:f7351c3bda080a6d1266d49480dc65d7e590375f178252f858f87a04b1a4d685", + "before_started_at": "2026-09-27T07:57:13.649366033Z", + "image": "sha256:f7351c3bda080a6d1266d49480dc65d7e590375f178252f858f87a04b1a4d685", + "name": "/qdl_v2_stable_candidate-binance_bar_edge-1", + "oom": false, + "restarts": 0, + "role": "binance_bar_edge", + "started_at": "2026-09-27T07:57:13.649366033Z", + "status": "running" + }, + { + "before_image": "sha256:7fe348060734e4f51824b02faed7020465bb8dc754ad5299cb88befba7f9f69f", + "before_started_at": "2026-09-27T05:38:27.027889454Z", + "image": "sha256:658a9570c5fc23f4906413aa2460f4d82e21772cc63c9ed900363d4d13d54023", + "name": "/qdl_v2_stable_candidate-ingestor_binance_usdm-1", + "oom": false, + "restarts": 0, + "role": "ingestor_binance_usdm", + "started_at": "2026-09-30T17:44:26.540267501Z", + "status": "running" + }, + { + "before_image": "sha256:dfa18828cbc07b3ae6a95ec7343f6c214fdee2d836197b4be8e9904420762cd8", + "before_started_at": "2026-09-27T05:29:58.000340326Z", + "image": "sha256:dfa18828cbc07b3ae6a95ec7343f6c214fdee2d836197b4be8e9904420762cd8", + "name": "/qdl_v2_stable_candidate-market_cache-1", + "oom": false, + "restarts": 0, + "role": "market_cache", + "started_at": "2026-09-27T05:29:58.000340326Z", + "status": "running" + }, + { + "before_image": "sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9", + "before_started_at": "2026-09-17T06:46:05.30391214Z", + "image": "sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9", + "name": "/qdl_v2_stable_candidate-kafka3-1", + "oom": false, + "restarts": 0, + "role": "kafka3", + "started_at": "2026-09-17T06:46:05.30391214Z", + "status": "running" + }, + { + "before_image": "sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9", + "before_started_at": "2026-09-17T09:54:28.712963177Z", + "image": "sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9", + "name": "/qdl_v2_stable_candidate-kafka2-1", + "oom": false, + "restarts": 1, + "role": "kafka2", + "started_at": "2026-09-17T09:54:28.712963177Z", + "status": "running" + }, + { + "before_image": "sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9", + "before_started_at": "2026-09-17T09:51:22.711409533Z", + "image": "sha256:9516fb7634bad307d17c33b589fde9023003b0cb761374f500002b980a3149b9", + "name": "/qdl_v2_stable_candidate-kafka1-1", + "oom": false, + "restarts": 5, + "role": "kafka1", + "started_at": "2026-09-17T09:51:22.711409533Z", + "status": "running" + }, + { + "before_image": "sha256:dfa18828cbc07b3ae6a95ec7343f6c214fdee2d836197b4be8e9904420762cd8", + "before_started_at": "2026-09-16T06:38:55.588316335Z", + "image": "sha256:dfa18828cbc07b3ae6a95ec7343f6c214fdee2d836197b4be8e9904420762cd8", + "name": "/qdl_v2_stable_candidate-stable_redis-1", + "oom": false, + "restarts": 0, + "role": "stable_redis", + "started_at": "2026-09-16T06:38:55.588316335Z", + "status": "running" + }, + { + "before_image": "sha256:dbfb57844977513ae7ec0a4782e04da0213028a789753c6b991f26043b615d65", + "before_started_at": "2026-09-15T04:31:56.058490893Z", + "image": "sha256:dbfb57844977513ae7ec0a4782e04da0213028a789753c6b991f26043b615d65", + "name": "/data_layer_service", + "oom": false, + "restarts": 0, + "role": "data_layer", + "started_at": "2026-09-15T04:31:56.058490893Z", + "status": "running" + } + ], + "schema": "qdl.release.affected-delta-certificate.v1", + "sdk": { + "contains_service_internals": false, + "dependencies": [ + "grpcio>=1.70.0,<2.0.0", + "httpx>=0.28.0,<1.0.0", + "protobuf>=6.31.1,<7.0.0", + "pydantic>=2.0.0,<3.0.0", + "PyJWT[crypto]>=2.15.1,<3.0.0" + ], + "generated_contract_digest": "2a25a601ae80b2ee4b36e6d1cb8ca44a13b8a037bf69a96f45659d1782a3a624", + "name": "qdl-sdk", + "python_requires": ">=3.10", + "reproducible_timestamp": "2020-01-01T00:00:00Z", + "schema": "qdl.sdk.release.v1", + "sdk_source_digest": "9930844e1522e138465900601d3f382631af22d8567623608e5f8dc5fee3790c", + "version": "2.0.6", + "wheel": "qdl_sdk-2.0.6-py3-none-any.whl", + "wheel_sha256": "d44fbf559c0c7eb52abf2a40091bcecc9e5ea025fed9c9b0bbbc6db6329395ad" + }, + "security_packaging": { + "candidate_image": "sha256:b40b49c361c7474691ef2d68eba8bcb4a22045cb2d010e2c73a8c0c68cc04a65", + "candidate_source": "e366f752914db8d1d43cb03a970b0278635ba88d", + "ci": "36760348660 all4jobsPASS on e366f752914db8d1d43cb03a970b0278635ba88d", + "packaged_auth_sdk_tests": { + "fail": 0, + "pass": 32 + }, + "query_rollout": "Owner approved; onlyquery_kn_1/2 recreated; originalconfig/TLS/state retained; bothhealthy,128typedreadback withstrictquality refusals.", + "readback": { + "attempted": 128, + "auth_manifest_errors": 0, + "refusal_analysis": "query-security-reviewed.json", + "refusals": 3, + "usable": 125 + }, + "runtime_deployed": true, + "scope": "No assertion that unchanged Python/V1/TS images have these dependency fixes.", + "source_floor": { + "PyJWT": "2.15.1", + "urllib3": "2.8.0" + }, + "targeted_auth_sdk_tests": { + "fail": 0, + "pass": 32 + } + }, + "source_components": { + "bar_edge": "4fe7e92", + "native_market_projector": "02cd827", + "python_query": "e366f752914db8d1d43cb03a970b0278635ba88d", + "query_stream": "fba8863", + "rust_behavior_commit": "fbbe34e", + "rust_image_source": "0e00c817289ce32ea767cea002c859814349fc72", + "sdk_packaging_commit": "e366f752914db8d1d43cb03a970b0278635ba88d" + }, + "status": "PASS", + "tests": { + "sdk_packaging": { + "fail": 0, + "pass": 2, + "runtime_change": false + } + } +} diff --git a/upgrade/evidence/releases/v2.2.1/endpoint-report.json b/upgrade/evidence/releases/v2.2.1/endpoint-report.json new file mode 100644 index 00000000..1cee5a36 --- /dev/null +++ b/upgrade/evidence/releases/v2.2.1/endpoint-report.json @@ -0,0 +1,6874 @@ +{ + "aggregate": [ + { + "attempted": 10, + "feed": "BAR", + "max_ms": 20.689978962764144, + "median_ms": 14.289901591837406, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "BOOK_DELTA", + "max_ms": 38.902620086446404, + "median_ms": 12.01496203429997, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "BOOK_SNAPSHOT", + "max_ms": 45.98168400116265, + "median_ms": 24.57058907020837, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "MARK_INDEX_PRICE", + "max_ms": 26.64185082539916, + "median_ms": 11.222365545108914, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "QUOTE", + "max_ms": 25.13775695115328, + "median_ms": 8.996241143904626, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "TRADE", + "max_ms": 15.094537055119872, + "median_ms": 11.27187698148191, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "BAR", + "max_ms": 37.825681967660785, + "median_ms": 16.023776959627867, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "BOOK_DELTA", + "max_ms": 25.4819979891181, + "median_ms": 11.148278019391, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "BOOK_SNAPSHOT", + "max_ms": 49.03210490010679, + "median_ms": 29.848029487766325, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "MARK_INDEX_PRICE", + "max_ms": 18.841461976990104, + "median_ms": 11.130759958177805, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "QUOTE", + "max_ms": 29.4035489205271, + "median_ms": 9.26159752998501, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 10, + "feed": "TRADE", + "max_ms": 15.116487862542272, + "median_ms": 8.699479978531599, + "p99_ms": null, + "rejected": 2, + "usable": 8, + "venue": "OKX" + } + ], + "binding_replica_reads": [ + { + "call_to_usable_ms": 9.432550054043531, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1006, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 1016, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790743027694151, + "usable": true, + "venue": "BINANCE" + }, + { + "call_until_rejection_ms": 13.29477084800601, + "error_detail": "V2 view is not authoritative/execution eligible", + "error_type": "DataLayerClientError", + "feed": "TRADE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "STALE", + "execution_eligible": false, + "flags": [ + "LAST_EVENT_STALE" + ], + "freshness_ms": 4872, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 594, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790743269656636, + "usable": false, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.632400073111057, + "feed": "TRADE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1175, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 778, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790743455875299, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 32.61132794432342, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 2047, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790743640593288, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 11.053350986912847, + "feed": "BOOK_DELTA", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 925, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 596, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790743824818437, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.592957984656096, + "feed": "BAR", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": "1m", + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 23997, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790743986442863, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.14880896359682, + "feed": "BOOK_DELTA", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 970, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 938, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790744150562125, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 14.340166002511978, + "feed": "QUOTE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 826, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 156, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790744327388992, + "usable": true, + "venue": "OKX" + }, + { + "call_until_rejection_ms": 9.0782071929425, + "error_detail": "V2 view is not authoritative/execution eligible", + "error_type": "DataLayerClientError", + "feed": "TRADE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "STALE", + "execution_eligible": false, + "flags": [ + "LAST_EVENT_STALE" + ], + "freshness_ms": 3203, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 781, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790744492385799, + "usable": false, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.794220983982086, + "feed": "BAR", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": "1m", + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 24700, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790744669108568, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 39.10869196988642, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1667, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790744853625628, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.04276897199452, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 971, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 983, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790745044178276, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.248552145436406, + "feed": "BOOK_DELTA", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1023, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 990, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790745220104643, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.47613800317049, + "feed": "BAR", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": "1m", + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 25390, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790745379844391, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 7.445418043062091, + "feed": "TRADE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 752, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 831, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790745543839467, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 31.195443822070956, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 855, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790745719246171, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 6.966839078813791, + "feed": "TRADE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1080, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 91, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790745901753183, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.608781965449452, + "feed": "QUOTE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 929, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 897, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790746077391322, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.437627090141177, + "feed": "TRADE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1397, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 472, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790746236511521, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.521977113559842, + "feed": "BOOK_DELTA", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1000, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 71, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790746397779565, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 22.46361505240202, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1700, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790746556816839, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 17.52235391177237, + "feed": "BAR", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": "1m", + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 26768, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790746730656744, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.47055595740676, + "feed": "BAR", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": "1m", + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 26933, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790746923033337, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 14.387136092409492, + "feed": "QUOTE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 899, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 791, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790747086082253, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 7.449529133737087, + "feed": "QUOTE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 1156, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 80, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790747251003148, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.592007078230381, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1338, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 1344, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790747425586630, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 11.758751003071666, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1361, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 477, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790747586868085, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.148728059604764, + "feed": "TRADE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 791, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 873, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790747749151703, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 15.685277059674263, + "feed": "BAR", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 27926, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790747912814468, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.37986297160387, + "feed": "BOOK_DELTA", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 824, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 977, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790748079031094, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.495369926095009, + "feed": "QUOTE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 695, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 936, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790748239861300, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.793825913220644, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 973, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 287, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790748398868190, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.82639786042273, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1962, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790748559189656, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.49989801645279, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 864, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 619, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790748732380015, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.354529902338982, + "feed": "BOOK_DELTA", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 955, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 743, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790748891402342, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.783197918906808, + "feed": "TRADE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1027, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 999, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790749051406269, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.87771381996572, + "feed": "QUOTE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 827, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 887, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790749215466252, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 20.689978962764144, + "feed": "BAR", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": "1m", + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 29395, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790749380339606, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.722089929506183, + "feed": "QUOTE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 932, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 352, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790749551583106, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.059505028650165, + "feed": "TRADE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1025, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 678, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790749714830130, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 37.825681967660785, + "feed": "BAR", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 29896, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790749878426056, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 19.13041388615966, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1224, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790750066857622, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 13.485180912539363, + "feed": "TRADE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 848, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 87, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790750237013267, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 19.51807690784335, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 825, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790750401190448, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.026678977534175, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1495, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 490, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790750571908526, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.647439958527684, + "feed": "BOOK_DELTA", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 636, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 572, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790750731502468, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 29.4035489205271, + "feed": "QUOTE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 767, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 684, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790750894860199, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 22.222691914066672, + "feed": "BOOK_DELTA", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 817, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 921, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790751074775587, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 19.734301138669252, + "feed": "BOOK_DELTA", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 671, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 82, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790751248545532, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 22.73307414725423, + "feed": "BAR", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": "1m", + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 31437, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790751419178875, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 14.593958156183362, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 516, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 1514, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790751592484451, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.732191054150462, + "feed": "QUOTE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 539, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 527, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790751758314303, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 29.506566002964973, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 824, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790751917579925, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.925353176891804, + "feed": "QUOTE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 601, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 757, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790752099560499, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 20.131282974034548, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 977, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790752259042112, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 22.64084992930293, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 834, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790752430697219, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.380588075146079, + "feed": "BOOK_DELTA", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1005, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 178, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790752604558934, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.436022913083434, + "feed": "BAR", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": "1m", + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 32775, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790752764455543, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 18.532793037593365, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 810, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 791, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790752928445979, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 25.13775695115328, + "feed": "QUOTE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 835, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 767, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790753097518313, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 15.020781895145774, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1081, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 124, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790753273164670, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 30.2232860121876, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1246, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790753438756747, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.689854992553592, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1067, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 468, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790753620645665, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.267744014039636, + "feed": "BOOK_DELTA", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 982, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 352, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790790753779909175, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 26.64185082539916, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 962, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 972, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790753960257887, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 6.83249905705452, + "feed": "TRADE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1422, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 363, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790754210517313, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 7.67448078840971, + "feed": "TRADE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1554, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 537, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790754386090066, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 49.03210490010679, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 978, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790754560145113, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 14.64993879199028, + "feed": "BOOK_DELTA", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 766, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 239, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790754760388213, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 16.73286920413375, + "feed": "BAR", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": "1m", + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 34939, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790754925711222, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 25.4819979891181, + "feed": "BOOK_DELTA", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 648, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 619, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790755093112092, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.791004005819559, + "feed": "QUOTE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 796, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 58, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790755312117558, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 15.116487862542272, + "feed": "TRADE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1108, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 633, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790755472416843, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 14.248264022171497, + "feed": "BAR", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": "1m", + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 35708, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790755666853978, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 21.758154965937138, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1667, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790755861138604, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.562961015850306, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 969, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 976, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790756034380100, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.504857007414103, + "feed": "BOOK_DELTA", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 829, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 699, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790756225344658, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 16.216858057305217, + "feed": "BAR", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": "1m", + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 36398, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790756385344524, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 15.094537055119872, + "feed": "TRADE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 638, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 87, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790756552195119, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 36.52388113550842, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1384, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790756742876279, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.590623831376433, + "feed": "TRADE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 900, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 1040, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790756931056182, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.724306942895055, + "feed": "QUOTE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 632, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 854, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790757110013947, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.76050498522818, + "feed": "TRADE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 729, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 779, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790757269254285, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 22.444518050178885, + "feed": "BOOK_DELTA", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 936, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 907, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790757430568833, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 45.98168400116265, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1544, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790757604616219, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 15.71027492173016, + "feed": "BAR", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": "1m", + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 37843, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790757802089535, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 16.36227685958147, + "feed": "BAR", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": "1m", + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 38010, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790757996872183, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.234177017584443, + "feed": "QUOTE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 665, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 788, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790758163814014, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 6.6195649560540915, + "feed": "QUOTE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 909, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 63, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790758322548810, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 14.710233081132174, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1423, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 421, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790758497554507, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.387687107548118, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1109, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 483, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790758662703035, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 7.669311016798019, + "feed": "TRADE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1110, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 1083, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790758825645365, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.561257928609848, + "feed": "BAR", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 38994, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790758983844978, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.720309171825647, + "feed": "BOOK_DELTA", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 697, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 879, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790759146850572, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.39476683177054, + "feed": "QUOTE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 828, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 922, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790759307022698, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.087957976385951, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1005, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 274, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790759465926895, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 24.05232796445489, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 628, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790759624535114, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.963885182514787, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1348, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 608, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790759799852940, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.508439037948847, + "feed": "BOOK_DELTA", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 846, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 652, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790759960463734, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.76655988395214, + "feed": "TRADE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1232, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 189, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790760121597270, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.067129110917449, + "feed": "QUOTE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 584, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 896, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790760280852949, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 13.06272903457284, + "feed": "BAR", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": "1m", + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 40451, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790760440454610, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 11.061954079195857, + "feed": "QUOTE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "STALE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE", + "LAST_EVENT_STALE" + ], + "freshness_ms": 2027, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 302, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790760604103094, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.580578025430441, + "feed": "TRADE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1674, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 644, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790760765785665, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 29.969539027661085, + "feed": "BAR", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 40949, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790760928804500, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 26.677563088014722, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1011, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790761110140664, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 11.783248977735639, + "feed": "TRADE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 2163, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 161, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790761288255026, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 27.30029192753136, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1365, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790761450579537, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 11.852724011987448, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1547, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 551, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790761629482009, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 14.14568698965013, + "feed": "BOOK_DELTA", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 877, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 457, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790761792372795, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 11.103617027401924, + "feed": "QUOTE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 1211, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 653, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790761957361934, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 11.382484110072255, + "feed": "BOOK_DELTA", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 737, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 782, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790762119044035, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 38.902620086446404, + "feed": "BOOK_DELTA", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 757, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 954, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790762281271130, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 14.690130949020386, + "feed": "BAR", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": "1m", + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 42483, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790762471131369, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.517096914350986, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1550, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 1556, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790762636398745, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.928714880719781, + "feed": "QUOTE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 799, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 493, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790762796410901, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 36.27476515248418, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 877, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790762957878889, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 15.377553878352046, + "feed": "QUOTE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 867, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 759, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790763145457454, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 21.88900299370289, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1564, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790763311392632, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 30.189492972567677, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1289, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790763484574654, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 11.243205051869154, + "feed": "BOOK_DELTA", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1068, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 41, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790763666649124, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 14.331539161503315, + "feed": "BAR", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": "1m", + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 43840, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790763828499016, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 18.841461976990104, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 866, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 694, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790763993347465, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.428068133071065, + "feed": "QUOTE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 666, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 725, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790764162712608, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.502768913283944, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 637, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 9, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790764321750604, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 20.32172493636608, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1287, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790764482854366, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 12.290380895137787, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1545, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 342, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790764654358161, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.418982883915305, + "feed": "BOOK_DELTA", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 626, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 198, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790790764817270687, + "usable": true, + "venue": "OKX" + } + ], + "consumer_age_semantics": "Redis first-observed updated watermark age includes up to 1000ms polling delay; not exact commit latency or proof every cached price remains eligible. MARK has no advancing consumer Kafka watermark; no inferred MARK apply latency.", + "consumer_observation": { + "consumers": { + "market_data_service.jsonl": { + "cache_rows_min_max": [ + 0, + 0 + ], + "elapsed_seconds": 299.995868966, + "health_states": { + "READY": 11 + }, + "latency": {}, + "ready_demand_pairs": [ + [ + 60, + 60 + ] + ], + "samples": 301, + "unique_heartbeats": 11, + "watermark_regressions": 0 + }, + "paper-market-redis-correction.jsonl": { + "cache_rows_min_max": [ + 50, + 50 + ], + "elapsed_seconds": 175.969257874, + "health_states": { + "READY": 7 + }, + "latency": { + "BINANCE/BOOK_DELTA": { + "max_ms": 1955.189208984375, + "median_ms": 439.9295654296875, + "min_ms": 112.52001953125, + "n": 878, + "p99_ms": 978.218505859375 + }, + "BINANCE/BOOK_SNAPSHOT": { + "max_ms": 3056.189208984375, + "median_ms": 2131.69091796875, + "min_ms": 917.06005859375, + "n": 29 + }, + "BINANCE/QUOTE": { + "max_ms": 1966.189208984375, + "median_ms": 437.37890625, + "min_ms": 143.78662109375, + "n": 879, + "p99_ms": 944.4404296875 + }, + "BINANCE/TRADE": { + "max_ms": 1985.189208984375, + "median_ms": 542.355712890625, + "min_ms": 117.52001953125, + "n": 877, + "p99_ms": 1416.106689453125 + }, + "OKX/BOOK_DELTA": { + "max_ms": 1990.189208984375, + "median_ms": 483.454345703125, + "min_ms": 182.6455078125, + "n": 880, + "p99_ms": 1086.169921875 + }, + "OKX/BOOK_SNAPSHOT": { + "max_ms": 2596.87060546875, + "median_ms": 1586.4324951171875, + "min_ms": 1283.31005859375, + "n": 30 + }, + "OKX/QUOTE": { + "max_ms": 2033.189208984375, + "median_ms": 445.235595703125, + "min_ms": 133.6728515625, + "n": 880, + "p99_ms": 1184.892578125 + }, + "OKX/TRADE": { + "max_ms": 3089.452880859375, + "median_ms": 701.189208984375, + "min_ms": 148.5126953125, + "n": 645, + "p99_ms": 1564.65283203125 + } + }, + "ready_demand_pairs": [ + [ + 60, + 60 + ] + ], + "samples": 177, + "unique_heartbeats": 7, + "watermark_regressions": 0 + }, + "ts-p183-p183_market_data-1.jsonl": { + "cache_rows_min_max": [ + 19, + 19 + ], + "elapsed_seconds": 299.992929524, + "health_states": { + "READY": 11 + }, + "latency": { + "BINANCE/BOOK_DELTA": { + "max_ms": 1495.02685546875, + "median_ms": 466.0186767578125, + "min_ms": 144.98095703125, + "n": 600, + "p99_ms": 1030.436767578125 + }, + "BINANCE/BOOK_SNAPSHOT": { + "max_ms": 2951.7890625, + "median_ms": 1800.9296875, + "min_ms": 711.967041015625, + "n": 527, + "p99_ms": 2765.80322265625 + }, + "BINANCE/QUOTE": { + "max_ms": 2014.02685546875, + "median_ms": 437.7734375, + "min_ms": 129.71142578125, + "n": 599, + "p99_ms": 1156.8193359375 + }, + "BINANCE/TRADE": { + "max_ms": 2225.02685546875, + "median_ms": 568.7735595703125, + "min_ms": 160.640625, + "n": 586, + "p99_ms": 1393.80615234375 + }, + "OKX/BOOK_DELTA": { + "max_ms": 1218.0703125, + "median_ms": 416.096923828125, + "min_ms": 114.817626953125, + "n": 594, + "p99_ms": 1022.100830078125 + }, + "OKX/BOOK_SNAPSHOT": { + "max_ms": 2915.76123046875, + "median_ms": 1518.054931640625, + "min_ms": 516.70654296875, + "n": 543, + "p99_ms": 2520.393310546875 + }, + "OKX/QUOTE": { + "max_ms": 3565.9296875, + "median_ms": 485.6600341796875, + "min_ms": 137.4287109375, + "n": 590, + "p99_ms": 2566.28662109375 + }, + "OKX/TRADE": { + "max_ms": 2087.873046875, + "median_ms": 604.656494140625, + "min_ms": 143.4287109375, + "n": 265, + "p99_ms": 1502.228271484375 + } + }, + "ready_demand_pairs": [ + [ + 22, + 22 + ] + ], + "samples": 301, + "unique_heartbeats": 11, + "watermark_regressions": 0 + } + }, + "notes": [ + "paper heartbeat300s; market-cache observation176s after observer route correction", + "event-to-observed-Redis includes up to1s sampling delay; not exact commit timestamp", + "OKX real transport disconnects occurred; do not claim zero disconnects" + ], + "resource": { + "samples": 25, + "whole_project_cpu_max": 5.1211, + "whole_project_cpu_mean": 4.70066 + }, + "state": "REVIEWED_WITH_OBSERVED_RECOVERY", + "unique_execution_products": 54 + }, + "date_utc": "2026-09-30", + "interruption_accounting": { + "okx_transport_disconnects": 4, + "paper_session_refusals": 12, + "recovered": true, + "root_socket_close_reason": "not logged; no provider/host attribution claimed", + "sandbox_session_refusals": 8 + }, + "limitations": [ + "No p99 for snapshot groups with fewer than100 samples.", + "Paper cache observed176s after Redis selection correction; heartbeat300s.", + "No 24/7 uptime, arbitrary50-alpha or indefinite5000/s certification.", + "Two OKX TRADE refusals attributed by exact public history ID/time; no stale price accepted.", + "Binance3d and VN/DNSE V2 explicitly excluded by owner." + ], + "measurement": "SDK snapshot request start to validated usable return; metadata resolution/startup separate", + "query_security_readback": { + "aggregate": [ + { + "attempted": 10, + "feed": "BAR", + "max_ms": 18.020760966464877, + "median_ms": 14.38249764032662, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "BOOK_DELTA", + "max_ms": 37.17497200705111, + "median_ms": 11.91028708126396, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "BOOK_SNAPSHOT", + "max_ms": 45.301543083041906, + "median_ms": 24.953530402854085, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "MARK_INDEX_PRICE", + "max_ms": 49.488725839182734, + "median_ms": 9.854138479568064, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "QUOTE", + "max_ms": 31.781164929270744, + "median_ms": 10.488643078133464, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "TRADE", + "max_ms": 29.223432997241616, + "median_ms": 9.380538947880268, + "p99_ms": null, + "rejected": 1, + "usable": 9, + "venue": "BINANCE" + }, + { + "attempted": 10, + "feed": "BAR", + "max_ms": 35.11799406260252, + "median_ms": 15.455134911462665, + "p99_ms": null, + "rejected": 0, + "usable": 10, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "BOOK_DELTA", + "max_ms": 12.32337299734354, + "median_ms": 9.44897800218314, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "BOOK_SNAPSHOT", + "max_ms": 52.96654417179525, + "median_ms": 21.612165844999254, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "MARK_INDEX_PRICE", + "max_ms": 20.63632500357926, + "median_ms": 9.368087514303625, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 12, + "feed": "QUOTE", + "max_ms": 13.692469103261828, + "median_ms": 8.754554437473416, + "p99_ms": null, + "rejected": 0, + "usable": 12, + "venue": "OKX" + }, + { + "attempted": 10, + "feed": "TRADE", + "max_ms": 18.729645991697907, + "median_ms": 10.846275486983359, + "p99_ms": null, + "rejected": 2, + "usable": 8, + "venue": "OKX" + } + ], + "binding_replica_reads": [ + { + "call_to_usable_ms": 49.488725839182734, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 411, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 417, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810994355595072, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.06308607570827, + "feed": "TRADE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1393, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 389, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810994665657434, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.431583970785141, + "feed": "TRADE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1064, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 583, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810994842773252, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.481417817994952, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 942, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810995037596171, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 12.32337299734354, + "feed": "BOOK_DELTA", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 810, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 683, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810995210005413, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 35.0016369484365, + "feed": "BAR", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": "1m", + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 55405, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810995373604272, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.886068128049374, + "feed": "BOOK_DELTA", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 789, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 60, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810995559262436, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.273669052869081, + "feed": "QUOTE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 950, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 640, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810995749228700, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 7.898355834186077, + "feed": "TRADE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1952, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 553, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810995908075141, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 16.11487101763487, + "feed": "BAR", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": "1m", + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 56124, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810996091844496, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 41.60477197729051, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1596, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810996277852002, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.022363064810634, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1404, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 1402, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810996470857217, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.05156198516488, + "feed": "BOOK_DELTA", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 647, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 65, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810996644863869, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 16.92242198623717, + "feed": "BAR", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": "1m", + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 56820, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810996805463860, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.7322310321033, + "feed": "TRADE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1229, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 689, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810996972866328, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 21.920586936175823, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1967, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810997149032695, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 18.729645991697907, + "feed": "TRADE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1073, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 717, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810997322275950, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.4815480299294, + "feed": "QUOTE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 565, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 375, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810997513561737, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.380538947880268, + "feed": "TRADE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1379, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 1375, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810997674598145, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.642908979207277, + "feed": "BOOK_DELTA", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 635, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 209, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810997834904198, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 29.024792136624455, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1149, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810997996224804, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 13.492486905306578, + "feed": "BAR", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": "1m", + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 58205, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810998176869503, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 20.452653989195824, + "feed": "BAR", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": "1m", + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 58373, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810998358377166, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.336498057469726, + "feed": "QUOTE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 855, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 300, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810998529355118, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 11.00929081439972, + "feed": "QUOTE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 839, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 556, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810998693205433, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.520881041884422, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 814, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 1809, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810998884848152, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.252148050814867, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 948, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 25, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810999045856049, + "usable": true, + "venue": "OKX" + }, + { + "call_until_rejection_ms": 9.208969073370099, + "feed": "TRADE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "STALE", + "execution_eligible": false, + "flags": [ + "LAST_EVENT_STALE" + ], + "freshness_ms": 3713, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 88, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810999204607394, + "usable": false, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 14.960739063099027, + "feed": "BAR", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 59377, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810999364379323, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.099513921886683, + "feed": "BOOK_DELTA", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 942, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 931, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810999529905682, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 29.848494101315737, + "feed": "QUOTE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 1210, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 441, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810999690871306, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.021239053457975, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1113, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 851, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790810999871276920, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 26.451079873368144, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 945, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811000031914854, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.516447972506285, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1091, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 185, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811000209646140, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 16.831265995278955, + "feed": "BOOK_DELTA", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 763, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 761, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811000373844787, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 16.49167505092919, + "feed": "TRADE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 748, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 184, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811000541233674, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 31.781164929270744, + "feed": "QUOTE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 895, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 439, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811000708273879, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 16.64903503842652, + "feed": "BAR", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": "1m", + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 904, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811000890633905, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.335457976907492, + "feed": "QUOTE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 802, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 879, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811001058143020, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.994543924927711, + "feed": "TRADE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1562, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 1089, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811001219084218, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 35.11799406260252, + "feed": "BAR", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 1406, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811001378599207, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 45.301543083041906, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1583, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811001564288928, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 29.223432997241616, + "feed": "TRADE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1117, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 461, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811001760974080, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 18.845013109967113, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 811, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811001940872368, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.202355122193694, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1030, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 40, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811002110862804, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.974422933533788, + "feed": "BOOK_DELTA", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 580, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 613, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811002269538180, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 13.692469103261828, + "feed": "QUOTE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 534, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 236, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811002429109872, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 31.510137021541595, + "feed": "BOOK_DELTA", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 859, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 954, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811002593351598, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 37.17497200705111, + "feed": "BOOK_DELTA", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 856, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 116, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811002775901522, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 15.326251974329352, + "feed": "BAR", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": "1m", + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 2975, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811002963701646, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.309446776285768, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1066, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 67, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811003129629677, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.637340972200036, + "feed": "QUOTE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 892, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 93, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811003301495661, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 17.790155950933695, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1162, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811003460677249, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.664449073374271, + "feed": "QUOTE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 642, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 316, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811003629711735, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 37.96493192203343, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1216, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811003790855986, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 20.08475107140839, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1585, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811003980045847, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.360584964975715, + "feed": "BOOK_DELTA", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 654, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 522, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811004151387458, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.568861177191138, + "feed": "BAR", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": "1m", + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 4321, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811004310262017, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 6.982878083363175, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1253, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 427, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811004474331692, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.312837082892656, + "feed": "QUOTE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 797, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 212, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811004631837634, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 13.428292935714126, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1106, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 750, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811004792750701, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 20.512596936896443, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 557, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811004956688377, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 20.63632500357926, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 917, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 88, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811005128551941, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.402227889746428, + "feed": "BOOK_DELTA", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 703, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 574, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 1, + "request_ns": 1790811005299828378, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.187395917251706, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 441, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 449, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811005465198154, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 14.414103934541345, + "feed": "TRADE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 2068, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 1010, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811005690475159, + "usable": true, + "venue": "OKX" + }, + { + "call_until_rejection_ms": 8.175212889909744, + "feed": "TRADE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "STALE", + "execution_eligible": false, + "flags": [ + "LAST_EVENT_STALE" + ], + "freshness_ms": 3640, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 1191, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811005875260484, + "usable": false, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.14545600488782, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 958, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811006056543459, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.095371002331376, + "feed": "BOOK_DELTA", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 729, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 503, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811006229283857, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 12.349573895335197, + "feed": "BAR", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": "1m", + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 6399, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811006388861960, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 10.368140880018473, + "feed": "BOOK_DELTA", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 574, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 847, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811006551701508, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 7.9507180489599705, + "feed": "QUOTE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 544, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 454, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811006733626934, + "usable": true, + "venue": "OKX" + }, + { + "call_until_rejection_ms": 8.326632902026176, + "feed": "TRADE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "STALE", + "execution_eligible": false, + "flags": [ + "LAST_EVENT_STALE" + ], + "freshness_ms": 9746, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 988, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811006892125249, + "usable": false, + "venue": "OKX" + }, + { + "call_to_usable_ms": 12.029455043375492, + "feed": "BAR", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": "1m", + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 7099, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811007072236500, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 52.96654417179525, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1557, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811007251686330, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 7.985491072759032, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1398, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 397, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811007456092025, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.284334886819124, + "feed": "BOOK_DELTA", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 532, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 905, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811007630876822, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 13.145314995199442, + "feed": "BAR", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": "1m", + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 7802, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811007791704424, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 7.8452820889651775, + "feed": "TRADE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 977, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 90, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811007955422476, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 22.141798865050077, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1149, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811008130680093, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.731641039252281, + "feed": "TRADE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1872, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 1049, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811008304219162, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 7.829521084204316, + "feed": "QUOTE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 943, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 195, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811008483211943, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.208364946767688, + "feed": "TRADE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 719, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 760, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811008641566479, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.231959003955126, + "feed": "BOOK_DELTA", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 799, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 72, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811008800241635, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 26.953021064400673, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 889, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811008960085824, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.104080058634281, + "feed": "BAR", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": "1m", + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 9165, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811009138563233, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 15.584017848595977, + "feed": "BAR", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": "1m", + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 9334, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811009321686504, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.102098014205694, + "feed": "QUOTE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 776, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 9, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811009487857750, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.191819164901972, + "feed": "QUOTE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 778, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 375, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811009646458031, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 8.651823038235307, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1753, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 750, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811009824442409, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.417501045390964, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 399, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 909, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811009983595576, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.555560071021318, + "feed": "TRADE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1312, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 83, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811010143574914, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.349965050816536, + "feed": "BAR", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 10314, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811010303669579, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 11.462989030405879, + "feed": "BOOK_DELTA", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 671, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 694, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811010467139986, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.437711909413338, + "feed": "QUOTE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 815, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 133, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811010629273279, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 7.904856000095606, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 566, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 711, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811010789226095, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.742913872003555, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 854, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811010947682748, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.318673983216286, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "a6884fb3-1fa0-53e0-9621-d01ba5f9a2de", + "interval": null, + "market": "SWAP", + "native_symbol": "SOL-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 900, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 38, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811011120664722, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 12.35758513212204, + "feed": "BOOK_DELTA", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 608, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 487, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811011280544414, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 11.260967003181577, + "feed": "TRADE", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 786, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 807, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811011443533070, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.060947922989726, + "feed": "QUOTE", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 983, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 106, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811011605345314, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 18.020760966464877, + "feed": "BAR", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": "1m", + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 11780, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811011765024493, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.730947971343994, + "feed": "QUOTE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 1197, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 628, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811011933681145, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 7.891335058957338, + "feed": "TRADE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 974, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 976, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811012093039114, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 12.434880016371608, + "feed": "BAR", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": "1m", + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 12262, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811012251406161, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.19841007515788, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1156, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811012414378287, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 14.989319955930114, + "feed": "TRADE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 624, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 131, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811012587393743, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 23.86126690544188, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1016, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811012752900768, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 10.617666877806187, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e4ce7249-a4e8-5073-beb8-dab908596c34", + "interval": null, + "market": "USDM", + "native_symbol": "SOLUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 847, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 858, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811012928194597, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.830134058371186, + "feed": "BOOK_DELTA", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 573, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 292, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811013089328017, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.778160903602839, + "feed": "QUOTE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 758, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 942, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811013248775352, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 16.03714586235583, + "feed": "BOOK_DELTA", + "instrument_uid": "a953e16e-7138-5562-b5e8-c337a44d0b65", + "interval": null, + "market": "USDM", + "native_symbol": "BTCUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 746, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 614, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811013408345214, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.964204858988523, + "feed": "BOOK_DELTA", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 677, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 778, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811013575274728, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 17.047169851139188, + "feed": "BAR", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": "1m", + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FIELD_MISSING" + ], + "freshness_ms": 13750, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811013735892057, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 11.241635074838996, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "b2d78145-b541-58e2-941b-06c44c65e45f", + "interval": null, + "market": "USDM", + "native_symbol": "BNBUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 826, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 831, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811013903521024, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.306180825456977, + "feed": "QUOTE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 516, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 757, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811014065327237, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 18.834471935406327, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "fb26214c-7b9b-5961-95b2-55154755af0f", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 926, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811014224203016, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 21.430494962260127, + "feed": "QUOTE", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 980, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 887, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811014394735126, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 26.04579390026629, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": null, + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 775, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811014566829624, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 23.403137922286987, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1750, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811014744280158, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.665997000411153, + "feed": "BOOK_DELTA", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1023, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 91, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811014919046184, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 15.1961341034621, + "feed": "BAR", + "instrument_uid": "8aedd349-6999-5874-b0dd-34c6451c0b3a", + "interval": "1m", + "market": "USDM", + "native_symbol": "DOGEUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 15092, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811015079315121, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 8.667822927236557, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "f2e37e2b-1386-5a32-9b79-0fd39ec7a5a3", + "interval": null, + "market": "SWAP", + "native_symbol": "BNB-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 565, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 140, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811015245017138, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.509437019005418, + "feed": "QUOTE", + "instrument_uid": "ee93fabf-68df-5b50-8924-51bf25a5a757", + "interval": null, + "market": "USDM", + "native_symbol": "ETHUSDT", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "DELIVERY_ON_CHANGE" + ], + "freshness_ms": 552, + "gap_open": false, + "policy_id": "crypto_primary_v2", + "provider_session_liveness_ms": 880, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811015404252919, + "usable": true, + "venue": "BINANCE" + }, + { + "call_to_usable_ms": 9.456312982365489, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 672, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 459, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811015564338944, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 26.150631019845605, + "feed": "BOOK_SNAPSHOT", + "instrument_uid": "e49b54ae-c23d-5351-9e64-47934aac28f8", + "interval": null, + "market": "SWAP", + "native_symbol": "ETH-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 1327, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": null, + "provider_session_state": "NOT_APPLICABLE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811015724337183, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.08017996698618, + "feed": "MARK_INDEX_PRICE", + "instrument_uid": "6c7c9256-2905-5c75-a149-fa0ac36bbbc7", + "interval": null, + "market": "SWAP", + "native_symbol": "DOGE-USDT-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [ + "FRESHNESS_BASIS_PROVIDER_CONFIRMATION" + ], + "freshness_ms": 1215, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 798, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811015902391645, + "usable": true, + "venue": "OKX" + }, + { + "call_to_usable_ms": 9.068988962098956, + "feed": "BOOK_DELTA", + "instrument_uid": "8482b54b-6d9d-5aad-abe8-e22447d183a1", + "interval": null, + "market": "SWAP", + "native_symbol": "BTC-USD-SWAP", + "quality": { + "complete": true, + "event_recency_state": "LIVE", + "execution_eligible": true, + "flags": [], + "freshness_ms": 966, + "gap_open": false, + "policy_id": "crypto_liquid_v2", + "provider_session_liveness_ms": 232, + "provider_session_state": "LIVE", + "state": "LIVE" + }, + "replica": 2, + "request_ns": 1790811016062058504, + "usable": true, + "venue": "OKX" + } + ], + "image": "sha256:b40b49c361c7474691ef2d68eba8bcb4a22045cb2d010e2c73a8c0c68cc04a65", + "review": { + "attempted": 128, + "auth_or_manifest_errors": 0, + "diagnostic_reads": "BoundedexplicitKafkaassign/read_committed/nooffsetstore/nocommit. Two earlierrole/groupACLdenialsretained, noACLchange; correctedexistingknprojectorgroupauthenticatedmanualassigndoesnotjoin/rebalancegroup.", + "policy": "Last-tradeexecutionage3s unchanged; everyineligibleviewrejected; requestlatency and age are separate. Short propagation after quiet interval can temporarily refuse, not always-eligible SLA.", + "rejections": [ + { + "classification": "QUIET_LAST_TRADE", + "proof": "PubliclatestaggTrade a1102112607 covers rawtrade3566012294..3566012310; servednativeid3566012310 islast. AggregateT5496 versusrawT5497 are not identical timestamps; never compareaggregateIDtorawID.", + "symbol": "SOLUSDT", + "venue": "BINANCE" + }, + { + "classification": "QUIET_LAST_TRADE", + "proof": "ExactprovidertradeID239930010 andtimestamp1790810997172 match.", + "symbol": "BNB-USDT-SWAP", + "venue": "OKX" + }, + { + "classification": "NEW_TRADE_PROPAGATION_NOT_QUIET", + "proof": "Providertrade1639459093 occurred228.260484ms before request. Rawhostreceived27.402661ms afterprovider; canonicalrecordCreateTime1790811005721. Historicalread provescanonicalpreservation, not transactioncommitvisibility atqueryinstant. Exactcommit/cacheapplytime unavailable; no fabricatedquiet orlatencyclaim. Follow-up7/10usable,3age-policyrefusals thennewertradevisible, no stucksnapshot; do notreplace125/128withthisfollow-up.", + "symbol": "SOL-USDT-SWAP", + "venue": "OKX" + } + ], + "scope": "64entitlementproducts x2queryreplicas, actualsandboxidentity; no order/stream/ACK", + "state": "AFFECTED_QUERY_SECURITY_READBACK_PASS", + "typed_views": 128, + "usable": 125 + }, + "source": "e366f752914db8d1d43cb03a970b0278635ba88d" + }, + "release": "v2.2.1", + "schema": "qdl.release.endpoint-delta.v1", + "scope": "64 entitlement products x two query replicas; actual paper60 and sandbox22 demand", + "unchanged_endpoints": { + "evidence": "../v2.2.0/endpoint-report.json", + "note": "Historical measurements only: warmup/batch/reference/diagnostics/other unchanged routes not rebenchmarked in this repair.", + "retested": false, + "sha256": "d87e7f70650d954cb31051627e9c85b93d1d7372ad7b1753f1ec9018487f2eec" + }, + "units": "milliseconds" +}