diff --git a/docs/architecture.md b/docs/architecture.md index 73a9d353..3037968c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,5 +1,14 @@ # Architecture and domain contracts +D430 lets the curve page load only the exact definition dependencies of the +selected curve. Pricing, frame validation and publication authorization use +the original inputs and policies. See [the decision](decisions.md#d430--read-exact-curve-definition-dependencies). + +D429 separates first-use curve catalog hydration from its bounded price result +and streams owned read bytes and calendar batch input. Existing projections and +authorization remain authoritative; see [the decision](decisions.md#d429--separate-curve-catalog-hydration-and-bound-read-serialization) +and [qualified workloads](verification.md#curve-hydration-and-read-memory-d429). + D427 adds opt-in personal [assistant history search](assistant-workspace.md#personal-chat-history-search-d427) and retains walkthrough guidance in conversation history. Search uses current user/agent authority and returns prior interactions with their original context; diff --git a/docs/async-reads.md b/docs/async-reads.md index fac07da7..c1a368e5 100644 --- a/docs/async-reads.md +++ b/docs/async-reads.md @@ -94,10 +94,15 @@ before admission; the client renews a drained channel before submitting further requests. Request and serialized-result caps are 32 KiB and 16 MiB. A channel lasts five minutes and uses periodic authorization checks and heartbeat writes. Work cancellation is cooperative: synchronous graph evaluation -cannot be forcibly preempted by a cancellation token. A serialized-result cap is -not a peak-memory guarantee. Publication evidence is bounded to 512 source or -reference/calendar coordinates, 4,096 original book identities and 4,096 retained -instrument authority proofs. +cannot be forcibly preempted by a cancellation token. The result is serialized +once into an owned UTF-8 buffer, with its byte budget checked at writer commits +before publication, including DOM values. Publication writes that buffer in 16 +KiB chunks inside the existing authorization callback; one write deadline covers +the complete event and its final flush. The existing 64-level result nesting +limit is preserved inside the additional envelope level. A serialized-result cap +is not a peak-memory guarantee. Publication evidence is bounded to 512 source or +reference/calendar coordinates, 4,096 original book identities and 4,096 +retained instrument authority proofs. Changing a selection rejects the old consumer immediately. A submitted cancellation keeps its execution slot until a terminal server response or an @@ -110,6 +115,25 @@ an aborted HTTP submission that never reached admission to release its slot. Disconnect rejects pending reads. A subsequent explicit request establishes a new channel; there is no silent replay or durable delivery guarantee. +The curve page hydrates its catalog through the existing ordinary +`workspace/catalog` read before its first price read. It validates both results +against the same captured selection and installs them together. Later price +reads reuse the scoped catalog. This keeps a large catalog outside the price +result's byte budget; it adds a first-use HTTP/database read. The explicit +`includeWorkspace` API option retains its combined-result contract and byte cap. + +The page uses `workspace/catalog?frameDetail=summary` to omit per-frame input +IDs and requests `sourceScope: "curve_lineage_v1"` with `includeSources: true` +to receive only the point lineage anchors and their visible fact versions. +The full frame remains the authority and correction-admission witness. +The page schedules the curve's advisory close strip after the prices paint. + +For both ordinary HTTP and `curve.evaluate.v1`, the page also sends +`definitionScope: "curve_dependencies_v1"`. This selector requires the +curve-lineage projection and a separate catalog read. The server reads +the selected curve's declared dependency closure under the existing caller +and exact frame authority. Omitting the selector uses the full definition read. + Financial commands retain their existing request identities, transaction ordering and uncertain-outcome recovery. Retained Risk jobs keep their existing durable admission and report contracts. Progressive trade rows retain their dedicated diff --git a/docs/decisions.md b/docs/decisions.md index 9f4aaddb..c06fc80a 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -6651,6 +6651,65 @@ hydration advantage for the tested Protobuf candidates over gzip JSON; see the measurements do not establish a production latency guarantee. Broader transport, projection-size or deployment changes require separate qualification. + +## D429 — Separate curve catalog hydration and bound read serialization + +The curve page reads its catalog through the existing `workspace/catalog` HTTP +operation, then requests prices without embedding that catalog in the SSE result. +Both reads retain the captured profile, generation, selected frame and source +revision checks; the page installs their data only while that selection remains +current. A hydrated catalog is reused within its existing scope. First hydration +adds one ordinary HTTP read and its database work. Explicit API callers may still +request the existing combined projection. + +The page requests frame summaries and sends an explicit curve-lineage source +projection. The evaluation still binds the same full input frame, checks its +original pins and reauthorizes before publication. The price payload includes +the anchors used by its points and visible versions of those same facts. +Correction eligibility remains tied to the full selected frame. Browser indexes +preserve exact-lookup failures. The close strip starts after the chart can paint +and a superseded strip read is cancelled. This scheduling applies to the +advisory strip when selecting a curve, while choosing a close still uses its +authorized availability read. + +Keep the channel's 16 MiB serialized-result limit. Enforce actual committed JSON +bytes at the UTF-8 writer boundary, including synchronous JSON DOM converters, +and retain the successful buffer for publication. The same authorized +publication callback writes its correlated envelope and result in bounded chunks +under one write deadline. It preserves exact values, structured refusals and +cancellation; a truncated event cannot be completed as a successful response. +The existing 64-level result nesting bound is preserved inside its additional +envelope level. + +Calendar pin batches read the same native function once, stream its UTF-8 payload +through a sequential database reader, and retain individual rows and denied-frame +identities. Numeric lexeme conversion, complete pin selection, reached-frame +visibility evidence and denial/error precedence keep their existing contracts. +The aggregate UTF-16 string and aggregate JSON document are unnecessary copies. +No schema, stored format, financial input or command-identity change is involved. + +The logical byte limit is not a heap ceiling. Encoding reservations, evaluation, +metadata and the complete calendar observation union still consume memory. +[Qualification](verification.md#curve-hydration-and-read-memory-d429) records +separate browser, serializer, catalog and batch workloads; it does not qualify +arbitrary batch sizes, simultaneous large reads or production latency. + +## D430 — Read exact curve definition dependencies + +The curve page requests the versioned definition scope +`curve_dependencies_v1` with the compact curve-lineage result. The ordinary +HTTP and asynchronous operations admit the same scope. The native read follows +declared dependencies inside the bound exact definition and retains all +selected curve members, mappings, schedules and rate tenors needed by the +existing evaluator. Series and contract metadata for every original selected +pin remain available for full snapshot validation. Row security and publication +checks use the original caller, frame and input identities. + +There is no cross-request definition cache. Keep the full definition read for +callers that omit the selector and for legacy supplementary sources without a +recorded exact source matrix. Retain gzip JSON and the separate catalog read. +[Verification](verification.md#curve-definition-dependency-reads-d430) qualifies +the warm price path on two owned synthetic workloads. ## D429 — Manage a registered risk worker service from Jobs An installation binds one reviewed Swarm service and a dedicated risk-worker login to its workspace. Jobs can request zero, one or two replicas. Scaling requires both workspace permission and current installation operator registration. A separate permission grants access to bounded worker and node diagnostics. Installations choose the roles that receive these permissions. diff --git a/docs/plan.md b/docs/plan.md index 7d055ba7..b8ae19be 100644 --- a/docs/plan.md +++ b/docs/plan.md @@ -1,5 +1,19 @@ # Delivery plan and acceptance criteria +D429 qualifies separate catalog hydration, bounded asynchronous JSON serialization +and streamed calendar batch decoding in owned synthetic workloads. Larger +concurrent workloads, first-use request-count costs and production latency remain +capacity/interaction measurements; the logical response cap is unchanged. See +[verification](verification.md#curve-hydration-and-read-memory-d429). + +D430 adds an explicit dependency-scoped definition read to curve evaluation. +[Local qualification](verification.md#curve-definition-dependency-reads-d430) +covers exact response parity and warm click-to-chart time with unrelated +contracts at two market sizes. Cold hydration, production-like latency and +concurrent capacity remain open. A curve-scoped display projection and managed +display formatting in the price response remain proposed changes. + + D427 adds personal chat-history search through an explicit composer checkbox and repairs retained walkthrough narration and final-pause verification. Its bounded contract is in [workspace assistance](assistant-workspace.md#personal-chat-history-search-d427). diff --git a/docs/verification.md b/docs/verification.md index 52ac0008..08511c16 100644 --- a/docs/verification.md +++ b/docs/verification.md @@ -5550,3 +5550,138 @@ not isolated as a comparable standalone CPU stage. The large history projection has API parity; its unavailable-input UI states are outside the covered history render case. These limits support retaining JSON, not a universal claim about binary encoding or a production service-level guarantee. + +## Curve hydration and read memory (D429) + +Release checks on 2026-11-18 reproduce the old behavior before each memory fix: +serializing an oversized lazy result walks all 8,192 rows before rejecting it; +an oversized JSON object allocates 268,397,056 bytes before the stream limit +runs; materializing a 71,095,510-byte synthetic pin batch exhausts a 256 MiB +managed heap. The corrected writer stops the lazy result at row 2,049. Oversized +JSON object/element checks allocate 41,975,840 / 67,100,520 bytes respectively; +a definitely oversized string refuses before its encoded copy is allocated. +These are serialization allocation checks, excluding construction of their inputs. + +The complete async-read checks pass exact logical UTF-8 limits at 16 MiB minus +one byte, exactly 16 MiB and one byte over, including escaped non-ASCII text. +They cover exact financial text, null/empty fields, the existing 64-level result +plus its envelope, serialization cancellation, correlated failures, partial-write cancellation, +one total event deadline, gzip/identity negotiation and ready-event flushing. +Writing a precomputed 4 MiB result adds 896 allocated bytes in its measured check; +the counting sink is preallocated and excluded. This is a publication-stage +allocation result, not the memory cost of evaluating that result. + +An opt-in backend HTTP case uses actual managed native-curve metadata, 20,000 +contracts, 20,000 observations, one selected contract and three sequential base +price reads. It serves a complete 17,026,511-byte catalog separately; its gzip +body is 3,928,369 bytes. Each price result preserves its complete precomputed +business projection and carries no catalog. This Release process includes both +the Kestrel server and its HTTP validation client. Its managed heap limit is +768 MiB and measured peak resident size is 759,054,336 bytes (724 MiB). + +A separate input-decoding case reads 525,123 original synthetic observations +from a 252,887,126-byte UTF-8 batch and builds the native observation indexes. +It preserves every row, exact decimal text and denied identity. Under the same +768 MiB managed heap limit, measured peak resident size is 798,736,384 bytes +(762 MiB). This case includes batch decoding/indexing, not an API server, +database process or simultaneous curve evaluation. The complete batch remains +resident; larger or concurrent batches are outside this measured envelope. + +Both cases run on macOS 26.6.2 arm64, 14 logical CPUs, 48 GiB host RAM, +.NET/ASP.NET Core 10.0.13 and SDK 10.0.402, using Release builds. The RSS source +is the operating system's `time -l`, rather than the runtime's unavailable +peak-working-set counter on this platform. The managed heap constraint is not +a Linux container/cgroup qualification. HTTP bytes above are compressed body +bytes over loopback, excluding headers, TLS and proxy framing. These cases do +not measure production latency, bandwidth, browser rendering or concurrent-user +capacity. + +The owned actual-browser regression passes 13 checks, including a real power +curve, first hydration with over 16 MiB of injected unselected catalog metadata, +rapid curve switches, a catalog response delivered after supersession, pending +Risk reads and exact startup readiness. The metadata injection occurs after the +small fixture's HTTP response; it qualifies browser behavior rather than server +memory. The separate backend-scale case above covers server serialization. + +The page projection comparison in the owned asynchronous HTTP fixture covers +four frames, 30 catalog contracts and six points of a nested power curve at one +exact close. The compact price result retains the same points, trace, selected +source and correction eligibility. It carries 31 observation rows versus 40 +in the full-frame response and 51,718 versus 57,747 bytes as compact JSON +re-encoded by the test client. Frame summaries reduce the separate catalog +from 201,998 to 185,542 bytes by the same measure. Ordinary HTTP and SSE return +identical compact projections. Further owned checks cover exact rate tenors, +empty eligible delivery, a non-primary source revision and unknown projection +refusal. The payload counts exclude the response envelope and compression, and +both variants used the full market calculation input load. The owned calendar +run passed 67 browser checks. It confirms that the close strip starts after its +selected prices draw and that editing the context cancels a pending strip +response in the tested flow. + +Native PostgreSQL calendar checks preserve the unchanged answer hashes, source +matrices, reached/unvisited denial precedence, original authority, current +publication permission, close replacement and profile authorization. The async +suite separately verifies profile/sign-in fences. Statement counts remain 101 +for 24, 48 and 96 candidates. The macOS run adapts `ps` sampling only for the +check harness's Linux `/proc` counters; business assertions are unchanged. CPU +samples have 0.01-second resolution and calendar peak RSS is unavailable in that +sampler; these samples establish no performance claim. API/checks and Worker +Release builds pass. + +The optional managed HTTP case is reproducible with: + +```sh +dotnet build tests/OpenEtrm.Api.Checks -c Release +DOTNET_GCHeapHardLimit=30000000 dotnet \ + tests/OpenEtrm.Api.Checks/bin/Release/net10.0/OpenEtrm.Api.Checks.dll \ + --large-catalog-http +``` + +`DOTNET_GCHeapHardLimit` is hexadecimal here: `30000000` selects 768 MiB. +The file-input check accepts `--observation-pin-payload FILE EXPECTED_ROWS` for +an explicitly generated complete synthetic batch; generated payloads and local +operational evidence stay outside the repository. No schema migration or new +dependency is required. The first curve interaction adds an ordinary catalog +HTTP/database read; its added round-trip cost and production interaction latency +remain to be measured after an approved release. + +## Curve definition dependency reads (D430) + +The owned synthetic native preview compared full and dependency-scoped reads +with the same compact JSON projection, gzip configuration and browser page. +The only request difference was `definitionScope`. The target nested base +plus basis curve had 76 output points, 152 participating contracts, 152 +consumed observations per close and 228 trace nodes. Four synthetic closes +were present. Scaling unrelated contracts and series raised the total native +definition from 3,031 to 12,031 contracts. Every measured HTTP result matched +its full-definition counterpart exactly. + +| Contracts in definition | HTTP p50, full → scoped | HTTP sample max | SQL statements per read, median | Warm chart p50, full → scoped | Warm chart sample max | +| ---: | ---: | ---: | ---: | ---: | ---: | +| 3,031 | 102.7 → 70.3 ms | 117.9 → 82.6 ms | 98 → 47 | 131.5 → 101.2 ms | 143.9 → 111.8 ms | +| 12,031 | 191.7 → 170.8 ms | 222.9 → 188.3 ms | 98 → 47 | 233.2 → 103.4 ms | 253.8 → 107.8 ms | + +Runs used macOS 26.6.2 arm64, 14 logical CPUs, 48 GiB RAM, PostgreSQL +18.7, .NET/ASP.NET Core 10.0.13, SDK 10.0.402 Release builds and headless +Chrome 157.0.0.0. Each scale used nine alternating paired HTTP reads and +eight alternating paired browser clicks on loopback. Chrome consumed the real +SSE stream and changed only the outgoing definition selector. Browser timing starts +with the click from an already hydrated six-point curve and ends after +the selected 76-point table is ready and two animation frames have elapsed. +The HTTP statement count comes from native preview runtime logs. Median +summed statement duration was 66.0 → 50.9 ms and 109.2 → 153.5 ms at the +two scales. A separate 12,031-contract HTTP run also had exact parity and +98 → 47 statements, with round-trip medians of 186.5 → 71.2 ms and summed +statement medians of 107.2 → 55.4 ms. The difference between the large-scale +HTTP runs remains unexplained. These sample maxima describe run variation +within each run. Process peak memory, cold hydration, calibrated proxy latency, +concurrent users and a 96-close dataset remain open. The dependency-scoped +reader applies to price evaluation. + +The native check passed 50 HTTP/SSE assertions, including identical compact +power and rate responses under working and certified contexts. A scoped +curve blocked on the publication gate refused disclosure after its authority +expired. Separate checks passed an exact non-primary recorded source and an +expired curve with no eligible output. The 13 owned browser checks passed, +including selection supersession and late catalog delivery. All 2,244 +frontend tests passed. diff --git a/src/OpenEtrm.Api/AsyncReadChannel.cs b/src/OpenEtrm.Api/AsyncReadChannel.cs index 30fc1070..9f9b8b72 100644 --- a/src/OpenEtrm.Api/AsyncReadChannel.cs +++ b/src/OpenEtrm.Api/AsyncReadChannel.cs @@ -1,3 +1,4 @@ +using System.Buffers; using System.Security.Cryptography; using System.Text.Json; using System.Text.Json.Serialization; @@ -10,7 +11,7 @@ namespace OpenEtrm.Api; internal sealed record AsyncReadError(string Code,string Message); internal sealed record AsyncReadFailure(string Operation,Guid RequestId,string ExceptionType,string? Method,string? SqlState); -internal sealed record AsyncReadCompletion(AsyncReadRequest Request,ReadPlan Plan,string State,JsonElement? Result,AsyncReadError? Error, +internal sealed record AsyncReadCompletion(AsyncReadRequest Request,ReadPlan Plan,string State,ReadOnlyMemory? Result,AsyncReadError? Error, Func? Recheck=null); // Ephemeral reads for the workbench. This channel retains request hashes, @@ -20,6 +21,7 @@ internal sealed class AsyncReadChannel:IAsyncDisposable internal const string Protocol="async_reads_v1"; internal const int MaximumActive=2,MaximumRequests=128,MaximumRequestBytes=32*1024,MaximumResultBytes=16*1024*1024; internal static readonly TimeSpan Lifetime=TimeSpan.FromMinutes(5),ReadLimit=TimeSpan.FromSeconds(60); + static readonly JsonSerializerOptions ResultJson=new(AsyncReadContract.Json){MaxDepth=64,Converters={new ResultStringConverter()}}; readonly object gate=new(); readonly CancellationTokenSource lifetime; readonly CancellationToken stopping; @@ -91,10 +93,16 @@ async Task Evaluate(AsyncReadRequest request,ReadPlan plan,Pending pending) try { var outcome=await AsyncReadVisibility.Observed(plan).Execute(pending.Cancellation.Token);pending.Cancellation.Token.ThrowIfCancellationRequested(); - var bytes=JsonSerializer.SerializeToUtf8Bytes(outcome.Value,AsyncReadContract.Json); - if(bytes.Length>MaximumResultBytes)throw Refusal(422,"async_result_size","The read result exceeds the response limit."); - using var json=JsonDocument.Parse(bytes); - completion=new(request,plan,"completed",json.RootElement.Clone(),null,outcome.Recheck); + var output=new ResultBuffer(pending.Cancellation.Token); + using(var writer=new Utf8JsonWriter(output,new JsonWriterOptions{MaxDepth=ResultJson.MaxDepth})) + { + // Use the budgeted writer directly: DOM converters may otherwise + // buffer the entire value before a Stream sees its first write. + JsonSerializer.Serialize(writer,outcome.Value,ResultJson);writer.Flush(); + } + pending.Cancellation.Token.ThrowIfCancellationRequested(); + // The completion owns this buffer until publication or channel disposal. + completion=new(request,plan,"completed",output.Written,null,outcome.Recheck); } catch(OperationCanceledException){completion=new(request,plan,"cancelled",null,new("request_cancelled","The read was cancelled or exceeded its time limit."));} catch(Exception e) @@ -108,6 +116,41 @@ async Task Evaluate(AsyncReadRequest request,ReadPlan plan,Pending pending) if(!Closed&&!responses.Writer.TryWrite(completion))Close(); } } + static AsyncReadException ResultLimit()=>Refusal(422,"async_result_size","The read result exceeds the response limit."); + sealed class ResultBuffer(CancellationToken ct):IBufferWriter + { + readonly ArrayBufferWriter buffer=new(16*1024); + internal ReadOnlyMemory Written=>buffer.WrittenMemory; + public void Advance(int count) + { + ct.ThrowIfCancellationRequested(); + if(count>MaximumResultBytes-buffer.WrittenCount)throw ResultLimit(); + buffer.Advance(count); + } + public Memory GetMemory(int sizeHint=0) + { + ct.ThrowIfCancellationRequested(); + // The JSON writer reserves for worst-case escaping. Honor valid + // hints without treating reserved capacity as serialized bytes. + if(sizeHint>6*MaximumResultBytes+256)throw ResultLimit(); + var size=Math.Max(sizeHint,16*1024); + // Short spans force even a synchronous DOM converter to commit its + // bytes regularly, where Advance enforces the logical result cap. + return buffer.GetMemory(size)[..size]; + } + public Span GetSpan(int sizeHint=0)=>GetMemory(sizeHint).Span; + } + sealed class ResultStringConverter:JsonConverter + { + public override string? Read(ref Utf8JsonReader reader,Type type,JsonSerializerOptions options)=>reader.GetString(); + public override void Write(Utf8JsonWriter writer,string value,JsonSerializerOptions options) + { + // Every UTF-16 code unit needs at least one JSON byte. Refuse a + // definitely oversized token before the JSON writer rents its buffer. + if(value.Length>MaximumResultBytes)throw ResultLimit(); + writer.WriteStringValue(value); + } + } void ReportFailure(AsyncReadRequest request,Exception error) { var method=error.TargetSite; diff --git a/src/OpenEtrm.Api/AsyncReadContract.cs b/src/OpenEtrm.Api/AsyncReadContract.cs index 234861d5..9906ab78 100644 --- a/src/OpenEtrm.Api/AsyncReadContract.cs +++ b/src/OpenEtrm.Api/AsyncReadContract.cs @@ -31,6 +31,7 @@ internal sealed class AsyncReadException(int status,string code,string message): // URLs, commands, retained-job submission and cancellation are not operations. internal static class AsyncReadContract { + internal const string CurveDependencies="curve_dependencies_v1"; internal static readonly JsonSerializerOptions Json=new(JsonSerializerDefaults.Web) {AllowDuplicateProperties=false,Converters={new ExactDecimalJsonConverter()}}; internal static ReadInput Decode(AsyncReadRequest request) @@ -70,7 +71,13 @@ ReadInput Trading(FuturesReadRequest? context,bool risk=false)=>context is {Snap ReadInput Status()=>new(operation,body,null,["RunScenario"]); switch(body) { - case BuildRequest evaluate:return Market(evaluate.SnapshotId); + case BuildRequest evaluate: + if(evaluate.SourceScope is not null&&(evaluate.SourceScope!=AsyncReadProjections.CurveLineage||!evaluate.IncludeSources)) + throw Invalid(); + if(evaluate.DefinitionScope is not null&&(evaluate.DefinitionScope!=CurveDependencies|| + evaluate.SourceScope!=AsyncReadProjections.CurveLineage||evaluate.IncludeWorkspace)) + throw Invalid(); + return Market(evaluate.SnapshotId); case HistoryRequest history:return Market(history.SnapshotId); case CurveShapesRequest shapes:return Market(shapes.SnapshotId); case FuturesReadRequest directory:return Trading(directory); diff --git a/src/OpenEtrm.Api/AsyncReadProjections.cs b/src/OpenEtrm.Api/AsyncReadProjections.cs index 8123ef86..0526efa5 100644 --- a/src/OpenEtrm.Api/AsyncReadProjections.cs +++ b/src/OpenEtrm.Api/AsyncReadProjections.cs @@ -1,21 +1,26 @@ using System.Collections.Immutable; using System.Text.Json; +using System.Text.Json.Nodes; namespace OpenEtrm.Api; // HTTP and SSE share these projections; transport never changes financial terms. internal static class AsyncReadProjections { + internal const string CurveLineage="curve_lineage_v1"; internal static object Curve(PriceRead read,PinnedContext pin,BuildRequest body,object? workspace=null) { if(pin.Context is null&&!body.IncludeSources&&!body.IncludeWorkspace)return read.Result; - var json=JsonSerializer.SerializeToNode(read.Result,AsyncReadContract.Json)!.AsObject(); + var json=read.UsedObservationIds.HasValue&&read.Result is JsonObject projection?projection: + JsonSerializer.SerializeToNode(read.Result,AsyncReadContract.Json)!.AsObject(); if(pin.Context is {} view)json["context"]=JsonSerializer.SerializeToNode(view,AsyncReadContract.Json); if(body.IncludeSources) { json["observations"]=JsonSerializer.SerializeToNode(read.Observations); - json["observationScope"]="selected_frame"; - json["inputSelection"]=JsonSerializer.SerializeToNode(read.InputSelection,AsyncReadContract.Json); + var used=read.UsedObservationIds; + json["observationScope"]=used.HasValue?CurveLineage:"selected_frame"; + var selection=used is {} ids?read.InputSelection with{ObservationIds=ids}:read.InputSelection; + json["inputSelection"]=JsonSerializer.SerializeToNode(selection,AsyncReadContract.Json); } if(body.IncludeWorkspace)json["workspace"]=JsonSerializer.SerializeToNode(workspace??throw new InvalidOperationException("The requested workspace projection is missing."),AsyncReadContract.Json); return json; diff --git a/src/OpenEtrm.Api/AsyncReads.cs b/src/OpenEtrm.Api/AsyncReads.cs index 5d64bd0a..0f627177 100644 --- a/src/OpenEtrm.Api/AsyncReads.cs +++ b/src/OpenEtrm.Api/AsyncReads.cs @@ -177,10 +177,10 @@ await Authorize(owner,response.Plan.Input,delivery.Token,response.State=="comple } return Results.Empty; } - static async Task Write(HttpContext c,string? name,object? body,CancellationToken ct) + internal static async Task Write(HttpContext c,string? name,object? body,CancellationToken ct) { using var write=CancellationTokenSource.CreateLinkedTokenSource(ct);write.CancelAfter(TimeSpan.FromSeconds(5)); - await c.Response.Body.WriteAsync(JsonReadTransport.EncodeEvent(name,body),write.Token); + await JsonReadTransport.WriteEvent(c.Response.Body,name,body,write.Token); await c.Response.Body.FlushAsync(write.Token); } internal static bool Matches(string method,PathString path)=>HttpMethods.IsGet(method)&&path.Equals("/api/async-reads/events",StringComparison.OrdinalIgnoreCase); diff --git a/src/OpenEtrm.Api/JsonReadTransport.cs b/src/OpenEtrm.Api/JsonReadTransport.cs index 9a42ff33..35689049 100644 --- a/src/OpenEtrm.Api/JsonReadTransport.cs +++ b/src/OpenEtrm.Api/JsonReadTransport.cs @@ -1,10 +1,9 @@ -using System.Buffers; using System.Text; using System.Text.Json; namespace OpenEtrm.Api; -internal sealed record AsyncReadFrame(string Protocol,Guid ChannelId,Guid RequestId,string Operation,string State,Guid? SnapshotId,JsonElement? Result,AsyncReadError? Error); +internal sealed record AsyncReadFrame(string Protocol,Guid ChannelId,Guid RequestId,string Operation,string State,Guid? SnapshotId,ReadOnlyMemory? Result,AsyncReadError? Error); internal static class JsonReadTransport { @@ -21,25 +20,37 @@ internal static bool Compress(HttpContext context) return HttpMethods.IsGet(request.Method)&&(path.Equals("/async-reads/events",StringComparison.Ordinal)||path.Equals("/workspace",StringComparison.Ordinal)|| path.Equals("/workspace/catalog",StringComparison.Ordinal)||path.Equals("/workspace/navigation",StringComparison.Ordinal)); } - internal static ReadOnlyMemory EncodeEvent(string? name,object? body) + static ReadOnlyMemory EncodeEvent(string? name,object? body) { if(name is null)return ": ping\n\n"u8.ToArray(); - var buffer=new ArrayBufferWriter();buffer.Write(Encoding.UTF8.GetBytes($"event: {name}\ndata: ")); - using(var writer=new Utf8JsonWriter(buffer)) + return Encoding.UTF8.GetBytes($"event: {name}\ndata: {JsonSerializer.Serialize(body,JsonSerializerOptions.Web)}\n\n"); + } + internal static async Task WriteEvent(Stream stream,string? name,object? body,CancellationToken ct) + { + if(body is not AsyncReadFrame frame) + { + await stream.WriteAsync(EncodeEvent(name,body),ct);return; + } + if(name!="response")throw new InvalidOperationException("A read response requires its correlated response event."); + var header=JsonSerializer.SerializeToUtf8Bytes(new{protocol=frame.Protocol,channelId=frame.ChannelId,requestId=frame.RequestId, + operation=frame.Operation,state=frame.State,snapshotId=frame.SnapshotId},JsonSerializerOptions.Web); + await stream.WriteAsync("event: response\ndata: "u8.ToArray(),ct); + await stream.WriteAsync(header.AsMemory(0,header.Length-1),ct); + await stream.WriteAsync(",\"result\":"u8.ToArray(),ct); + await Chunks(frame.Result??"null"u8.ToArray()); + await stream.WriteAsync(",\"error\":"u8.ToArray(),ct); + await Chunks(JsonSerializer.SerializeToUtf8Bytes(frame.Error,JsonSerializerOptions.Web)); + await stream.WriteAsync("}\n\n"u8.ToArray(),ct); + + async Task Chunks(ReadOnlyMemory bytes) { - if(body is AsyncReadFrame frame) + const int chunk=16*1024; + while(!bytes.IsEmpty) { - writer.WriteStartObject();writer.WriteString("protocol",frame.Protocol);writer.WriteString("channelId",frame.ChannelId); - writer.WriteString("requestId",frame.RequestId);writer.WriteString("operation",frame.Operation);writer.WriteString("state",frame.State); - if(frame.SnapshotId is {} snapshot)writer.WriteString("snapshotId",snapshot);else writer.WriteNull("snapshotId"); - writer.WritePropertyName("result"); - // Completion owns compact validated JSON. Preserve it without walking - // and serializing the result document again at stream publication. - if(frame.Result is {} result)writer.WriteRawValue(result.GetRawText(),skipInputValidation:true);else writer.WriteNullValue(); - writer.WritePropertyName("error");JsonSerializer.Serialize(writer,frame.Error,JsonSerializerOptions.Web);writer.WriteEndObject(); + ct.ThrowIfCancellationRequested(); + var length=Math.Min(chunk,bytes.Length); + await stream.WriteAsync(bytes[..length],ct);bytes=bytes[length..]; } - else JsonSerializer.Serialize(writer,body,JsonSerializerOptions.Web); } - buffer.Write("\n\n"u8);return buffer.WrittenMemory; } } diff --git a/src/OpenEtrm.Api/MarketInputSelection.cs b/src/OpenEtrm.Api/MarketInputSelection.cs index 5656c260..ef630f8b 100644 --- a/src/OpenEtrm.Api/MarketInputSelection.cs +++ b/src/OpenEtrm.Api/MarketInputSelection.cs @@ -95,15 +95,33 @@ internal static async Task RootSource(PreviewDatabase database, return exact; } - internal static async Task<(PreviewData Data,InputFrame Frame)> ForCurve(PreviewDatabase database,NpgsqlConnection connection, - InputFrame frame,string curve,CancellationToken ct) + internal static async Task<(PreviewData Data,InputFrame Frame,CatalogueSource Root)> ForCurve( + PreviewDatabase database,NpgsqlConnection connection,InputFrame frame,string curve,CancellationToken ct,bool scoped=false) { - var root=await Root(database,connection,frame,ct); + CatalogueSource own; + PreviewData? root=null; + if(scoped) + { + own=await RootSource(database,connection,frame,ct); + var manifest=own.Apply(database.Manifest); + await MarketBundle.Require(connection,manifest,ct); + // Admit the original root pins before resolving the selected curve. + _=await PreviewDatabase.ReadObservationPins(connection,manifest,frame.ObservationIds,ct); + } + else + { + root=await Root(database,connection,frame,ct); + own=CatalogueSource.From(root); + } var catalogue=await new CatalogueStore(database.Manifest.CatalogId).ReadBound(connection,ct,frame.Time.Date); var rates=await SelectedRates(database,connection,frame,catalogue,ct); var source=SelectedCurve(frame,curve,catalogue,rates); - var own=new CatalogueSource(root.DefinitionRevision.RevisionId,root.ReferenceRevision.CatalogId,root.ReferenceRevision.RevisionId); - return await Resolve(database,connection,frame,source,[(root,frame with{Source=own})],false,ct); + if(scoped&&source==own) + root=await database.Read(connection,ct,ppaBundle:own,observationIds:frame.ObservationIds, + observationScope:ObservationReadScope.ExactPins,dependencyCurve:curve); + ImmutableArray<(PreviewData Data,InputFrame Frame)> supplied=root is null?[]:[(root,frame with{Source=own})]; + var selected=await Resolve(database,connection,frame,source,supplied,false,ct,dependencyCurve:scoped?curve:null); + return (selected.Data,selected.Frame,own); } // The dated catalogue owns generic rate choices; retained execution sources do @@ -222,7 +240,7 @@ internal static void Require((PreviewData Data,InputFrame Frame) source,InputFra internal static async Task<(PreviewData Data,InputFrame Frame)> Resolve(PreviewDatabase database,NpgsqlConnection connection, InputFrame frame,CatalogueSource exact,ImmutableArray<(PreviewData Data,InputFrame Frame)> supplied,bool requirePinned,CancellationToken ct,MarketInputReadScope? reads=null, - ReferenceReadScope referenceScope=ReferenceReadScope.DefinitionClosure) + ReferenceReadScope referenceScope=ReferenceReadScope.DefinitionClosure,string? dependencyCurve=null) { if(supplied.IsDefault)throw Invalid("Supply explicit input sources."); MarketSourceInputs? recorded=null; @@ -240,7 +258,7 @@ internal static void Require((PreviewData Data,InputFrame Frame) source,InputFra { if(retained.Source!=exact)throw Invalid("The recorded input source has different exact references."); var projection=retained.Project(frame); - var data=reads is null?await database.Read(connection,ct,ppaBundle:exact,observationIds:projection.ObservationIds,referenceScope:referenceScope,observationScope:ObservationReadScope.ExactPins): + var data=reads is null?await database.Read(connection,ct,ppaBundle:exact,observationIds:projection.ObservationIds,referenceScope:referenceScope,observationScope:ObservationReadScope.ExactPins,dependencyCurve:dependencyCurve): await reads.Read(projection,exact,projection.ObservationIds,referenceScope,ObservationReadScope.ExactPins,true,ct); found=[(data,projection)]; } diff --git a/src/OpenEtrm.Api/MarketReads.cs b/src/OpenEtrm.Api/MarketReads.cs index c6c2061d..25a3f2cf 100644 --- a/src/OpenEtrm.Api/MarketReads.cs +++ b/src/OpenEtrm.Api/MarketReads.cs @@ -1,6 +1,7 @@ using System.Collections.Immutable; using System.Globalization; using System.Text.Json; +using System.Text.Json.Nodes; using Npgsql; using OpenEtrm.MarketData; @@ -16,7 +17,10 @@ public ImmutableDictionary> References(bool c catalogOnly?Data.References.SetItem("calendar_days",[]):Data.References; } public sealed record PriceInputSelection(Guid SnapshotId,CatalogueSource Source,ImmutableArray ObservationIds,bool CorrectionsSupported); -public sealed record PriceRead(object Result,ImmutableArray Observations,PriceInputSelection InputSelection,WorkspaceRead? Workspace=null); +public sealed record PriceRead(object Result,ImmutableArray Observations,PriceInputSelection InputSelection,WorkspaceRead? Workspace=null) +{ + internal ImmutableArray? UsedObservationIds{get;init;} +} // KnownAt is the latest certified publication instant among the sample's observations; null while any is not yet certified. public sealed record HistorySample(DateTimeOffset At,string? Value,Guid SnapshotId,Guid[] ObservationIds,string? UnavailableReason,DateTimeOffset? KnownAt=null); // How a history read chooses the revision each sample shows (D237); the caller must name one. @@ -179,27 +183,39 @@ public async Task SearchReferences(NpgsqlConnection connection, } // The pinned frame and the data its own observations select, for the date of the pin's context. - async Task<(InputFrame Frame,PreviewData Data)> Pinned(NpgsqlConnection connection,PinnedContext pin,string curve,CancellationToken ct) + async Task<(InputFrame Frame,PreviewData Data,CatalogueSource Root)> Pinned(NpgsqlConnection connection,PinnedContext pin,string curve,bool scoped,CancellationToken ct) { var frame=pin.RequireFrame(); if(pin.Context is {} selected && PreviewSession.BusinessDay(frame)!=DateWorkspace.Date(selected.EffectiveOn)) throw new MarketDataException("market_date","Choose inputs for the selected context date. No earlier market date is substituted."); await pin.Bind(connection,database.Manifest.CatalogId,ct); - var inputs=await MarketInputSelection.ForCurve(database,connection,frame,curve,ct); - return (inputs.Frame,inputs.Data); + var inputs=await MarketInputSelection.ForCurve(database,connection,frame,curve,ct,scoped); + return (inputs.Frame,inputs.Data,inputs.Root); } - public async Task Evaluate(NpgsqlConnection connection,PinnedContext pin,string curve,CancellationToken ct,bool includeWorkspace=false,CurveDisplaySelection? display=null) + public async Task Evaluate(NpgsqlConnection connection,PinnedContext pin,string curve,CancellationToken ct,bool includeWorkspace=false,CurveDisplaySelection? display=null,bool curveLineage=false,bool scoped=false) { - var (frame,data)=await Pinned(connection,pin,curve,ct); + var (frame,data,primarySource)=await Pinned(connection,pin,curve,scoped,ct); if(display is not null)CurveDisplay.Require(display); var result=data.IsRateCurve(curve)?data.Result(frame,curve,await RateCurveResolver.Resolve(connection,data,frame,curve,ct)): CurveDisplay.Result(data,frame,curve,display??new(CurveDisplay.Version,CurveDisplay.Live)); + ImmutableArray? used=null; + if(curveLineage) + { + var projection=result as JsonObject??JsonSerializer.SerializeToNode(result,AsyncReadContract.Json)!.AsObject(); + var lineage=projection["points"]!.AsArray().SelectMany(point=>point!["lineage"]!.AsArray()) + .Select(id=>id!.GetValue()).ToHashSet(); + var selected=frame.ObservationIds.Where(lineage.Contains).ToImmutableArray(); + if(selected.Length!=lineage.Count)throw new InvalidOperationException("Curve evidence differs from the selected observation pins."); + if(projection.ContainsKey("observationIds"))projection["observationIds"]=JsonSerializer.SerializeToNode(selected,AsyncReadContract.Json); + used=selected; + result=projection; + } var observations=await PreviewDatabase.ReadObservationFacts(connection,data.DefinitionRevision.CatalogId, - data.DefinitionRevision.RevisionId,frame.ObservationIds,ct); + data.DefinitionRevision.RevisionId,used??frame.ObservationIds,ct); var source=new CatalogueSource(data.DefinitionRevision.RevisionId,data.ReferenceRevision.CatalogId,data.ReferenceRevision.RevisionId); var root=pin.RequireFrame(); - var primarySource=await MarketInputSelection.RootSource(database,connection,root,ct); + if(!scoped)primarySource=await MarketInputSelection.RootSource(database,connection,root,ct); var primary=root.ObservationIds.ToHashSet(); var legacy=(database.Manifest.InputMarkets??[]).Select(m=>(m.RevisionId,Pins:(database.InputMarketPins(root,m)??[]).ToHashSet())).ToArray(); var correctionsSupported=frame.ObservationIds.All(id=> @@ -208,7 +224,7 @@ public async Task Evaluate(NpgsqlConnection connection,PinnedContext var owners=legacy.Where(m=>m.Pins.Contains(id)).ToArray(); return owners.Length==1&&owners[0].RevisionId==source.DefinitionRevisionId; }); - return new(result,observations,new(frame.Id,source,frame.ObservationIds,correctionsSupported),includeWorkspace?await WorkspaceMetadata(connection,data,true,ct):null); + return new(result,observations,new(frame.Id,source,frame.ObservationIds,correctionsSupported),includeWorkspace?await WorkspaceMetadata(connection,data,true,ct):null){UsedObservationIds=used}; } public const int MaximumShapedCurves=64; diff --git a/src/OpenEtrm.Api/ObservationPinPayload.cs b/src/OpenEtrm.Api/ObservationPinPayload.cs new file mode 100644 index 00000000..e578c549 --- /dev/null +++ b/src/OpenEtrm.Api/ObservationPinPayload.cs @@ -0,0 +1,31 @@ +using System.Collections.Immutable; +using System.Text.Json; +using System.Text.Json.Serialization; +using OpenEtrm.Api.Presentation; + +namespace OpenEtrm.Api; + +// Decode the database's complete batch without retaining a second aggregate +// text/JSON document beside its individual observation rows. +internal sealed record ObservationPinPayload( + [property:JsonRequired] ImmutableArray Rows, + [property:JsonRequired] ImmutableHashSet Denied) +{ + static readonly JsonSerializerOptions Json=new(JsonSerializerDefaults.Web){Converters={new ObservationRowConverter()}}; + internal static async Task Read(Stream body,CancellationToken ct) + { + var payload=await JsonSerializer.DeserializeAsync(body,Json,ct); + if(payload is null||payload.Rows.IsDefault||payload.Denied is null)throw new JsonException("The complete observation pin batch is required."); + return payload; + } + sealed class ObservationRowConverter:JsonConverter + { + public override JsonElement Read(ref Utf8JsonReader reader,Type type,JsonSerializerOptions options) + { + using var row=JsonDocument.ParseValue(ref reader); + return ReferenceJson.PreserveNumericLexemes(row.RootElement.GetRawText()); + } + public override void Write(Utf8JsonWriter writer,JsonElement value,JsonSerializerOptions options)=> + throw new NotSupportedException("The observation pin payload is a database read contract."); + } +} diff --git a/src/OpenEtrm.Api/PreviewDatabase.CurveDependencies.cs b/src/OpenEtrm.Api/PreviewDatabase.CurveDependencies.cs new file mode 100644 index 00000000..7d493aba --- /dev/null +++ b/src/OpenEtrm.Api/PreviewDatabase.CurveDependencies.cs @@ -0,0 +1,110 @@ +using System.Collections.Immutable; +using System.Text.Json; +using Npgsql; +using OpenEtrm.Api.Presentation; + +namespace OpenEtrm.Api; + +public sealed partial class PreviewDatabase +{ + private static async Task>> ReadDefinitions( + NpgsqlConnection connection,PreviewManifest manifest,CancellationToken ct) + { + var definitions=new Dictionary>(); + foreach(var (table,key) in DefinitionTables) + { + var revisionColumn=table=="revisions"?"id":"revision_id"; + definitions[table]=await Rows(connection, + $"SELECT row_to_json(t)::text FROM (SELECT * FROM market_data.{table} WHERE catalog_id=$1 AND {revisionColumn}=$2 ORDER BY {key}) t", + [manifest.CatalogId,manifest.RevisionId],ct); + } + return definitions; + } + + private static async Task>> ReadCurveDefinitions( + NpgsqlConnection connection,PreviewManifest manifest,string curve,Guid[] pinnedSeries,CancellationToken ct) + { + // Retain every member of each reached curve so normal model validation + // still checks complete mappings, series, units and expiry rules. The + // frame's other exact pins also keep their series contracts for the + // existing full-snapshot validation and identity. + const string closure=""" + WITH RECURSIVE curve_codes(code) AS ( + SELECT $3::text + UNION + SELECT i.source_curve_code::text + FROM market_data.curve_inputs i JOIN curve_codes c ON c.code=i.curve_code + WHERE i.catalog_id=$1 AND i.revision_id=$2 AND i.source_curve_code IS NOT NULL + ), members AS MATERIALIZED ( + SELECT m.* FROM curve_codes c + JOIN market_data.curve_members m ON m.curve_code=c.code + WHERE m.catalog_id=$1 AND m.revision_id=$2 + ), pinned_series AS MATERIALIZED ( + SELECT id,contract_id FROM market_data.series + WHERE catalog_id=$1 AND revision_id=$2 AND id=ANY($4::uuid[]) + ), initial_contract_ids AS ( + SELECT contract_id FROM members + UNION SELECT contract_id FROM pinned_series WHERE contract_id IS NOT NULL + ), contract_ids(contract_id) AS ( + SELECT contract_id FROM initial_contract_ids + UNION + SELECT benchmark.id FROM contract_ids selected + JOIN market_data.contracts c ON c.catalog_id=$1 AND c.revision_id=$2 AND c.id=selected.contract_id + JOIN ref_data.futures_basis_settlements basis + ON (basis.catalog_id,basis.revision_id,basis.series_code)= + (c.reference_catalog_id,c.reference_revision_id,c.futures_series_code) + JOIN market_data.contracts benchmark + ON (benchmark.catalog_id,benchmark.revision_id,benchmark.futures_series_code)= + (c.catalog_id,c.revision_id,basis.benchmark_series_code) + ), inputs AS MATERIALIZED ( + SELECT i.* FROM curve_codes c + JOIN market_data.curve_inputs i ON i.curve_code=c.code + WHERE i.catalog_id=$1 AND i.revision_id=$2 + ), rate_tenors AS MATERIALIZED ( + SELECT * FROM market_data.rate_curve_tenors + WHERE catalog_id=$1 AND revision_id=$2 AND curve_code=$3 + ), series_ids AS MATERIALIZED ( + SELECT DISTINCT source_series_id AS id FROM inputs WHERE source_series_id IS NOT NULL + UNION SELECT series_id FROM rate_tenors + UNION SELECT id FROM pinned_series + ) + """; + const string pin="catalog_id=$1 AND revision_id=$2"; + var queries=new Dictionary + { + ["revisions"]="SELECT * FROM market_data.revisions WHERE catalog_id=$1 AND id=$2", + ["revision_periods"]=$"SELECT * FROM market_data.revision_periods WHERE {pin}", + ["curves"]=$"SELECT * FROM market_data.curves WHERE {pin} AND code IN (SELECT code FROM curve_codes)", + ["curve_members"]="SELECT * FROM members", + ["curve_inputs"]="SELECT * FROM inputs", + ["constituent_mappings"]=$"SELECT * FROM market_data.constituent_mappings WHERE {pin} AND curve_code IN (SELECT code FROM curve_codes)", + ["curve_expiry_rules"]=$"SELECT * FROM market_data.curve_expiry_rules WHERE {pin} AND curve_code IN (SELECT code FROM curve_codes)", + ["contracts"]=$"SELECT * FROM market_data.contracts WHERE {pin} AND id IN (SELECT contract_id FROM contract_ids)", + ["contract_intervals"]=$"SELECT * FROM market_data.contract_intervals WHERE {pin} AND contract_id IN (SELECT contract_id FROM contract_ids)", + ["series"]=$"SELECT * FROM market_data.series WHERE {pin} AND (contract_id IN (SELECT contract_id FROM contract_ids) OR id IN (SELECT id FROM series_ids))", + ["rate_curves"]=$"SELECT * FROM market_data.rate_curves WHERE {pin} AND code=$3", + ["rate_curve_tenors"]="SELECT * FROM rate_tenors" + }; + var statements=DefinitionTables.Where(t=>queries.ContainsKey(t.Table)).Select(t=>(t.Table,t.Key,queries[t.Table])); + var definitions=await ReadDefinitionGroups(connection,closure,statements,[manifest.CatalogId,manifest.RevisionId,curve,pinnedSeries],ct); + foreach(var (table,_) in DefinitionTables)definitions.TryAdd(table,[]); + return definitions; + } + + // Each branch selects native rows under the bound caller and existing RLS. + // Return rows separately so grouping never allocates an aggregate JSON value. + private static async Task>> ReadDefinitionGroups( + NpgsqlConnection connection,string prefix,IEnumerable<(string Table,string Key,string Query)> groups, + object[] parameters,CancellationToken ct) + { + var selected=groups.ToArray(); + var rows=selected.ToDictionary(g=>g.Table,_=>ImmutableArray.CreateBuilder()); + var sql=prefix+" "+string.Join(" UNION ALL ",selected.Select(g=> + $"(SELECT '{g.Table}' AS group_name,row_to_json(t)::text FROM ({g.Query} ORDER BY {g.Key}) t)")); + await using var command=ScheduleStore.Command(connection,sql,parameters); + await using var reader=await command.ExecuteReaderAsync(ct); + while(await reader.ReadAsync(ct)) + rows[reader.GetString(0)].Add(ReferenceJson.PreserveNumericLexemes(reader.GetString(1))); + return rows.ToDictionary(g=>g.Key,g=>g.Value.ToImmutable()); + } +} diff --git a/src/OpenEtrm.Api/PreviewDatabase.Observations.cs b/src/OpenEtrm.Api/PreviewDatabase.Observations.cs index c2225abf..76ad7ee3 100644 --- a/src/OpenEtrm.Api/PreviewDatabase.Observations.cs +++ b/src/OpenEtrm.Api/PreviewDatabase.Observations.cs @@ -19,14 +19,16 @@ internal async Task ReadObservationPinSets(NpgsqlConnection var selected=frames.Where(frame=>!frame.ObservationIds.IsEmpty).ToArray(); if(selected.Length==0)return new(metadata.WithObservations([]),[]); var requests=JsonSerializer.Serialize(selected.Select(frame=>new{id=frame.Id,pins=frame.ObservationIds})); - await using var command=ScheduleStore.Command(connection,"SELECT security.read_observation_pin_sets($1,$2,$3::jsonb)::text", + await using var command=ScheduleStore.Command(connection,"SELECT pg_catalog.convert_to(security.read_observation_pin_sets($1,$2,$3::jsonb)::text,'UTF8')", metadata.DefinitionRevision.CatalogId,metadata.DefinitionRevision.RevisionId,requests); - using var json=JsonDocument.Parse((string)(await command.ExecuteScalarAsync(ct))!); - var rows=json.RootElement.GetProperty("rows").EnumerateArray().Select(row=>ReferenceJson.PreserveNumericLexemes(row.GetRawText())).ToImmutableArray(); - Interlocked.Add(ref observationRowsRead,rows.Length); + await using var reader=await command.ExecuteReaderAsync(System.Data.CommandBehavior.SequentialAccess,ct); + if(!await reader.ReadAsync(ct))throw new JsonException("The observation pin batch is missing."); + await using var body=reader.GetStream(0); + var payload=await ObservationPinPayload.Read(body,ct); + Interlocked.Add(ref observationRowsRead,payload.Rows.Length); // Prefetch is not proof consumption. The calendar records only the // reached frame's returned pins when it uses this shared storage. - return new(metadata.WithObservations(rows),[..json.RootElement.GetProperty("denied").EnumerateArray().Select(id=>id.GetGuid())]); + return new(metadata.WithObservations(payload.Rows),payload.Denied); } internal async Task ReadObservationWindows(NpgsqlConnection connection,PreviewData metadata, ImmutableArray windows,CancellationToken ct,ImmutableArray exactPins=default) diff --git a/src/OpenEtrm.Api/PreviewDatabase.cs b/src/OpenEtrm.Api/PreviewDatabase.cs index 8d9ebe74..0acbdd23 100644 --- a/src/OpenEtrm.Api/PreviewDatabase.cs +++ b/src/OpenEtrm.Api/PreviewDatabase.cs @@ -383,7 +383,7 @@ internal async Task CaptureHistoryClock(OpenEtrm.MarketData.Risk.AlignedFa public async Task Read(NpgsqlConnection connection, CancellationToken ct,Guid? selectedRevision=null,PreviewData? sharedReferences=null,ImmutableArray? observationIds=null, ReferenceReadScope referenceScope=ReferenceReadScope.DefinitionClosure,bool includeCalendarDays=true, - ObservationReadScope observationScope=ObservationReadScope.FactHistory,CatalogueSource? ppaBundle=null) + ObservationReadScope observationScope=ObservationReadScope.FactHistory,CatalogueSource? ppaBundle=null,string? dependencyCurve=null) { // Logical-head resolution and exact retained reads are distinct. Native // visibility and exact FK coordinates authorize either; deployment-local @@ -393,12 +393,17 @@ public async Task Read(NpgsqlConnection connection, CancellationTok ppaBundle=(await new CatalogueStore(Manifest.CatalogId).ReadBound(connection,ct)).RequireRole("primary"); if(ppaBundle is {} p && (p.DefinitionRevisionId==Guid.Empty || p.ReferenceCatalogId==Guid.Empty || p.ReferenceRevisionId==Guid.Empty))throw new PreviewDeniedException(); var selectedManifest=ppaBundle is {} bundle?Manifest with {RevisionId=bundle.DefinitionRevisionId,ReferenceCatalogId=bundle.ReferenceCatalogId,ReferenceRevisionId=bundle.ReferenceRevisionId}:selectedRevision is {} id?Manifest with {RevisionId=id}:Manifest; - var definitions = new Dictionary>(); - foreach (var (table,key) in DefinitionTables) + ImmutableArray? exactScopedPins=null; + if(dependencyCurve is not null) { - var revisionColumn = table == "revisions" ? "id" : "revision_id"; - definitions[table] = await Rows(connection, $"SELECT row_to_json(t)::text FROM (SELECT * FROM market_data.{table} WHERE catalog_id=$1 AND {revisionColumn}=$2 ORDER BY {key}) t", [selectedManifest.CatalogId, selectedManifest.RevisionId], ct); + if(observationScope!=ObservationReadScope.ExactPins||observationIds is null||referenceScope!=ReferenceReadScope.DefinitionClosure) + throw new InvalidOperationException("A dependency read requires exact observation pins and reference closure."); + exactScopedPins=await ReadObservationPins(connection,selectedManifest,observationIds.Value,ct); } + var definitions=dependencyCurve is null + ? await ReadDefinitions(connection,selectedManifest,ct) + : await ReadCurveDefinitions(connection,selectedManifest,dependencyCurve, + [..exactScopedPins!.Value.Select(row=>row.Id("series_id")).Distinct()],ct); if (definitions["revisions"].Length != 1 || definitions["revisions"][0].Text("state") != "sealed") throw new PreviewDeniedException(); if(ppaBundle is null&&selectedRevision is not null) selectedManifest=PreviewData.ResolveBundleManifest(selectedManifest,definitions["revisions"][0]); @@ -422,8 +427,8 @@ async Task RequireSharedReferenceAccess() } await MarketBundle.RequireReferenceVisibility(connection,selectedManifest,ct); await RequireSharedReferenceAccess(); - references=await ReadReferences(connection,selectedManifest,definitions,referenceScope,includeCalendarDays,sharedReferences,ct); - var observations = await (observationScope switch + references=await ReadReferences(connection,selectedManifest,definitions,referenceScope,includeCalendarDays,sharedReferences,ct,groupCompleteTables:dependencyCurve is not null); + var observations = exactScopedPins??await (observationScope switch { ObservationReadScope.ExactPins => ReadObservationPins(connection,selectedManifest, observationIds??throw new ArgumentException("Exact observation reads require explicit pins.",nameof(observationIds)),ct), @@ -439,7 +444,7 @@ async Task RequireSharedReferenceAccess() private static async Task>> ReadReferences(NpgsqlConnection connection, PreviewManifest manifest,IReadOnlyDictionary> definitions, - ReferenceReadScope scope,bool includeCalendarDays,PreviewData? sharedReferences,CancellationToken ct) + ReferenceReadScope scope,bool includeCalendarDays,PreviewData? sharedReferences,CancellationToken ct,bool groupCompleteTables=false) { var loaded=new Dictionary>(); var bounded=new HashSet(["futures_series","futures_products","futures_milestones","futures_lot_resolutions", @@ -469,7 +474,14 @@ async Task> ReadTable(string table,string? column=nu foreach(var table in ReferenceTables)loaded[table]=await ReadTable(table); return loaded; } - foreach(var table in ReferenceTables.Where(t=>!bounded.Contains(t)))loaded[table]=await ReadTable(table); + var completeTables=ReferenceTables.Where(t=>!bounded.Contains(t)); + if(groupCompleteTables&&sharedReferences is null) + { + var statements=completeTables.Select(table=>(table,ReferenceKey(table), + $"SELECT * FROM ref_data.{table} WHERE catalog_id=$1 AND revision_id=$2")); + loaded=await ReadDefinitionGroups(connection,"",statements,[manifest.ReferenceCatalogId,manifest.ReferenceRevisionId],ct); + } + else foreach(var table in completeTables)loaded[table]=await ReadTable(table); // Follow only exact-version declared settlement links. These metadata // references do not introduce market dependencies or pricing inputs. diff --git a/src/OpenEtrm.Api/Program.cs b/src/OpenEtrm.Api/Program.cs index 7911aa24..fb4ef03f 100644 --- a/src/OpenEtrm.Api/Program.cs +++ b/src/OpenEtrm.Api/Program.cs @@ -1378,8 +1378,7 @@ await database.WithIdentity(Profile(session),"ReadMarketData",async(connection,c },c.RequestAborted); session.PauseLive(pin);return Results.Ok(new{context=pin.Context,frame=new NavigationFrame(pin.RequireFrame().Id,pin.RequireFrame().Name,pin.RequireFrame().At,pin.RequireFrame().BusinessDate)}); })); -object WorkspaceProjection(PreviewSession session,WorkspaceRead read,bool catalogOnly,PinnedContext? exact=null)=>CaptureWorkspaceProjection(session,exact)(read,catalogOnly); -Func CaptureWorkspaceProjection(PreviewSession session,PinnedContext? exact=null) +Func CaptureWorkspaceProjection(PreviewSession session,PinnedContext? exact=null,bool frameSummaries=false) { var context=exact is null?session.MarketContext:exact.Context; @@ -1391,18 +1390,23 @@ Func CaptureWorkspaceProjection(PreviewSession sessio var data=read.Data;var referenceSources=read.ReferenceSources;var references=read.References(catalogOnly); return new {referenceSources,unitCatalog=read.Units,unitCatalogError=read.Units is null?"unit_catalog":null,referenceDisplay=read.ReferenceDisplay,unitReferenceDisplay=read.UnitReferenceDisplay,context,profile,example=database.Manifest.Example,futuresConvention=database.Manifest.FuturesConvention,curveNames=read.CurveNames,powerCurveCodes=PowerPreview.CurveCodes(data),definitions=data.Definitions,references,referenceScope=catalogOnly?"definition_closure":"all_visible",observations=catalogOnly?ImmutableArray.Empty:data.Observations,observationScope=catalogOnly?"on_demand":"all_visible", // Each offered frame names its recorded close; one without a close is a working snapshot (D334). - frames,frameCatalog,initialFrameIds}; + frames=frameSummaries?(object)frames.Select(f=>new NavigationFrame(f.Id,f.Name,f.At,f.BusinessDate,f.OriginCloseId,f.Close)).ToArray():frames, + frameCatalog,initialFrameIds}; }; } Task Workspace(HttpContext c,bool catalogOnly)=>Locked(c,async session=> { + var detail=c.Request.Query["frameDetail"]; + var frameSummaries=catalogOnly&&detail.Count==1&&detail[0]=="summary"; + if(detail.Count!=0&&!frameSummaries)throw new AsyncReadException(422,"workspace_frame_detail","Choose the supported catalog frame detail."); + var read=await database.WithIdentity(Profile(session),"ReadMarketData",async(connection,ct)=> { var workspace=await marketReads.Workspace(connection,Pin(session,ContextVersions.SnapshotId(c.Request.Query)),catalogOnly,ct,c.Request.Query["curve"].FirstOrDefault()); return(workspace,plan:await DiscoverFrames(connection,session,ct)); },c.RequestAborted); session.ApplyFrames(read.plan); - return Results.Ok(WorkspaceProjection(session,read.workspace,catalogOnly)); + return Results.Ok(CaptureWorkspaceProjection(session,frameSummaries:frameSummaries)(read.workspace,catalogOnly)); }); api.MapGet("/workspace",(Func>)(c=>Workspace(c,false))); api.MapGet("/workspace/catalog",(Func>)(c=>Workspace(c,true))); @@ -2870,7 +2874,8 @@ Task AsReader(string permission,FuncAsReader("ReadMarketData",async(connection,t)=> { - var read=await marketReads.Evaluate(connection,pin,curve.Curve,t,curve.IncludeWorkspace,curve.ContractDisplay); + var read=await marketReads.Evaluate(connection,pin,curve.Curve,t,curve.IncludeWorkspace,curve.ContractDisplay,curve.SourceScope==AsyncReadProjections.CurveLineage, + curve.DefinitionScope==AsyncReadContract.CurveDependencies); var result=AsyncReadProjections.Curve(read,pin,curve,curve.IncludeWorkspace?workspace(read.Workspace??throw new InvalidOperationException("The requested workspace projection is missing."),true):null); var source=read.InputSelection.Source;var observations=read.InputSelection.ObservationIds; return new ReadOutcome(result,async(publication,cancellation)=> @@ -3074,7 +3079,13 @@ public sealed record ProfileRequest(string Profile); // A request whose session is missing, unknown or past its end. public sealed class PreviewSessionEnded : Exception; public sealed record LocalTimeRequest(string? LocalTime,string? TimeZone); -public sealed record BuildRequest(Guid SnapshotId,string Curve,bool IncludeSources=false,bool IncludeWorkspace=false,CurveDisplaySelection? ContractDisplay=null); +public sealed record BuildRequest(Guid SnapshotId,string Curve,bool IncludeSources=false,bool IncludeWorkspace=false,CurveDisplaySelection? ContractDisplay=null) +{ + [System.Text.Json.Serialization.JsonIgnore(Condition=System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull)] + public string? SourceScope{get;init;} + [System.Text.Json.Serialization.JsonIgnore(Condition=System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull)] + public string? DefinitionScope{get;init;} +} public sealed record HistoryRequest(Guid SnapshotId,string Curve,Guid? ContractId=null,Guid? SeriesId=null,string? RevisionPolicy=null,ImmutableArray? Overlays=null); [System.Text.Json.Serialization.JsonUnmappedMemberHandling(System.Text.Json.Serialization.JsonUnmappedMemberHandling.Disallow)] public sealed record CurveShapesRequest(Guid SnapshotId,ImmutableArray Curves,CurveDisplaySelection? ContractDisplay=null); diff --git a/tests/OpenEtrm.Api.Checks/AsyncReadBudgetChecks.cs b/tests/OpenEtrm.Api.Checks/AsyncReadBudgetChecks.cs new file mode 100644 index 00000000..50ddeb56 --- /dev/null +++ b/tests/OpenEtrm.Api.Checks/AsyncReadBudgetChecks.cs @@ -0,0 +1,84 @@ +using System.Text.Json; +using System.Text.Json.Nodes; +using OpenEtrm.Api; + +internal static class AsyncReadBudgetChecks +{ + internal static async Task Run() + { + await using var channel=new AsyncReadChannel(Guid.NewGuid(),CancellationToken.None);channel.Connect(); + var emitted=0;const int total=8192; + IEnumerable Rows() + { + var row=new string('x',8192); + for(var i=0;iTask.FromResult(new ReadOutcome(new{rows=Rows()},(_,_)=>Task.CompletedTask))); + channel.Submit(request,plan); + using var timeout=new CancellationTokenSource(TimeSpan.FromSeconds(15)); + var response=await channel.Read(timeout.Token); + if(response.State!="failed"||response.Error?.Code!="async_result_size"||response.Result is not null) + throw new Exception("An oversized read must retain its structured refusal and disclose no partial result."); + if(emitted>=total)throw new Exception($"The response budget must stop serializing an oversized read, rather than expanding all {emitted} rows first."); + Console.WriteLine($"PASS async response budget: expansion stops at {emitted} of {total} synthetic rows with the existing structured refusal"); + await LargeDom(); + await LargeToken(); + await Cancellation(); + } + static async Task LargeDom() + { + var rows=new JsonArray();var row=new string('x',8192); + for(var i=0;i<8192;i++)rows.Add(row); + var value=new JsonObject{["rows"]=rows}; + await Check(value); + using var document=JsonDocument.Parse(JsonSerializer.SerializeToUtf8Bytes(value)); + await Check(document.RootElement); + static async Task Check(object value) + { + await using var channel=new AsyncReadChannel(Guid.NewGuid(),CancellationToken.None);channel.Connect(); + var request=new AsyncReadRequest(channel.Id,Guid.NewGuid(),"curve.evaluate.v1",JsonSerializer.SerializeToElement(new{snapshotId=Guid.NewGuid(),curve="synthetic_power"})); + var plan=new ReadPlan(AsyncReadContract.Decode(request),_=>Task.FromResult(new ReadOutcome(value,(_,_)=>Task.CompletedTask))); + var before=GC.GetTotalAllocatedBytes(true);channel.Submit(request,plan); + using var timeout=new CancellationTokenSource(TimeSpan.FromSeconds(15));var response=await channel.Read(timeout.Token); + var allocated=GC.GetTotalAllocatedBytes(true)-before; + if(response.Error?.Code!="async_result_size"||response.Result is not null||allocated>80L*1024*1024) + throw new Exception($"An oversized {value.GetType().Name} must stop before buffering its complete encoded payload: {allocated} bytes."); + Console.WriteLine($"PASS oversized {value.GetType().Name}: bounded serialization allocation {allocated} bytes"); + } + } + static async Task LargeToken() + { + await using var channel=new AsyncReadChannel(Guid.NewGuid(),CancellationToken.None);channel.Connect(); + var value=new string('x',AsyncReadChannel.MaximumResultBytes*2); + var request=new AsyncReadRequest(channel.Id,Guid.NewGuid(),"curve.evaluate.v1",JsonSerializer.SerializeToElement(new{snapshotId=Guid.NewGuid(),curve="synthetic_power"})); + var plan=new ReadPlan(AsyncReadContract.Decode(request),_=>Task.FromResult(new ReadOutcome(value,(_,_)=>Task.CompletedTask))); + var before=GC.GetTotalAllocatedBytes(true);channel.Submit(request,plan); + using var timeout=new CancellationTokenSource(TimeSpan.FromSeconds(10));var response=await channel.Read(timeout.Token); + var allocated=GC.GetTotalAllocatedBytes(true)-before; + if(response.Error?.Code!="async_result_size"||response.Result is not null||allocated>8*1024*1024) + throw new Exception($"A definitely oversized string must refuse before allocating its encoded token: {allocated} bytes."); + Console.WriteLine($"PASS oversized token: refused without encoded copy; {allocated} allocated bytes"); + } + static async Task Cancellation() + { + await using var channel=new AsyncReadChannel(Guid.NewGuid(),CancellationToken.None);channel.Connect(); + var request=new AsyncReadRequest(channel.Id,Guid.NewGuid(),"curve.evaluate.v1",JsonSerializer.SerializeToElement(new{snapshotId=Guid.NewGuid(),curve="synthetic_power"})); + var emitted=0; + IEnumerable Rows() + { + for(var i=0;i<4096;i++) + { + emitted++;if(emitted==50)channel.Cancel(request.RequestId); + yield return new string('x',512); + } + } + channel.Submit(request,new(AsyncReadContract.Decode(request),_=>Task.FromResult(new ReadOutcome(new{rows=Rows()},(_,_)=>Task.CompletedTask)))); + using var timeout=new CancellationTokenSource(TimeSpan.FromSeconds(10));var response=await channel.Read(timeout.Token); + if(response.State!="cancelled"||response.Error?.Code!="request_cancelled"||response.Result is not null||emitted==4096) + throw new Exception("Cancellation during serialization must stop expansion and publish no partial result."); + Console.WriteLine($"PASS serialization cancellation: stopped at {emitted} synthetic rows"); + } + +} diff --git a/tests/OpenEtrm.Api.Checks/AsyncReadChecks.cs b/tests/OpenEtrm.Api.Checks/AsyncReadChecks.cs index 3911abab..8d64900b 100644 --- a/tests/OpenEtrm.Api.Checks/AsyncReadChecks.cs +++ b/tests/OpenEtrm.Api.Checks/AsyncReadChecks.cs @@ -33,6 +33,7 @@ internal static async Task Run() drain.SetResult();await Task.WhenAll(firstDisposal,secondDisposal); if(!channel.Stopping.IsCancellationRequested)throw new Exception("Retired channels retain a safe cancelled lifetime token."); Refused("async_channel_unavailable",()=>channel.Submit(request,Plan(request,Work))); + await AsyncReadBudgetChecks.Run(); await CancellationAdmission(); await DetachedRequestContext(); DelegatedTransport(); @@ -92,12 +93,14 @@ static async Task DetachedRequestContext() if(!ReferenceEquals(accessor.HttpContext,submitting)||ambient.Value!="synthetic submission") throw new Exception("Scheduling a detached read must preserve the submitting request's own context."); using var timeout=new CancellationTokenSource(TimeSpan.FromSeconds(3));var completed=await channel.Read(timeout.Token); - if(completed.State!="completed"||completed.Result is not {} result||result.GetProperty("requestContext").GetBoolean()||result.GetProperty("ambient").ValueKind!=JsonValueKind.Null) + using var result=JsonDocument.Parse(completed.Result??throw new Exception("The completed result is missing.")); + if(completed.State!="completed"||result.RootElement.GetProperty("requestContext").GetBoolean()||result.RootElement.GetProperty("ambient").ValueKind!=JsonValueKind.Null) throw new Exception("A detached read must not inherit the submitting HTTP context or its ambient execution state."); var alreadyDetached=request with{RequestId=Guid.NewGuid()}; using(ExecutionContext.SuppressFlow())channel.Submit(alreadyDetached,plan); var repeated=await channel.Read(timeout.Token); - if(repeated.State!="completed"||repeated.Result!.Value.GetProperty("requestContext").GetBoolean()) + using var repeatedResult=JsonDocument.Parse(repeated.Result??throw new Exception("The completed retry is missing.")); + if(repeated.State!="completed"||repeatedResult.RootElement.GetProperty("requestContext").GetBoolean()) throw new Exception("An already detached caller must still admit and complete its explicit read."); } finally{accessor.HttpContext=null;} diff --git a/tests/OpenEtrm.Api.Checks/AsyncReadStreamChecks.cs b/tests/OpenEtrm.Api.Checks/AsyncReadStreamChecks.cs new file mode 100644 index 00000000..078c634f --- /dev/null +++ b/tests/OpenEtrm.Api.Checks/AsyncReadStreamChecks.cs @@ -0,0 +1,126 @@ +using System.Text; +using System.Text.Json; +using Microsoft.AspNetCore.Http; +using OpenEtrm.Api; + +internal static class AsyncReadStreamChecks +{ + internal static async Task Run() + { + var id=Guid.NewGuid();var snapshot=Guid.NewGuid(); + var request=new AsyncReadRequest(id,Guid.NewGuid(),"curve.evaluate.v1",JsonSerializer.SerializeToElement(new{snapshotId=snapshot,curve="synthetic"})); + var plan=new ReadPlan(AsyncReadContract.Decode(request),_=>throw new InvalidOperationException("Transport must not reevaluate a completed result.")); + var payload=JsonSerializer.SerializeToElement(new{points=new string('x',4*1024*1024),label="synthetic λ \"\\\n\r",missing=(string?)null}); + var completed=new AsyncReadCompletion(request,plan,"completed",JsonSerializer.SerializeToUtf8Bytes(payload,AsyncReadContract.Json),null); + using var sink=new BoundedSink(5*1024*1024);var context=new DefaultHttpContext();context.Response.Body=sink; + var before=GC.GetTotalAllocatedBytes(true); + await Write(context,id,completed,CancellationToken.None); + var allocated=GC.GetTotalAllocatedBytes(true)-before; + if(allocated>1024*1024)throw new Exception($"A completed UTF8 result must not allocate another whole response while streaming: {allocated} bytes."); + var bytes=sink.GetBuffer().AsSpan(0,(int)sink.Length); + var prefix="event: response\ndata: "u8.ToArray(); + if(!bytes.StartsWith(prefix)||!bytes.EndsWith("\n\n"u8))throw new Exception("A response must be one complete SSE event."); + using(var document=JsonDocument.Parse(sink.GetBuffer().AsMemory(prefix.Length,(int)sink.Length-prefix.Length-2))) + { + var wire=document.RootElement; + if(wire.GetProperty("protocol").GetString()!=AsyncReadChannel.Protocol||wire.GetProperty("channelId").GetGuid()!=id|| + wire.GetProperty("requestId").GetGuid()!=request.RequestId||wire.GetProperty("snapshotId").GetGuid()!=snapshot|| + wire.GetProperty("operation").GetString()!=request.Operation||wire.GetProperty("state").GetString()!="completed"|| + wire.GetProperty("error").ValueKind!=JsonValueKind.Null||!JsonElement.DeepEquals(wire.GetProperty("result"),payload)) + throw new Exception("Streaming must preserve the complete correlated JSON value, escaping and null fields."); + } + using var cancellation=new CancellationTokenSource();using var partial=new BoundedSink(5*1024*1024,cancellation); + context=new DefaultHttpContext();context.Response.Body=partial; + try{await Write(context,id,completed,cancellation.Token);throw new Exception("A cancelled response must stop writing.");} + catch(OperationCanceledException)when(cancellation.IsCancellationRequested){} + if(partial.Length!=partial.CancelledAt||partial.GetBuffer().AsSpan(0,(int)partial.Length).EndsWith("\n\n"u8)) + throw new Exception("Cancellation must neither write more bytes nor terminate a partial event as successful."); + using var errorSink=new BoundedSink(4096);context=new DefaultHttpContext();context.Response.Body=errorSink; + var expectedError=new AsyncReadError("synthetic_refusal","synthetic λ \"\\\n"); + var failed=completed with{State="failed",Result=null,Error=expectedError}; + await Write(context,id,failed,CancellationToken.None); + var text=Encoding.UTF8.GetString(errorSink.GetBuffer(),0,(int)errorSink.Length); + using(var document=JsonDocument.Parse(text[prefix.Length..^2])) + { + var wire=document.RootElement; + if(wire.GetProperty("result").ValueKind!=JsonValueKind.Null||wire.GetProperty("error").GetProperty("code").GetString()!=expectedError.Code|| + wire.GetProperty("error").GetProperty("message").GetString()!=expectedError.Message) + throw new Exception("Refusals must retain their structured error and null result."); + } + using var slow=new SlowSink();context=new DefaultHttpContext();context.Response.Body=slow; + using var outer=new CancellationTokenSource(TimeSpan.FromSeconds(15)); + try{await Write(context,id,failed,outer.Token);throw new Exception("A stalled event must have a bounded write lifetime.");} + catch(OperationCanceledException)when(!outer.IsCancellationRequested){} + Console.WriteLine($"PASS async UTF8 response: 4MiB synthetic value, bounded writes, exact JSON, cancellation and one event deadline; streaming allocation {allocated} bytes"); + await ResultBudget(); + await NestingBudget(); + } + static Task Write(HttpContext context,Guid id,AsyncReadCompletion response,CancellationToken ct)=> + AsyncReads.Write(context,"response",new AsyncReadFrame(AsyncReadChannel.Protocol,id,response.Request.RequestId, + response.Request.Operation,response.State,response.Plan.Input.SnapshotId,response.Result,response.Error),ct); + static async Task NestingBudget() + { + await using var channel=new AsyncReadChannel(Guid.NewGuid(),CancellationToken.None);channel.Connect(); + foreach(var depth in new[]{64,65}) + { + using var value=JsonDocument.Parse(new string('[',depth)+"0"+new string(']',depth),new JsonDocumentOptions{MaxDepth=65}); + var request=new AsyncReadRequest(channel.Id,Guid.NewGuid(),"curve.evaluate.v1",JsonSerializer.SerializeToElement(new{snapshotId=Guid.NewGuid(),curve="synthetic"})); + var plan=new ReadPlan(AsyncReadContract.Decode(request),_=>Task.FromResult(new ReadOutcome(value.RootElement,(_,_)=>Task.CompletedTask))); + channel.Submit(request,plan); + using var timeout=new CancellationTokenSource(TimeSpan.FromSeconds(3));var response=await channel.Read(timeout.Token); + if(depth==64) + { + if(response.State!="completed")throw new Exception("The existing 64-level result contract must complete."); + using var sink=new BoundedSink(4096);var context=new DefaultHttpContext();context.Response.Body=sink; + await Write(context,channel.Id,response,CancellationToken.None); + var prefix="event: response\ndata: "u8.Length; + using var wire=JsonDocument.Parse(sink.GetBuffer().AsMemory(prefix,(int)sink.Length-prefix-2),new JsonDocumentOptions{MaxDepth=65}); + if(!JsonElement.DeepEquals(wire.RootElement.GetProperty("result"),value.RootElement)) + throw new Exception("The original 64-level result must survive its extra envelope level unchanged."); + } + else if(response.State!="failed"||response.Result is not null||response.Error?.Code!="async_read_failed") + throw new Exception("A result beyond the existing 64-level result limit must fail before response publication."); + } + Console.WriteLine("PASS async result nesting: existing 64-level result bound and extra envelope level preserved"); + } + static async Task ResultBudget() + { + await using var channel=new AsyncReadChannel(Guid.NewGuid(),CancellationToken.None);channel.Connect(); + foreach(var extra in new[]{-1,0,1}) + { + // The escaped non-ASCII character occupies six serialized bytes, + // so a character count cannot accidentally implement this limit. + var value=new string('x',AsyncReadChannel.MaximumResultBytes-2-6+extra)+"λ"; + var request=new AsyncReadRequest(channel.Id,Guid.NewGuid(),"curve.evaluate.v1",JsonSerializer.SerializeToElement(new{snapshotId=Guid.NewGuid(),curve="synthetic"})); + channel.Submit(request,new(AsyncReadContract.Decode(request),_=>Task.FromResult(new ReadOutcome(value,(_,_)=>Task.CompletedTask)))); + using var timeout=new CancellationTokenSource(TimeSpan.FromSeconds(10));var response=await channel.Read(timeout.Token); + if(extra<=0) + { + if(response.State!="completed"||response.Result?.Length!=AsyncReadChannel.MaximumResultBytes+extra) + throw new Exception("Valid near-limit results must retain their exact serialized UTF8 bytes."); + } + else if(response.State!="failed"||response.Error?.Code!="async_result_size"||response.Result is not null) + throw new Exception("An over-limit result must retain its structured refusal without sending its payload."); + } + Console.WriteLine("PASS async result limit: exact serialized UTF8 byte boundary and structured oversize refusal"); + } + sealed class BoundedSink(int capacity,CancellationTokenSource? cancel=null):MemoryStream(capacity) + { + internal long CancelledAt; + public override ValueTask WriteAsync(ReadOnlyMemory buffer,CancellationToken ct=default) + { + ct.ThrowIfCancellationRequested(); + if(buffer.Length>64*1024)throw new Exception($"Response write exceeded 64KiB: {buffer.Length} bytes."); + base.Write(buffer.Span); + if(cancel is not null&&Length>=64*1024){CancelledAt=Length;cancel.Cancel();} + return ValueTask.CompletedTask; + } + public override Task WriteAsync(byte[] buffer,int offset,int count,CancellationToken ct)=>WriteAsync(buffer.AsMemory(offset,count),ct).AsTask(); + } + sealed class SlowSink:MemoryStream + { + public override async ValueTask WriteAsync(ReadOnlyMemory buffer,CancellationToken ct=default) + {await Task.Delay(TimeSpan.FromSeconds(2),ct);base.Write(buffer.Span);} + public override Task WriteAsync(byte[] buffer,int offset,int count,CancellationToken ct)=>WriteAsync(buffer.AsMemory(offset,count),ct).AsTask(); + } +} diff --git a/tests/OpenEtrm.Api.Checks/JsonReadTransportChecks.cs b/tests/OpenEtrm.Api.Checks/JsonReadTransportChecks.cs index 556e0834..7fa03f95 100644 --- a/tests/OpenEtrm.Api.Checks/JsonReadTransportChecks.cs +++ b/tests/OpenEtrm.Api.Checks/JsonReadTransportChecks.cs @@ -27,14 +27,16 @@ bool Compressed(string method,string path) foreach(var state in new[]{"completed","failed","cancelled"}) { var frame=new AsyncReadFrame(AsyncReadChannel.Protocol,channel,request,"curve.evaluate.v1",state,state=="completed"?channel:null, - state=="completed"?result.RootElement:null,state=="completed"?null:new("synthetic_refusal","Synthetic input refused.")); - var encoded=Encoding.UTF8.GetString(JsonReadTransport.EncodeEvent("response",frame).Span); + state=="completed"?(ReadOnlyMemory?)JsonSerializer.SerializeToUtf8Bytes(result.RootElement):null,state=="completed"?null:new("synthetic_refusal","Synthetic input refused.")); + using var output=new MemoryStream();await JsonReadTransport.WriteEvent(output,"response",frame,CancellationToken.None); + var encoded=Encoding.UTF8.GetString(output.ToArray()); if(!encoded.StartsWith("event: response\ndata: ",StringComparison.Ordinal)||!encoded.EndsWith("\n\n",StringComparison.Ordinal))throw new Exception("SSE framing changed."); using var actual=JsonDocument.Parse(encoded["event: response\ndata: ".Length..^2]); - var expected=JsonSerializer.SerializeToElement(new{protocol=frame.Protocol,channelId=channel,requestId=request,operation=frame.Operation,state,snapshotId=frame.SnapshotId,result=frame.Result,error=frame.Error},JsonSerializerOptions.Web); + var expected=JsonSerializer.SerializeToElement(new{protocol=frame.Protocol,channelId=channel,requestId=request,operation=frame.Operation,state,snapshotId=frame.SnapshotId,result=state=="completed"?(JsonElement?)result.RootElement:null,error=frame.Error},JsonSerializerOptions.Web); if(!JsonElement.DeepEquals(expected,actual.RootElement))throw new Exception("Raw JSON changed a response value, identity or null/missing distinction."); } - if(Encoding.UTF8.GetString(JsonReadTransport.EncodeEvent(null,null).Span)!=": ping\n\n")throw new Exception("Heartbeat framing changed."); + using var heartbeat=new MemoryStream();await JsonReadTransport.WriteEvent(heartbeat,null,null,CancellationToken.None); + if(Encoding.UTF8.GetString(heartbeat.ToArray())!=": ping\n\n")throw new Exception("Heartbeat framing changed."); await Streaming(); Console.WriteLine("PASS JSON read transport: scoped compression, exact response parity, missing/null/empty, SSE framing and negotiated gzip flushing"); } @@ -49,10 +51,10 @@ static async Task Streaming() { var gate=release;context.Response.ContentType="text/event-stream"; context.Features.Get()?.DisableBuffering();await context.Response.StartAsync(context.RequestAborted); - await context.Response.Body.WriteAsync(JsonReadTransport.EncodeEvent("ready",new{protocol=AsyncReadChannel.Protocol}),context.RequestAborted); + await JsonReadTransport.WriteEvent(context.Response.Body,"ready",new{protocol=AsyncReadChannel.Protocol},context.RequestAborted); await context.Response.Body.FlushAsync(context.RequestAborted); await gate.Task.WaitAsync(context.RequestAborted); - await context.Response.Body.WriteAsync(JsonReadTransport.EncodeEvent(null,null),context.RequestAborted); + await JsonReadTransport.WriteEvent(context.Response.Body,null,null,context.RequestAborted); await context.Response.Body.FlushAsync(context.RequestAborted); }); app.MapGet("/api/bootstrap",()=>Results.Json(new{csrfToken="synthetic-token"})); diff --git a/tests/OpenEtrm.Api.Checks/LargeCatalogReadChecks.cs b/tests/OpenEtrm.Api.Checks/LargeCatalogReadChecks.cs new file mode 100644 index 00000000..0635385e --- /dev/null +++ b/tests/OpenEtrm.Api.Checks/LargeCatalogReadChecks.cs @@ -0,0 +1,84 @@ +using System.Collections.Immutable; +using System.Diagnostics; +using System.IO.Compression; +using System.Net; +using System.Text.Json; +using Microsoft.AspNetCore.Builder; +using Microsoft.AspNetCore.Hosting; +using Microsoft.AspNetCore.Http; +using Microsoft.AspNetCore.Http.Features; +using Microsoft.AspNetCore.ResponseCompression; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using OpenEtrm.Api; + +// An opt-in managed HTTP workload, separate from browser and database qualification. +internal static class LargeCatalogReadChecks +{ + internal static async Task Run() + { + const int contracts=20000; + var fixture=PreviewDataChecks.Fixture(contracts); + var observations=fixture.Observations.Select(row=>JsonSerializer.SerializeToElement(row.EnumerateObject().ToDictionary(property=>property.Name, + property=>property.Name=="value"?JsonSerializer.SerializeToElement("100.000"):property.Value))).ToImmutableArray(); + var data=new PreviewData(fixture.Manifest,fixture.Definitions,fixture.References,observations); + var frame=fixture.Frame with{CurveContracts=ImmutableDictionary>.Empty.Add("future",[fixture.Frame.CurveContracts["future"][0]])}; + var source=new CatalogueSource(data.DefinitionRevision.RevisionId,data.ReferenceRevision.CatalogId,data.ReferenceRevision.RevisionId); + var input=new PriceInputSelection(frame.Id,source,frame.ObservationIds,false); + object Price()=>AsyncReadProjections.Curve(new(CurveDisplay.Result(data,frame,"future",new(CurveDisplay.Version,CurveDisplay.Live)),data.Observations,input), + PinnedContext.Current(frame),new(frame.Id,"future",IncludeSources:true)); + var catalog=new{definitions=data.Definitions,references=data.References,frames=new[]{frame},observations=Array.Empty(),observationScope="on_demand"}; + using var expected=JsonDocument.Parse(JsonSerializer.SerializeToUtf8Bytes(Price(),AsyncReadContract.Json)); + var builder=WebApplication.CreateBuilder();builder.Logging.ClearProviders();builder.WebHost.UseUrls("http://127.0.0.1:0"); + builder.Services.AddResponseCompression(options=>{options.Providers.Add();options.MimeTypes=["application/json","text/event-stream"];}); + builder.Services.Configure(options=>options.Level=CompressionLevel.Fastest); + await using var app=builder.Build();app.UseWhen(JsonReadTransport.Compress,branch=>branch.UseResponseCompression()); + app.MapGet("/api/workspace/catalog",()=>Results.Json(catalog,AsyncReadContract.Json)); + Func events=async context=> + { + await using var channel=new AsyncReadChannel(Guid.NewGuid(),context.RequestAborted);channel.Connect(); + var request=new AsyncReadRequest(channel.Id,Guid.NewGuid(),"curve.evaluate.v1",JsonSerializer.SerializeToElement(new{snapshotId=frame.Id,curve="future",includeSources=true})); + channel.Submit(request,new ReadPlan(AsyncReadContract.Decode(request),(CancellationToken _)=>Task.FromResult(new ReadOutcome(Price(),(_,_)=>Task.CompletedTask)))); + var response=await channel.Read(context.RequestAborted); + context.Response.ContentType="text/event-stream";context.Features.Get()?.DisableBuffering(); + await AsyncReads.Write(context,"response",new AsyncReadFrame(AsyncReadChannel.Protocol,channel.Id,request.RequestId,request.Operation, + response.State,frame.Id,response.Result,response.Error),context.RequestAborted); + }; + app.MapGet("/api/async-reads/events",(Delegate)events); + await app.StartAsync(); + try + { + using var client=new HttpClient(new HttpClientHandler{AutomaticDecompression=DecompressionMethods.None}){BaseAddress=new Uri(app.Urls.Single()),Timeout=TimeSpan.FromSeconds(60)}; + client.DefaultRequestHeaders.AcceptEncoding.ParseAdd("gzip"); + var timer=Stopwatch.StartNew(); + using var response=await client.GetAsync("/api/workspace/catalog");response.EnsureSuccessStatusCode(); + if(!response.Content.Headers.ContentEncoding.SequenceEqual(["gzip"]))throw new Exception("The large catalog must negotiate HTTP gzip."); + var catalogWire=await response.Content.ReadAsByteArrayAsync(); + using var zipped=new GZipStream(new MemoryStream(catalogWire),CompressionMode.Decompress); + using var decoded=new MemoryStream();await zipped.CopyToAsync(decoded);decoded.Position=0; + using var complete=await JsonDocument.ParseAsync(decoded); + if(decoded.Length<=AsyncReadChannel.MaximumResultBytes||complete.RootElement.GetProperty("definitions").GetProperty("contracts").GetArrayLength()!=contracts|| + complete.RootElement.GetProperty("references").GetProperty("futures_series").GetArrayLength()!=contracts) + throw new Exception("The complete catalog must exceed 16MiB and retain every synthetic contract and reference."); + var catalogMilliseconds=timer.Elapsed.TotalMilliseconds; + var reads=new List(); + for(var i=0;i<3;i++) + { + timer.Restart();using var eventResponse=await client.GetAsync("/api/async-reads/events");eventResponse.EnsureSuccessStatusCode(); + if(!eventResponse.Content.Headers.ContentEncoding.SequenceEqual(["gzip"]))throw new Exception("The price event must negotiate HTTP gzip."); + var wire=await eventResponse.Content.ReadAsByteArrayAsync();using var compressed=new GZipStream(new MemoryStream(wire),CompressionMode.Decompress); + using var reader=new StreamReader(compressed);var text=await reader.ReadToEndAsync(); + if(!text.StartsWith("event: response\ndata: ",StringComparison.Ordinal)||!text.EndsWith("\n\n",StringComparison.Ordinal))throw new Exception("The price frame was truncated."); + using var terminal=JsonDocument.Parse(text["event: response\ndata: ".Length..^2]); + var result=terminal.RootElement.GetProperty("result"); + if(terminal.RootElement.GetProperty("state").GetString()!="completed"||terminal.RootElement.GetProperty("snapshotId").GetGuid()!=frame.Id|| + !JsonElement.DeepEquals(expected.RootElement,result)||result.TryGetProperty("workspace",out _))throw new Exception("Separate hydration must preserve the exact complete price projection without embedding the catalog."); + reads.Add(new{milliseconds=timer.Elapsed.TotalMilliseconds,compressedBodyBytes=wire.Length}); + } + Console.WriteLine("LARGE_CATALOG="+JsonSerializer.Serialize(new{contracts,selectedContracts=1,observations=data.Observations.Length, + catalogBytes=decoded.Length,catalogCompressedBodyBytes=catalogWire.Length,catalogMilliseconds,reads, + heapLimit=GC.GetGCMemoryInfo().TotalAvailableMemoryBytes,runtime=Environment.Version.ToString(),platform=System.Runtime.InteropServices.RuntimeInformation.OSDescription})); + } + finally{await app.StopAsync();} + } +} diff --git a/tests/OpenEtrm.Api.Checks/ObservationPinPayloadChecks.cs b/tests/OpenEtrm.Api.Checks/ObservationPinPayloadChecks.cs new file mode 100644 index 00000000..7dd4bd9b --- /dev/null +++ b/tests/OpenEtrm.Api.Checks/ObservationPinPayloadChecks.cs @@ -0,0 +1,39 @@ +using System.Text; +using System.Text.Json; +using OpenEtrm.Api; + +internal static class ObservationPinPayloadChecks +{ + internal static async Task Run(string? path=null,int? expectedRows=null) + { + if(path is not null) + { + await using var source=File.OpenRead(path); + var result=await ObservationPinPayload.Read(source,CancellationToken.None); + if(result.Rows.Length!=expectedRows||result.Rows[0].GetProperty("value").GetString()!="-0.00000000000000000100"||result.Denied.Count!=1) + throw new Exception("The complete synthetic batch and exact numeric text must survive streaming."); + var fixture=PreviewDataChecks.Fixture(1);var first=result.Rows[0]; + var catalog=first.GetProperty("catalog_id").GetGuid();var revision=first.GetProperty("definition_revision_id").GetGuid(); + fixture.Definitions["revisions"]=[JsonSerializer.SerializeToElement(new{catalog_id=catalog,id=revision,reference_catalog_id=fixture.Manifest.ReferenceCatalogId,reference_revision_id=fixture.Manifest.ReferenceRevisionId,state="sealed"})]; + fixture.Definitions["series"]=[JsonSerializer.SerializeToElement(new{id=first.GetProperty("series_id").GetGuid(),contract_id=fixture.Definitions["contracts"][0].GetProperty("id").GetGuid(),quote_code="price",value_specification_code=(string?)null})]; + var metadata=new PreviewData(fixture.Manifest with{CatalogId=catalog,RevisionId=revision},fixture.Definitions,fixture.References,[]); + var indexed=metadata.WithObservations(result.Rows); + if(indexed.Observations.Length!=expectedRows||!indexed.HasObservation(first.GetProperty("id").GetGuid())||!indexed.HasObservation(result.Rows[^1].GetProperty("id").GetGuid())) + throw new Exception("The complete batch must also retain its native observation indexes."); + Console.WriteLine($"PIN_PAYLOAD rows={result.Rows.Length} indexedRows={indexed.Observations.Length} bytes={source.Length} heapLimit={GC.GetGCMemoryInfo().TotalAvailableMemoryBytes} runtime={Environment.Version}"); + return; + } + using var body=new MemoryStream(Encoding.UTF8.GetBytes("""{"rows":[{"ordinal":7,"value":-0.00000000000000000100,"label":"synthetic λ","empty":"","missing":null}],"denied":["00000000-0000-0000-0000-000000000001"]}""")); + var payload=await ObservationPinPayload.Read(body,CancellationToken.None); + var row=payload.Rows.Single(); + if(row.GetProperty("ordinal").GetInt32()!=7||row.GetProperty("value").GetString()!="-0.00000000000000000100"||row.GetProperty("label").GetString()!="synthetic λ"||row.GetProperty("empty").GetString()!=""||row.GetProperty("missing").ValueKind!=JsonValueKind.Null||!payload.Denied.Contains(Guid.Parse("00000000-0000-0000-0000-000000000001"))) + throw new Exception("Pin batches must retain row order, numeric lexemes, empty/null text and exact denied identities."); + foreach(var value in new[]{"{}","{\"rows\":[]}","{\"denied\":[]}","{\"rows\":[],\"denied\":["}) + { + using var malformed=new MemoryStream(Encoding.UTF8.GetBytes(value)); + try{await ObservationPinPayload.Read(malformed,CancellationToken.None);throw new Exception("Malformed or incomplete pin batches must not become empty successful results.");} + catch(Exception error)when(error is JsonException or KeyNotFoundException){} + } + Console.WriteLine("PASS observation pin payload: complete rows/denials, exact numeric text and missing/truncated refusal"); + } +} diff --git a/tests/OpenEtrm.Api.Checks/Program.cs b/tests/OpenEtrm.Api.Checks/Program.cs index 9ebb4d4c..6ba9f016 100644 --- a/tests/OpenEtrm.Api.Checks/Program.cs +++ b/tests/OpenEtrm.Api.Checks/Program.cs @@ -40,7 +40,10 @@ if(args is ["--power-hedge"]){StructuredPowerHedgeChecks.Run();return;} if(args is ["--assistant-prompts"]){TurnWorkerChecks.Run();AgentToolChecks.Run();await ModelProviderChecks.Run();return;} if(args is ["--json-read-transport"]){await JsonReadTransportChecks.Run();return;} -if(args is ["--async-reads"]){await JsonReadTransportChecks.Run();AsyncReadContractChecks.Run();await AsyncReadChecks.Run();await UngatedReadChecks.Run();return;} +if(args is ["--large-catalog-http"]){await LargeCatalogReadChecks.Run();return;} +if(args is ["--observation-pin-payload",var payloadPath,var payloadRows]){await ObservationPinPayloadChecks.Run(payloadPath,int.Parse(payloadRows,CultureInfo.InvariantCulture));return;} +if(args is ["--async-read-budget"]){await AsyncReadBudgetChecks.Run();return;} +if(args is ["--async-reads"]){await JsonReadTransportChecks.Run();await ObservationPinPayloadChecks.Run();await AsyncReadStreamChecks.Run();AsyncReadContractChecks.Run();await AsyncReadChecks.Run();await UngatedReadChecks.Run();return;} if(args is ["--ungated-reads"]){await UngatedReadChecks.Run();return;} if(args is ["--async-read-contracts"]){AsyncReadContractChecks.Run();return;} if(args is ["--calendar-authoring"]){CalendarAuthoringSchemaChecks.Run();return;} diff --git a/tests/browser/async-workbench.mjs b/tests/browser/async-workbench.mjs index c333b20d..50379fe7 100644 --- a/tests/browser/async-workbench.mjs +++ b/tests/browser/async-workbench.mjs @@ -3,13 +3,21 @@ export async function run({command,evaluate,wait,click,set,persona,check,idle,ur await command('Page.enable');await command('Runtime.enable'); const fixture=await command('Page.addScriptToEvaluateOnNewDocument',{source:`(()=>{ const send=window.fetch.bind(window),NativeSource=window.EventSource; - window.__reads={posts:[],acks:[],held:[],hold:sessionStorage.getItem('synthetic-held-startup-read')?[sessionStorage.getItem('synthetic-held-startup-read')]:[],responses:[],sources:0,release(){for(const item of this.held.splice(0))item.source.dispatchEvent(new MessageEvent('response',{data:item.data}));}}; + window.__reads={posts:[],acks:[],held:[],hold:sessionStorage.getItem('synthetic-held-startup-read')?[sessionStorage.getItem('synthetic-held-startup-read')]:[],responses:[],sources:0,catalogBytes:0,holdNextCatalog:false,catalogsHeld:[],releaseCatalogs(){for(const resolve of this.catalogsHeld.splice(0))resolve();},release(){for(const item of this.held.splice(0))item.source.dispatchEvent(new MessageEvent('response',{data:item.data}));}}; sessionStorage.removeItem('synthetic-held-startup-read'); window.fetch=async(input,init)=>{ const path=new URL(String(input),location.href).pathname; if(path==='/api/async-reads'&&init?.method==='POST')__reads.posts.push(JSON.parse(init.body)); const response=await send(input,init); if(path==='/api/async-reads'&&init?.method==='POST')__reads.acks.push({status:response.status,body:await response.clone().json()}); + if(path==='/api/workspace/catalog'&&response.ok){ + const catalog=await response.json(); + // Scale unselected metadata without changing the selected financial inputs. + catalog.definitions.synthetic_notes=Array.from({length:4096},(_,i)=>({id:'synthetic-note-'+i,label:'x'.repeat(4096)})); + const text=JSON.stringify(catalog);__reads.catalogBytes=new TextEncoder().encode(text).length; + if(__reads.holdNextCatalog){__reads.holdNextCatalog=false;__reads.heldCatalogCurve=new URL(String(input),location.href).searchParams.get('curve');await new Promise(resolve=>__reads.catalogsHeld.push(resolve));} + return new Response(text,{status:response.status,headers:response.headers}); + } return response; }; window.EventSource=class extends EventTarget{ @@ -27,10 +35,12 @@ export async function run({command,evaluate,wait,click,set,persona,check,idle,ur await click('#nav-market'); const selected=await evaluate("[...document.querySelector('#snapshot').options].find(o=>o.textContent.includes('Dec 18')).value"); await set('#snapshot',selected);await wait(idle,30000); - const curves=await evaluate("[...document.querySelectorAll('.curve-button')].slice(0,2).map(button=>button.dataset.curve)"); + const curves=await evaluate("['local_power',...[...document.querySelectorAll('.curve-button')].map(button=>button.dataset.curve).filter(code=>code!=='local_power')].slice(0,2)"); if(curves.length!==2)throw Error('The synthetic workbench needs two explicit curve choices.'); await evaluate("__reads.hold.push('curve.evaluate.v1')");await choose(curves[0]); await wait("__reads.held.length===1",30000); + await check('a power curve can hydrate a catalog larger than the read-result budget without embedding it in its price response', + "__reads.catalogBytes>16*1024*1024&&__reads.posts.some(p=>p.operation==='curve.evaluate.v1'&&p.body.curve==='local_power')&&__reads.posts.filter(p=>p.operation==='curve.evaluate.v1').every(p=>p.body.includeWorkspace!==true)"); await check('a real pending SSE curve leaves navigation and other curves usable', "!document.querySelector('#nav-futures').disabled&&[...document.querySelectorAll('.curve-button')].every(button=>!button.disabled)&&document.querySelector('#market-page').dataset.readState==='pending'"); await choose(curves[1]);await wait("document.querySelector('#market-page').dataset.readState!=='pending'&&document.querySelector('#contracts tbody tr')",30000); @@ -80,5 +90,12 @@ export async function run({command,evaluate,wait,click,set,persona,check,idle,ur await evaluate('new Promise(resolve=>requestAnimationFrame(()=>requestAnimationFrame(resolve)))'); await check('native startup readiness waits for the actual initial Risk scope result',"document.querySelector('#risk-page').dataset.readState==='pending'&&performance.getEntriesByName('leanetrm:workspace-ready','mark').length===0"); await evaluate('__reads.release()');await wait("performance.getEntriesByName('leanetrm:workspace-ready','mark').length===1&&"+idle,30000); - }finally{await evaluate('__reads?.release()').catch(()=>{});await command('Page.removeScriptToEvaluateOnNewDocument',{identifier:fixture.identifier});} + await evaluate('__reads.holdNextCatalog=true');await persona('viewer');await click('#nav-market'); + await choose(curves[0]);await wait('__reads.catalogsHeld.length===1',30000); + const heldCurve=await evaluate('__reads.heldCatalogCurve');const currentCurve=curves.find(code=>code!==heldCurve); + await choose(currentCurve);await wait("document.querySelector('#market-page').dataset.readState!=='pending'&&document.querySelector('#contracts tbody tr')",30000); + const currentPrices=await evaluate("document.querySelector('#contracts').textContent"); + await evaluate('__reads.releaseCatalogs();new Promise(resolve=>requestAnimationFrame(()=>requestAnimationFrame(resolve)))'); + await check('late catalog hydration cannot replace a newer curve or publish a stale error',`document.querySelector('#contracts').textContent===${JSON.stringify(currentPrices)}&&document.querySelector('#error').hidden`); + }finally{await evaluate('__reads?.releaseCatalogs();__reads?.release()').catch(()=>{});await command('Page.removeScriptToEvaluateOnNewDocument',{identifier:fixture.identifier});} } diff --git a/tests/browser/market-dates.mjs b/tests/browser/market-dates.mjs index dc8a1a32..e30a836d 100644 --- a/tests/browser/market-dates.mjs +++ b/tests/browser/market-dates.mjs @@ -3,11 +3,12 @@ export async function run({command,evaluate,wait,click,set,submit,persona,check, await command('Runtime.enable');await command('Page.enable'); await command('Emulation.setDeviceMetricsOverride',{width:1440,height:1100,deviceScaleFactor:1,mobile:false}); await command('Page.addScriptToEvaluateOnNewDocument',{source:` - window.marketDateRequests=[];window.marketDateCalls=[];window.curveCloseRequests=[]; + window.marketDateRequests=[];window.marketDateCalls=[];window.curveCloseRequests=[];window.curveCloseStarts=[]; const original=window.fetch;window.fetch=async(...args)=>{ const path=new URL(typeof args[0]==='string'?args[0]:args[0].url,location.href).pathname; // A curve's own short list of closes is a separate read from the calendars these counts follow. const curveCloses=path.endsWith('/market/dates')&&(b=>b.curve&&b.maximumDates)(JSON.parse(args[1].body)); + if(curveCloses)window.curveCloseStarts.push({curve:JSON.parse(args[1].body).curve,drawn:!!document.querySelector('#contracts tbody tr')}); if(!curveCloses)window.marketDateCalls.push(path); let response;if(path.endsWith('/market/dates')&&window.denyCalendar){window.denyCalendar=false;response=new Response(JSON.stringify({code:'access_denied'}),{status:403,headers:{'Content-Type':'application/json'}});}else response=await original(...args); // Before its newest date arrives, the workspace calendar ends one date earlier. Showing that to the app stands in for a @@ -33,7 +34,13 @@ export async function run({command,evaluate,wait,click,set,submit,persona,check, if(body.curve===null&&body.definition===null&&body.maximumDates===1){window.holdNextCalendar=false;await new Promise(resolve=>window.releaseCalendar=resolve);} } if(!curveCloses&&(path.endsWith('/market/dates')||path.endsWith('/custom-curves/preview')))window.marketDateRequests.push({path,body:JSON.parse(args[1].body),status:response.status,payload:await response.clone().json()}); - if(curveCloses)window.curveCloseRequests.push({body:JSON.parse(args[1].body),status:response.status,payload:await response.clone().json()}); + if(curveCloses){ + window.curveCloseRequests.push({body:JSON.parse(args[1].body),status:response.status,payload:await response.clone().json()}); + if(window.holdNextCurveCloses){ + window.holdNextCurveCloses=false;window.heldCurveCloseSignal=args[1].signal; + await new Promise(resolve=>window.releaseCurveCloses=resolve); + } + } return response; }; `}); @@ -214,7 +221,10 @@ export async function run({command,evaluate,wait,click,set,submit,persona,check, await click('#current-view-context');await wait("document.querySelector('#date-context').dataset.contextMode==='current'&&document.querySelectorAll('.curve-button').length>0&&document.querySelector('#snapshot').value!==''&&"+idle,30000); await wait("document.querySelector('.curve-button[data-curve=\"ng_monthly\"] small')?.textContent.includes('last close')",30000); await check('a listed curve states its last close, as the backend names it',`(async()=>{const session=document.querySelector('#close-session').value;const read=await(await fetch('/api/market/last-closes',{method:'POST',headers:{'Content-Type':'application/json','X-Preview-CSRF':(await(await fetch('/api/bootstrap')).json()).csrfToken},body:JSON.stringify({session})})).json();const last=read.curves.find(c=>c.curve==='ng_monthly');const label=new Intl.DateTimeFormat('en',{year:'numeric',month:'short',day:'numeric',timeZone:'UTC'}).format(new Date(last.businessDate+'T00:00:00Z'));return document.querySelector('.curve-button[data-curve="ng_monthly"] small').textContent.includes('last close '+label)&&document.querySelector('.market-card[data-curve="ng_monthly"] .market-card-last-close').textContent.includes(label);})()`); + await evaluate('window.curveCloseStarts=[]'); await click('.curve-button[data-curve="ng_monthly"]');await wait("document.querySelectorAll('#curve-closes .vol-close-chip').length>1&&"+idle,30000); + await check('the advisory close strip starts after the selected prices are drawn', + "window.curveCloseStarts.some(item=>item.curve==='ng_monthly')&&window.curveCloseStarts.filter(item=>item.curve==='ng_monthly').every(item=>item.drawn)"); await check('a curve lists the closes it can be priced at, the one in view marked',"document.querySelectorAll('#curve-closes .vol-close-chip[aria-current=true]').length===1&&!document.querySelector('#curve-closes').hidden"); const older="[...document.querySelectorAll('#curve-closes .vol-close-chip[aria-current=false]')].at(-1)"; await evaluate("window.asOfBefore=document.querySelector('#snapshot').value;window.olderClose="+older+".dataset.close"); @@ -237,4 +247,19 @@ export async function run({command,evaluate,wait,click,set,submit,persona,check, await check('choosing another close in that window retires its separate-window note',"document.querySelector('#close-window-note').hidden"); await command('Page.navigate',{url:url+'/?page=market&curve=ng_monthly&close=00000000-0000-4000-8000-000000000000'});await wait("document.querySelector('#close-window-note')?.textContent.length>0&&"+idle,30000); await check('a close this view does not offer is named as unavailable and nothing is chosen for it',"document.querySelector('#close-window-note').textContent.includes('not available')&&document.querySelector('#snapshot').value===''"); + await command('Page.navigate',{url:address}); + await wait("document.querySelector('#curve-closes .vol-close-chip[aria-current=true]')!==null&&"+idle,30000); + await evaluate('window.holdNextCurveCloses=true'); + await click('.curve-button[data-curve="interest_rates"]',30000); + await wait("typeof window.releaseCurveCloses==='function'&&!window.heldCurveCloseSignal.aborted",30000); + await evaluate(`(()=>{ + const field=document.querySelector('#date-context-on'); + if(window.heldCurveCloseSignal.aborted)throw Error('The fixture requires a live advisory close read.'); + field.value='2020-12-19';field.dispatchEvent(new Event('input',{bubbles:true})); + window.abortedByContextEdit=window.heldCurveCloseSignal.aborted; + })()`); + await wait("document.querySelector('#environment-context-note').textContent.includes('Context changed')"); + await evaluate('window.releaseCurveCloses();new Promise(resolve=>requestAnimationFrame(()=>requestAnimationFrame(resolve)))'); + await check('editing the context cancels the pending advisory close strip and keeps it hidden', + "window.abortedByContextEdit&&document.querySelector('#curve-closes').hidden&&document.querySelector('#curve-closes').children.length===0"); } diff --git a/tests/browser/market-navigation.mjs b/tests/browser/market-navigation.mjs index db7ba2e5..9f4f9ec6 100644 --- a/tests/browser/market-navigation.mjs +++ b/tests/browser/market-navigation.mjs @@ -6,15 +6,16 @@ export async function run({command,evaluate,wait,click,set,persona,check,idle,ur const body=path==='/api/evaluate'?JSON.parse(init.body):null;if(body)window.__pricingReads.push(body); if(path==='/api/market/current'&&window.__denySampleReset){window.__denySampleReset=false;return new Response(JSON.stringify({code:'access_denied'}),{status:403,headers:{'Content-Type':'application/json'}});} let response=await send(input,init); - if(body?.includeWorkspace&&window.__breakCombined){ - const mode=window.__breakCombined;window.__breakCombined=null;const data=await response.json(); - if(mode==='missing')delete data.workspace; - if(mode==='revision')data.workspace.definitions.revisions[0].id='mismatched'; + const mode=window.__breakHydration; + if(mode&&(mode==='sources'?body:path==='/api/workspace/catalog')){ + window.__breakHydration=null;const data=await response.json(); + if(mode==='missing')delete data.referenceScope; + if(mode==='revision')data.definitions.revisions[0].id='mismatched'; if(mode==='sources')delete data.observationScope; response=new Response(JSON.stringify(data),{status:response.status,headers:response.headers}); } if(path==='/api/workspace/navigation'&&new URLSearchParams(location.search).has('hold-navigation'))await new Promise(resolve=>window.__releaseNavigation=resolve); - if((path==='/api/workspace/catalog'||body?.includeWorkspace)&&window.__holdWorkspace){window.__holdWorkspace=false;await new Promise(resolve=>window.__releaseWorkspace=resolve);} + if(path==='/api/workspace/catalog'&&window.__holdWorkspace){window.__holdWorkspace=false;await new Promise(resolve=>window.__releaseWorkspace=resolve);} return response; };`}); try{ @@ -28,11 +29,11 @@ export async function run({command,evaluate,wait,click,set,persona,check,idle,ur await set('#snapshot',selected);await wait(idle); await check('selecting a market date does not load unused contracts or reference data',"!window.__navigationReads.includes('/api/workspace/catalog')"); await evaluate("window.__firstPriceStart=performance.now();[...document.querySelectorAll('.curve-button')].find(b=>b.textContent.startsWith('Local power forwards')).click()");await wait(idle); - await check('first pricing hydrates a complete workspace in one evaluate without a preceding catalog read',"!window.__navigationReads.includes('/api/workspace/catalog')&&window.__pricingReads.length===1&&window.__pricingReads[0].includeWorkspace===true&&document.querySelector('#curve-chart canvas')&&document.querySelector('#inputs tbody tr')&&document.querySelector('#error').hidden"); - await check('combined hydration sets the selected curve units before rendering the chart',"document.querySelector('#unit').textContent.trim()&&document.querySelector('#curve-chart').dataset.chartUnit===document.querySelector('#unit').textContent"); + await check('first pricing reads the catalog once before the compact evaluation',"window.__navigationReads.filter(p=>p==='/api/workspace/catalog').length===1&&window.__navigationReads.indexOf('/api/workspace/catalog')b.textContent.startsWith('Local power forwards')).click()");await wait(idle); - await check('repeated pricing uses the ordinary evaluation response without reloading catalog metadata',"window.__pricingReads.length===2&&!Object.hasOwn(window.__pricingReads[1],'includeWorkspace')&&!window.__navigationReads.includes('/api/workspace/catalog')&&document.querySelector('#error').hidden"); + await check('repeated pricing uses the ordinary evaluation response without reloading catalog metadata',"window.__pricingReads.length===2&&window.__pricingReads[1].includeWorkspace!==true&&window.__navigationReads.filter(p=>p==='/api/workspace/catalog').length===1&&document.querySelector('#error').hidden"); await persona('viewer'); await check('changing profiles selects the uniquely latest offered market date',"(async()=>{const frames=(await(await fetch('/api/workspace/navigation')).json()).frames,newest=Math.max(...frames.map(f=>Date.parse(f.at))),latest=frames.filter(f=>Date.parse(f.at)===newest);return document.querySelector('#snapshot').value===(latest.length===1?latest[0].id:'');})()"); // This check is about a curve chosen with no date selected, so the date the person arrived on is cleared first. @@ -49,13 +50,13 @@ export async function run({command,evaluate,wait,click,set,persona,check,idle,ur await check('presentation refresh after a cancelled hydration never treats navigation as full pricing data',"document.querySelector('#contracts').children.length===0&&document.querySelector('#curve-detail').hidden&&document.querySelector('#error').hidden"); for(const mode of ['missing','revision','sources']){ await persona('viewer');await set('#snapshot',selected);await wait(idle); - await evaluate(`window.__breakCombined=${JSON.stringify(mode)};[...document.querySelectorAll('.curve-button')].find(b=>b.textContent.startsWith('Local power forwards')).click()`);await wait(idle); - await check('combined '+mode+' rejects before publishing prices or sources',"!document.querySelector('#error').hidden&&document.querySelector('#contracts').children.length===0&&document.querySelector('#inputs').children.length===0"); - await evaluate("[...document.querySelectorAll('.curve-button')].find(b=>b.textContent.startsWith('Local power forwards')).click()");await wait(idle); - await check('a rejected combined '+mode+' cannot populate the workspace cache',"window.__pricingReads.at(-1).includeWorkspace===true&&document.querySelector('#curve-chart canvas')&&document.querySelector('#error').hidden"); + await evaluate(`window.__breakHydration=${JSON.stringify(mode)};[...document.querySelectorAll('.curve-button')].find(b=>b.textContent.startsWith('Local power forwards')).click()`);await wait(idle); + await check('invalid hydration '+mode+' rejects before publishing prices or sources',"!document.querySelector('#error').hidden&&document.querySelector('#contracts').children.length===0&&document.querySelector('#inputs').children.length===0"); + await evaluate("window.__catalogsBeforeRetry=window.__navigationReads.filter(p=>p==='/api/workspace/catalog').length;[...document.querySelectorAll('.curve-button')].find(b=>b.textContent.startsWith('Local power forwards')).click()");await wait(idle); + await check('a rejected '+mode+' response cannot populate the workspace cache',"window.__navigationReads.filter(p=>p==='/api/workspace/catalog').length===window.__catalogsBeforeRetry+1&&window.__pricingReads.at(-1).includeWorkspace!==true&&document.querySelector('#curve-chart canvas')&&document.querySelector('#error').hidden"); } await evaluate("window.__beforeDeniedSample=window.__pricingReads.length;window.__denySampleReset=true");await click('#example'); - await check('a denied sample context reset clears prior prices before requesting combined data',"!document.querySelector('#error').hidden&&window.__pricingReads.length===window.__beforeDeniedSample&&document.querySelector('#contracts').children.length===0&&document.querySelector('#inputs').children.length===0"); + await check('a denied sample context reset clears prior prices before requesting prices',"!document.querySelector('#error').hidden&&window.__pricingReads.length===window.__beforeDeniedSample&&document.querySelector('#contracts').children.length===0&&document.querySelector('#inputs').children.length===0"); await persona('viewer');await evaluate('window.__referenceStart=performance.now()');await click('#reference-area-curves');await click('#nav-table-currencies'); await check('first reference navigation explicitly hydrates its exact pin and unit metadata',"document.querySelector('#reference-page-status').textContent.startsWith('Rows ')&&document.querySelector('#reference tbody tr')&&document.querySelector('#error').hidden"); console.log('FIRST_REFERENCE_WITH_HYDRATION_MS='+await evaluate('performance.now()-window.__referenceStart')); diff --git a/tests/browser/startup-requests.mjs b/tests/browser/startup-requests.mjs index 3bf7d959..1c46a441 100644 --- a/tests/browser/startup-requests.mjs +++ b/tests/browser/startup-requests.mjs @@ -19,7 +19,7 @@ export async function run({command,evaluate,wait,click,submit,persona,check,idle window.__startupNavigationKind=data.kind;window.__startupNavigationHasDetail=['definitions','references','observations'].some(key=>Object.hasOwn(data,key)); }else if(path==='/api/workspace/catalog'){ window.__startupObservationCount=data.observations.length;window.__startupScope=data.observationScope; - }else if(path==='/api/workspace'||path==='/api/evaluate'&&data.observationScope==='selected_frame'){ + }else if(path==='/api/workspace'||path==='/api/evaluate'&&['selected_frame','curve_lineage_v1'].includes(data.observationScope)){ // Unrendered history must not add formatting round trips. Preserve the // real selected sources so normal exact source tables remain exercised. for(let i=0;i<2049;i++){ diff --git a/tools/check_async_workbench_http.py b/tools/check_async_workbench_http.py index f8d7c4b1..9ac3232e 100644 --- a/tools/check_async_workbench_http.py +++ b/tools/check_async_workbench_http.py @@ -92,7 +92,56 @@ def check(name, condition=True): history = c.call('portfolio/risk/history', history_body) curve = {'snapshotId':frame['id'], 'curve':'local_power', 'includeSources':True, 'includeWorkspace':True, 'contractDisplay':{'version':'curve_display_v1','contracts':'eligible_contracts_v1'}} - curve_points=c.call('evaluate',curve)['points'] + full_curve=c.call('evaluate',curve) + curve_points=full_curve['points'] + compact_body={**curve,'includeWorkspace':False,'sourceScope':'curve_lineage_v1'} + compact=c.call('evaluate',compact_body) + check('curve source scope refuses unknown choices and use without source detail', + c.call('evaluate',{**compact_body,'sourceScope':'synthetic_unknown'},expected=422)['code']=='async_request' + and c.call('evaluate',{**compact_body,'includeSources':False},expected=422)['code']=='async_request') + + pins={key for point in curve_points for key in point['lineage']} + fact_keys=('series_id','source_code','source_event_id','observed_at','role') + def fact(row): + return tuple(row[k] for k in fact_keys) + facts={fact(row) for row in full_curve['observations'] if row['id'] in pins} + check('compact curve returns exactly used pins and their visible corrections', + compact['observationScope']=='curve_lineage_v1' and 'workspace' not in compact + and set(compact['inputSelection']['observationIds'])==pins + and set(compact['observationIds'])==pins + and compact['observations']==[row for row in full_curve['observations'] if fact(row) in facts] + and {k:v for k,v in compact.items() if k not in ('observations','observationIds','observationScope','inputSelection')} + == {k:v for k,v in full_curve.items() if k not in ('workspace','observations','observationIds','observationScope','inputSelection')} + and compact['inputSelection']['correctionsSupported']==full_curve['inputSelection']['correctionsSupported'] + and len(compact['observations'])!!curve&&listedCurve().marketType==='rates'; -const row=(table:string,key:string,value:string,reference=false)=>exactRow((reference?data().references:data().definitions)[table],key,value); -const observation=(id:string)=>exactRow(data().observations,'id',id); +const row=(table:string,key:string,value:string,reference=false)=>indexedRow((reference?data().references:data().definitions)[table],key,value); +const observation=(id:string)=>indexedRow(data().observations,'id',id); const instant=(value:string)=>displayInstant(value,prefs.timeZone); const unit=(quote:string)=>quoteUnit(quote,data()); function notice(message:string){const shown=$('notice');text('notice',withNotice(shown.hidden?'':shown.textContent??'',publicText(message)));shown.hidden=false;} @@ -423,16 +423,17 @@ function renderLayout(){ function renderTimeZones(){document.querySelectorAll('[data-user-time-zone]').forEach(n=>{n.textContent=prefs.timeZone;});} function invalidateMarketDates(){ marketDateGeneration++;marketDatesLoaded=false; + curveClosesRead?.abort(); marketCalendar.invalidate(contextDirty?'Apply the changed history and knowledge context first.':'Open the calendar to check available closes.'); } type MarketDatesRequest={curve:string|null;timeZone:string;session:string;definition:unknown;maximumDates?:number}; const datesMismatch=()=>Error('The available market dates do not match this workspace. Reload the calendar.'); // What the calendar offers for one scope, checked as describing this workspace. It resolves to nothing when the // workspace or the scope changed while it loaded, since that answer no longer applies. -async function readAvailability(request:MarketDatesRequest):Promise{ +async function readAvailability(request:MarketDatesRequest,signal?:AbortSignal):Promise{ const owner=directory(),generation=marketDateGeneration,stale=()=>navigation!==owner||generation!==marketDateGeneration; let catalog:MarketDates; - try{catalog=await api('market/dates',request);}catch(error){if(stale())return undefined;throw error;} + try{catalog=await api('market/dates',request,signal);}catch(error){if(stale())return undefined;throw error;} if(stale())return undefined; const frames=catalog.dates.flatMap(group=>group.frames); // Every close's time must be readable, as the newest close and the badge compare them, before anything keeps the answer. @@ -548,21 +549,27 @@ async function selectCurve(code:string,signal?:AbortSignal){ const selected=curve?listedCurve().source:null; if(JSON.stringify(previous)!==JSON.stringify(selected))workspace=undefined; invalidateMarketDates(); - // A priced curve stays in view, marked as pending, until its successor's prices arrive; the page then changes in one step - // instead of emptying first. Its prices are forgotten at once, so nothing can be opened from the drawing that is leaving. - if(curve&&select('snapshot').value&&!$('curve-detail').hidden&&!$('curve-chart').classList.contains('empty')){ - forgetPrices(); - let closes:'held'|'none'|'stale'|undefined;const reading=loadCurveCloses().then(state=>closes=state); - try{await build(signal);} - // A read given up for a newer choice leaves the page to that choice; any other failure empties it, as a first choice does. - catch(error){if(!signal?.aborted||!curveRead){clearPrices();void reading.then(drawCurveCloses);}throw error;} - // The closes arrive with the prices when they were read in time; a slower read hides the leaving curve's closes until it ends. - drawCurveCloses(closes??'none');if(!closes)void reading.then(drawCurveCloses); - return; + const priced=curve&&select('snapshot').value&&!$('curve-detail').hidden&&!$('curve-chart').classList.contains('empty'); + if(priced)forgetPrices(); + else clearPrices(); + drawCurveCloses('none'); + if(!curve){renderContext();await loadBoard();return;} + const owner=navigation,generation=marketDateGeneration; + try{ + if(select('snapshot').value)await build(signal); + else{await loadWorkspace(signal);signal?.throwIfAborted();renderContext();} + } + catch(error){ + if(!signal?.aborted&&navigation===owner&&marketDateGeneration===generation)clearPrices(); + throw error; + } + finally{ + // Let the browser paint the selected prices before starting the advisory read. + requestAnimationFrame(()=>requestAnimationFrame(()=>{ + if(!signal?.aborted&&navigation===owner&&marketDateGeneration===generation) + void renderCurveCloses(); + })); } - clearPrices();void renderCurveCloses(); - if(!curve){renderContext();await loadBoard();} - else if(select('snapshot').value)await build(signal);else{await loadWorkspace(signal);signal?.throwIfAborted();renderContext();} } function chooseCurve(code:string){ if(busy)return; @@ -624,16 +631,20 @@ async function loadLastCloses(){ } // The closes the curve in view can price, newest first. Reading them is separate from pricing, so a close the curve is missing from still lists the closes it has. let curveCloses:{key:string;closes:{id:string;businessDate:string}[]}|undefined; +let curveClosesRead:AbortController|undefined; // Reads the closes of the curve in view unless they are held: 'held' when they can be drawn, 'none' when there are none to show, // 'stale' when another curve came into view meanwhile. async function loadCurveCloses():Promise<'held'|'none'|'stale'>{ if(!curve||closeSession===null)return 'none'; const code=curve,key=JSON.stringify([code,closeSession,prefs.timeZone,historicalContext()?.contextId??null]); if(curveCloses?.key===key)return 'held'; + curveClosesRead?.abort(); + const reading=curveClosesRead=new AbortController(); let catalog:MarketDates|undefined; // A read that failed or no longer applies is not remembered, so the next render asks again. - try{catalog=await readAvailability({curve:code,timeZone:prefs.timeZone,session:closeSession,definition:null,maximumDates:10});}catch{catalog=undefined;} - if(curve!==code)return 'stale'; + try{catalog=await readAvailability({curve:code,timeZone:prefs.timeZone,session:closeSession,definition:null,maximumDates:10},reading.signal);}catch{catalog=undefined;} + finally{if(curveClosesRead===reading)curveClosesRead=undefined;} + if(reading.signal.aborted||curve!==code)return 'stale'; if(!catalog)return 'none'; curveCloses={key,closes:catalog.dates.flatMap(day=>day.frames.map(close=>({id:close.id,businessDate:close.businessDate}))).reverse()}; return 'held'; @@ -759,16 +770,19 @@ function renderPrices(){ } async function build(signal?:AbortSignal){ if(contextDirty)throw Error('Apply the changed context before viewing prices.'); - const owner=directory(),previousWorkspace=workspace,includeWorkspace=!workspace,started=marketGeneration,scope=marketSelection(); + const owner=directory(),previousWorkspace=workspace,started=marketGeneration,scope=marketSelection(); const f=frame();if(!curve)throw Error('Choose a curve first.'); - const {workspace:includedWorkspace,...next}=await api('evaluate',{snapshotId:f.id,curve,contractDisplay:prefs.contractDisplay,includeSources:true,...(includeWorkspace?{includeWorkspace:true}:{})},signal); - signal?.throwIfAborted(); - if(navigation!==owner||workspace!==previousWorkspace||started!==marketGeneration||scope!==marketSelection())throw Error('The account or market selection changed. Reload before continuing.'); - if(next.snapshotId!==f.id||next.curve!==curve||next.observationScope!=='selected_frame'||!Array.isArray(next.observations))throw Error('The selected price sources are missing or inconsistent. Refresh and try again.'); - if(includeWorkspace!==!!includedWorkspace)throw Error('The requested market catalog is missing or inconsistent. Refresh and try again.'); - const hydrated=includeWorkspace?validateWorkspace(includedWorkspace!,owner,started,scope):previousWorkspace!; + const current=()=>{signal?.throwIfAborted();if(navigation!==owner||workspace!==previousWorkspace||started!==marketGeneration||scope!==marketSelection())throw staleSelection();}; + // Catalogue detail has its own HTTP read. Keep it out of the bounded price + // result, and install neither response until both match this exact selection. + const catalog=previousWorkspace??await readWorkspaceCatalog(signal);current(); + const hydrated=previousWorkspace??validateWorkspace(catalog,owner,started,scope); + const {workspace:unexpectedWorkspace,...next}=await api('evaluate',{snapshotId:f.id,curve,contractDisplay:prefs.contractDisplay,includeSources:true,sourceScope:"curve_lineage_v1",definitionScope:"curve_dependencies_v1"},signal); + current(); + if(next.snapshotId!==f.id||next.curve!==curve||next.observationScope!=='curve_lineage_v1'||!Array.isArray(next.observations))throw Error('The selected price sources are missing or inconsistent. Refresh and try again.'); + if(unexpectedWorkspace!==undefined)throw Error('The price response contains an unexpected market catalog. Refresh and try again.'); try{workspace=hydrated;data().observations=next.observations;result=next;renderContext();renderPrices();} - catch(error){clearPrices();if(includeWorkspace)workspace=undefined;throw error;} + catch(error){clearPrices();if(!previousWorkspace)workspace=undefined;throw error;} } const contractLabel=(p:Point)=>result?.kind==='rates'?(p.label&&p.label!==p.maturity?`${p.label} · ${displayDate(pointDate(p))}`:displayDate(pointDate(p))):deliveryLabel(p.deliveryFrom!,p.deliveryTo!); const staleSelection=()=>Error('The account or market selection changed. Reload before continuing.'); @@ -1060,8 +1074,11 @@ function validateWorkspace(next:Workspace,owner:MarketNavigation,started:number, async function loadWorkspace(signal?:AbortSignal){ if(workspace)return; const owner=directory(),started=marketGeneration,scope=marketSelection(); - const parameters=new URLSearchParams();if(curve)parameters.set('curve',curve);if(select('snapshot').value)parameters.set('snapshotId',select('snapshot').value); - const next=await api(`workspace/catalog${parameters.size?'?'+parameters.toString():''}`,undefined,signal);signal?.throwIfAborted();workspace=validateWorkspace(next,owner,started,scope); + const next=await readWorkspaceCatalog(signal);signal?.throwIfAborted();workspace=validateWorkspace(next,owner,started,scope); +} +function readWorkspaceCatalog(signal?:AbortSignal){ + const parameters=new URLSearchParams({frameDetail:'summary'});if(curve)parameters.set('curve',curve);if(select('snapshot').value)parameters.set('snapshotId',select('snapshot').value); + return api(`workspace/catalog?${parameters}`,undefined,signal); } // Only an explicitly saved, still-published default initializes a new view. async function loadCloseSessions(){ diff --git a/web/src/market-view.ts b/web/src/market-view.ts index f1993d0d..16286278 100644 --- a/web/src/market-view.ts +++ b/web/src/market-view.ts @@ -4,7 +4,7 @@ export type ReferenceRevision={catalogId:string;revisionId:string}; export type ReferenceSource={referenceRevision:ReferenceRevision;definitionRevisionId:string}; export type ReferenceDisplay={sourceRevision:ReferenceRevision;displayRevision:ReferenceRevision;version:number;references:Record}; export type Frame = {id:string; name:string; at:string; businessDate:string; originCloseId?:string|null;observationIds:string[];curveContracts:Record}; -export type Workspace = {referenceSources?:Record;futuresConvention:string;curveNames?:{code:string;name:string}[]|null;powerCurveCodes:string[];unitCatalogError?:string|null;unitCatalog?:{revision:unknown;references:Record};referenceDisplay?:ReferenceDisplay|null;unitReferenceDisplay?:ReferenceDisplay|null;profile:string;example:{at:string;curve:string}|null; definitions:Record; references:Record;referenceScope?:'definition_closure'|'all_visible';observationScope?:'on_demand';observations:Row[]; frames:Frame[]; initialFrameIds:string[]}; +export type Workspace = {referenceSources?:Record;futuresConvention:string;curveNames?:{code:string;name:string}[]|null;powerCurveCodes:string[];unitCatalogError?:string|null;unitCatalog?:{revision:unknown;references:Record};referenceDisplay?:ReferenceDisplay|null;unitReferenceDisplay?:ReferenceDisplay|null;profile:string;example:{at:string;curve:string}|null; definitions:Record; references:Record;referenceScope?:'definition_closure'|'all_visible';observationScope?:'on_demand';observations:Row[]; frames:FrameSummary[]; initialFrameIds:string[]}; import type {PricingNode} from './pricing-view.js'; import type {FrameSummary} from './market-navigation.js'; @@ -14,13 +14,13 @@ export type CurveFront={contractId:string;deliveryFrom:string;deliveryTo:string; export type CurveSummary={kind:'contracts';contractCount:number;front:CurveFront|null;availability:'no_remaining_contracts'|'no_priced_contract'|null}|{kind:'rates';maturityCount:number;firstMaturity:string|null;firstMark:string|null;markUnitLabel:string;dayCount:string}; export type PriceInputSelection={snapshotId:string;source:{definitionRevisionId:string;referenceCatalogId:string;referenceRevisionId:string};observationIds:string[];correctionsSupported:boolean}; export type CurveResult = {contractDisplay?:ContractDisplay;summary?:CurveSummary;kind?:'rates';convention?:{dayCount:string;compounding:string;interpolation:string;extrapolation:string}; - markUnitLabel?:string;currency?:string;definition?:{id:string;version:number}|null;availability?:'no_remaining_contracts';trace?:PricingNode[];observationScope?:'selected_frame';observations?:Row[];inputSelection?:PriceInputSelection;snapshotId:string; curve:string; at:string; points:Point[]; evaluated:number; numericalPolicy:string}; + markUnitLabel?:string;currency?:string;definition?:{id:string;version:number}|null;availability?:'no_remaining_contracts';trace?:PricingNode[];observationScope?:'selected_frame'|'curve_lineage_v1';observations?:Row[];inputSelection?:PriceInputSelection;snapshotId:string; curve:string; at:string; points:Point[]; evaluated:number; numericalPolicy:string}; // The evaluated source owns these pins; a directory frame names only the // primary source and cannot validate another source's prices. export function priceSourceInputIds(result:CurveResult,workspace:Workspace):ReadonlySet{ const selection=result.inputSelection,source=selection?.source,revision=workspace.definitions.revisions?.[0]; const validId=(id:unknown):id is string=>typeof id==='string'&&id.length>0; - if(result.observationScope!=='selected_frame'||!selection||selection.snapshotId!==result.snapshotId||!source|| + if(!['selected_frame','curve_lineage_v1'].includes(result.observationScope??'')||!selection||selection.snapshotId!==result.snapshotId||!source|| workspace.definitions.revisions?.length!==1||!revision|| !validId(source.definitionRevisionId)||source.definitionRevisionId!==revision.id|| !validId(source.referenceCatalogId)||source.referenceCatalogId!==revision.reference_catalog_id|| @@ -54,6 +54,25 @@ export function exactRow(rows:Row[],key:string,value:string):Row { if(matches.length!==1)throw Error('Some market details are missing or inconsistent. Refresh and try again.'); return matches[0]; } +// A page read installs new arrays when its selection changes. Index only +// those snapshots, with the same missing/duplicate checks as exactRow. +const rowIndexes=new WeakMap>>(); +export function indexedRow(rows:readonly Row[],key:string,value:string):Row { + let columns=rowIndexes.get(rows); + if(!columns){columns=new Map();rowIndexes.set(rows,columns);} + let index=columns.get(key); + if(!index){ + index=new Map(); + for(const row of rows){ + const field=row[key]; + if(typeof field==='string')index.set(field,index.has(field)?null:row); + } + columns.set(key,index); + } + const found=index.get(value); + if(!found)throw Error('Some market details are missing or inconsistent. Refresh and try again.'); + return found; +} // The name a reference row is shown by: its name, else its symbol or display code, else its code made readable. export function referenceLabel(row:Row){if(typeof row.name==='string')return row.name.replace(/^Synthetic /,'');if(typeof row.symbol==='string')return row.symbol;if(typeof row.display_code==='string')return row.display_code;return friendly(row.code);} // Overlay rows are for presentation only; callers retain the original financial rows. diff --git a/web/tests/market-view.test.mjs b/web/tests/market-view.test.mjs index 12ac5b2c..c4ddb147 100644 --- a/web/tests/market-view.test.mjs +++ b/web/tests/market-view.test.mjs @@ -1,6 +1,6 @@ import {test} from 'node:test'; import assert from 'node:assert/strict'; -import {summaryTiles,contractCountText,displayDate,frameOptions,deliveryMonth,deliveryLabel,historyPriceLabel,displayInstant,timeLabels,publicText,publicColumns,quoteUnit,curveName,setCurveNames,clockLabel,displayExactInstant} from '../dist/market-view.js'; +import {summaryTiles,contractCountText,displayDate,frameOptions,deliveryMonth,deliveryLabel,historyPriceLabel,displayInstant,timeLabels,publicText,publicColumns,quoteUnit,curveName,setCurveNames,clockLabel,displayExactInstant,indexedRow} from '../dist/market-view.js'; import {dataSections,referenceGroups} from '../dist/data-sections.js'; import {newPreferences,parsePreferences} from '../dist/preferences.js'; const id='12345678-1234-1234-1234-123456789012'; @@ -100,3 +100,13 @@ test('curve display choices persist explicitly and reject unknown policies',()=> for(const contractDisplay of [null,{}, {version:'unknown',contracts:'eligible_contracts_v1'},{version:'curve_display_v1',contracts:'guess'}]) assert.throws(()=>parsePreferences(JSON.stringify({...preference,contractDisplay}))); }); + +test('indexed page rows refuse missing and duplicate values and follow array replacement',()=>{ + const first={id:'first',value:'1'},duplicate={id:'ambiguous',value:'1'}; + const rows=[first,duplicate,{...duplicate,value:'2'}]; + assert.equal(indexedRow(rows,'id','first'),first); + assert.throws(()=>indexedRow(rows,'id','ambiguous'),/missing or inconsistent/); + assert.throws(()=>indexedRow(rows,'id','unknown'),/missing or inconsistent/); + const replacement=[{id:'first',value:'3'}]; + assert.equal(indexedRow(replacement,'id','first').value,'3'); +});